ipc: Remove unnecessary cache manipulation #646

Job	Run time
LP64-WIP	3m 22s
build-windows (imx8 imx8x imx8m imx8ulp)	8m 16s
build-linux (mnfst, imx8 imx8x imx8m imx8ulp)	3m 7s
build-windows (mtl)	6m 18s
manifest-check	56s
build-linux (mnfst, mtl)	3m 16s
build-windows (tgl tgl-h)	6m 55s
build-windows (-d, mtl)	6m 21s
build-linux (mnfst, lnl)	3m 7s
build-linux (mnfst, ptl)	3m 36s
build-linux (mnfst, tgl tgl-h)	3m 5s
build-linux (zmain, imx8 imx8x imx8m imx8ulp)	3m 15s
build-linux (zmain, mtl)	3m 8s
build-linux (zmain, lnl)	3m 19s
build-linux (zmain, ptl)	3m 32s
build-linux (zmain, tgl tgl-h)	3m 12s
build-linux (-d, mnfst, mtl)	3m 5s
build-linux (mnfst, --all)	3m 7s
compare-linux-win	31s
	1h 11m 28s

Provide feedback