Potential optimization wins

[vinhowe]

• Fusion, including extending the lazytensor IR
  • Concerned about how well this will work with the existing allocation optimization madness. Wonder if we can do fusion in such a way that preserves the information we'd need to do inplacing.
• Copying grads to respective allocations instead of relying on a shader to do this. This is empirically very expensive.
  • Maybe we extend the copy op + figure out a little algorithm to chunk copy operations based on their dependencies so we don't break up the compute pass every time we need to compute one? Not sure what the overhead for webgpu of switching in and out of a compute pass is, exactly.
• Actual profiling work + optimizing the running model around that

[vinhowe]

test