Enhance BackendBench with TritonBench

[xuzhao9]

TritonBench is a suite that focuses on custom kernels generated by AI Compilers including Triton/Helion/Tilelang/ThunderKittens.

It can be easily used by installing as a library:

git clone https://github.com/meta-pytorch/tritonbench.git; cd tritonbench
pip install -e .

It is used as the main benchmark suite for Helion development: https://github.com/pytorch/helion/tree/main/benchmarks

It would be great if we can collaborate with BackendBench:

1. to increase BackendBench's coverage on custom kernels. E.g. adding LLM-era important kernels like linear attention, flex_attention, etc.

2. BackendBench can piggyback on Tritonbench's ability to obtain profiling results, including NCU, Kineto, and Power Charts

3. TritonBench supports backward ops, which could be useful to add training support in BackendBench

[msaroufim]

Thanks @xuzhao9! A lot of the above sounds really compelling and here's how I'd roughly prioritize them with most important first

1. Power aware benchmarks: This is also something @gevtushenko called us out on at PTC that I think is crucial to get right
2. For LLM generated kernels: @jiannanWang I'm thinking we need a new suite here something like "LLM kernels" which would capture what users actually want. There are some things like rms form or fused linear cross entropy that should have been pytorch ops ages ago. Same comment for the backwards ops let's see what we need to have an important LLM benchmark suite
3. On the profiler: The main challenge I have still is the feedback is way too long for LLMs, so we'll need some feature engineering here