allow queues to set limits by GPU type

[sam-huang1223]

What you would like to be added?

queues should be able to set quotas by nvidia.com/gpu type, like H100 or B200 or etc.

Why is this needed?

allow usage of KAI in clusters containing multiple types of GPUs (a common occurence out in the wild)

[itsomri]

Hey Sam,

At the moment, you could achieve per-type quota manegment using scheduling shards.
The idea is to divide the cluster to distinct "node pools" (shards). Each shard will get it's own scheduler, and can be configured independently. Each shard gets it's own queues, so quota is also managed independently, and jobs are assigned to shards by a label.
That said, we recognize that there's also a use case for single-nodepool, per-type quota management - shards are a pretty strong level of isolation that might not fit every use case.