Update directory structure in submission_rules.adoc #195
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
| System names and implementation names may be arbitrary. | ||
|
|
||
| <benchmark> must be one of {**resnet50, retinanet, rnnt, bert-99, bert-99.9, dlrm-99, dlrm-99.9, 3d-unet-99, 3d-unet-99.9**}. The postfix '-99' and '-99.9' indicate that the accuracy must be >= 99% or 99.9% of the target accuracy. | ||
| All submission content should align to the above directory structure, even when not enforced by submission support tools such as submission-checker.py. Content should not reside outside of the <submitter_organization> direct sub-directories: {**systems, code, documentation, results**}. |
There was a problem hiding this comment.
I think we should avoid this sentence and force this in the submission checker to make it simpler for the submitters.
submission_rules.adoc
Outdated
| <benchmark> must be one of {**resnet50, retinanet, rnnt, bert-99, bert-99.9, dlrm-99, dlrm-99.9, 3d-unet-99, 3d-unet-99.9**}. The postfix '-99' and '-99.9' indicate that the accuracy must be >= 99% or 99.9% of the target accuracy. | ||
| All submission content should align to the above directory structure, even when not enforced by submission support tools such as submission-checker.py. Content should not reside outside of the <submitter_organization> direct sub-directories: {**systems, code, documentation, results**}. | ||
|
|
||
| <benchmark> must be one of {**3d-unet-99, 3d-unet-99.9, dlrm-v2-99, dlrm-v2-99.9, gptj-99, gptj-99.9, llama2-70b-99, llama2-70b-99.9, llama2-70b-interactive-99, llama2-70b-interactive-99.9, llama3.1-405b, mixtral-8x7b, resnet, retinanet, rgat, stable-diffusion-xl**}. The postfix '-99' and '-99.9' indicate that the accuracy must be >= 99% or 99.9% of the target accuracy. |
There was a problem hiding this comment.
Just a suggestion - To keep it safe across the rounds, it'll be good to avoid listing the benchmarks here and instead say applicable benchmark for the given round with even a link to the submission checker where they are listed.
There was a problem hiding this comment.
Thanks for the suggestion, Arjun. I will make the change.
|
The PR looks good to me; however, during our workgroup meeting, Pablo noted that implementing these changes may not be feasible before Inference v5.1 due to other priorities. We will need to revisit this in the next Inference WG meeting. If the situation remains unchanged, we will need to hold this PR until the v5.1 submission is complete. |
|
Are we planning on enforcing these directory rules through submission checker? |
The plan is 'yes'. But due to the short period between this issue and the v5.1 submission, it was decided to postpone until v6.0. |
|
Also, this seems to be specific to inference. How about training? Does the structure look good for that or there are issues we need to address there as well, while we are on it? |
The directory structures for training and inference are distinct. This change only applies to inference. |
pgmpablo157321
force-pushed
the
master
branch
from
dd36e1a to
76569b4
Compare
Related to Inference v5.0 post-mortem.
Fixing date references after PR was first generated.
pgmpablo157321
force-pushed
the
master
branch
from
76569b4 to
76ff0c1
Compare
|
@keithachorn-intel to update |
|
Directory 'code' updated to 'src' per WG request. @pgmpablo157321 @arjunsuresh , ready for merge. |
6 participants
Related to Inference v5.0 post-mortem.