Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
642 commits
Select commit Hold shift + click to select a range
3910973
Update build_wheels.yml | Recommendation from MLCommons Admins (#1895)
arjunsuresh Nov 6, 2024
6dcaae8
Update fid_score.py | Move the sys path update before the import. (#1…
arjunsuresh Nov 6, 2024
b0686db
Fix SDXL bug with diffuser 0.30.0 while giving downloaded model path …
arjunsuresh Nov 6, 2024
d3c01ed
Update GNN reference implementation: add DGL backend (#1903)
pgmpablo157321 Nov 6, 2024
dffd292
Update artifact actions from v3 to v4 in build_wheels.yml (#1906)
nathanwasson Nov 8, 2024
97122a2
Fix typo in fid_score.py, fail_safe for SDXL short runs, fix argument…
arjunsuresh Nov 14, 2024
e398b90
Update requirements.txt (#1907)
arjunsuresh Nov 14, 2024
4763c6b
Removed TEST05 compliance requirement (#1925)
arjunsuresh Nov 15, 2024
c570ff4
Fix a minor bug in preprocess_submission (#1926)
arjunsuresh Nov 19, 2024
b2b346c
Fix: GitHub workflow script injection (#1936)
joycebrum Nov 19, 2024
4d7a5c8
Add GNN calibration ids (#1916)
pgmpablo157321 Nov 19, 2024
acc3eb4
- Fixed syntax error on list .insert method (#1921)
Oseltamivir Nov 19, 2024
61a8a6a
Sync Docs (#1908)
arjunsuresh Nov 19, 2024
35c7e1f
🔄 synced file(s) with mlcommons/power-dev (#1918)
mlcommons-bot Nov 19, 2024
d28a530
Fix Dockerfile.eval: huggingface-cli doesn't exist (#1805)
qihqi Nov 26, 2024
45544f3
Llm large reference (#1951)
pgmpablo157321 Dec 3, 2024
feca9f6
Increment version to 4.1.24
mrmhodak Dec 3, 2024
b6f5a34
Fixes to SDXL accuracy output, submission preprocessor, fixes #1938 (…
arjunsuresh Dec 4, 2024
6ac00b1
[Automated Commit] Format Codebase
mlcommons-bot Dec 4, 2024
a2baf12
Add R-GAT calibration + remove old reference implementation (#1943)
pgmpablo157321 Dec 4, 2024
e7c301c
Increment version to 4.1.30
mrmhodak Dec 4, 2024
21d845e
Llm large fixes (#1959)
pgmpablo157321 Dec 4, 2024
dd9be96
Submission checker v5.0 (#1955)
pgmpablo157321 Dec 4, 2024
8ce46d5
Upgrade loadgen version to 5.0 (#1962)
nvzhihanj Dec 9, 2024
4968ba0
Fix loadgen build for version numbers having "0" (#1967)
arjunsuresh Dec 9, 2024
6afe996
Increment version to 5.0.1
pgmpablo157321 Dec 9, 2024
e69b504
Fix Dockerfile for 405B (#1960)
nvzhihanj Dec 10, 2024
cec62ff
Update backend_pytorch.py | Fix lock usage (#1964)
arjunsuresh Dec 10, 2024
bc5e6fb
Add llama3 metrics + remove llama3-99.9 (#1973)
pgmpablo157321 Dec 11, 2024
8948934
Fix submission checker for v5.0 rgat (#1974)
arjunsuresh Dec 12, 2024
2d2eb30
Fix test05 seeds missing error for v5.0 submission checker (#1976)
arjunsuresh Dec 12, 2024
fa27456
Fix llama3-405B docker workflow and performance sample count (#1978)
nvzhihanj Dec 13, 2024
1e48c0d
Increment version to 5.0.2
mrmhodak Dec 13, 2024
c71b83f
Fix submission generation for v5.0 (#1981)
arjunsuresh Dec 13, 2024
aebc018
More minor fixes for llama3.1-405b (#1983)
nvzhihanj Dec 17, 2024
3ae2b2a
Remove unused rgat files (#1961)
pgmpablo157321 Dec 18, 2024
03c9666
Update docker GPU, avoid long build time (#1966)
pgmpablo157321 Dec 18, 2024
867def4
Require equal issue mode for R-GAT (#1968)
pgmpablo157321 Dec 18, 2024
b3e1e8e
Increment version to 5.0.3
mrmhodak Dec 18, 2024
8397bec
Docs update for r-gat (#1969)
arjunsuresh Dec 18, 2024
647f9f8
[Automated Commit] Format Codebase
mlcommons-bot Dec 18, 2024
e6069aa
Update automated run command section - R-GAT (#1970)
anandhu-eng Dec 18, 2024
00945c3
Unify llama3 names to llama3.1-405b (#1982)
pgmpablo157321 Dec 19, 2024
6af0288
Increment version to 5.0.4
mrmhodak Dec 19, 2024
f5382b7
Create test-rgat.yml (#1984)
arjunsuresh Dec 19, 2024
939b2fe
Update compliance test table (#1987)
pgmpablo157321 Dec 19, 2024
ae23dfc
Create benchmark-checklist.md for r-gat (#1985)
arjunsuresh Dec 19, 2024
9abda7c
Add rgat accuracy format
pgmpablo157321 Dec 16, 2024
c2cd92e
Output accuracy in expected submission format
pgmpablo157321 Dec 19, 2024
be6ff52
[Automated Commit] Format Codebase
mlcommons-bot Dec 20, 2024
2fdb814
Automotive reference implementation (#1954)
arjunsuresh Dec 21, 2024
b9f22d6
Implement KSM to pull a PAT for commits in format.yml (#2011)
nathanwasson Jan 2, 2025
d6c3a8d
Don't do a loadgen release from dev branch, add python3.12,13 to load…
arjunsuresh Jan 7, 2025
e0d093d
#1777: Update mixtral reference score in the submission checker (#2021)
nvzhihanj Jan 7, 2025
1f73d4c
Avoid creating a lot of queries for equal issue mode (#2031)
pgmpablo157321 Jan 14, 2025
c40d6e1
Increment version to 5.0.6
mrmhodak Jan 14, 2025
88f4d23
Add mixtral dockerfile and standalone inference script (#2029)
nvzhihanj Jan 14, 2025
af05f0d
Parallelize evaluation of llama2 rouge scores (#1995)
v-shobhit Jan 14, 2025
be96b28
Update test_settings_internal.cc | Allow sample_concatenate_permutati…
arjunsuresh Jan 14, 2025
1618142
Increment version to 5.0.7
pgmpablo157321 Jan 14, 2025
5d83ed5
5.0 Random seeds (#2034)
pgmpablo157321 Jan 14, 2025
ded1712
Increment version to 5.0.8
mrmhodak Jan 14, 2025
1fa600d
RGAT remove in-memory flag (#2033)
pgmpablo157321 Jan 14, 2025
4fc54b5
Fix seeds: correct seeds in mlperf.conf (#2036)
pgmpablo157321 Jan 14, 2025
7fe4741
Increment version to 5.0.9
mrmhodak Jan 14, 2025
fc2557d
🔄 synced file(s) with mlcommons/power-dev (#1990)
mlcommons-bot Jan 15, 2025
ab7f700
Increment version to 5.0.10
arjunsuresh Jan 15, 2025
ef1bd3b
Fix reporting of samples per queries with equal issue mode (#2039)
pgmpablo157321 Jan 16, 2025
277cf38
Add missing mixtral requirements file (#2042)
nvzhihanj Jan 16, 2025
dcd0c3e
Update format.yml to us GITHUB_TOKEN & GitHub Actions bot (#2044)
nathanwasson Jan 16, 2025
ba71b21
Low latency checks to submission checker and report low latency (#2017)
pgmpablo157321 Jan 21, 2025
2ed0a3e
Increment version to 5.0.12
mrmhodak Jan 21, 2025
6315397
Create TEST01 audit.conf for rgat (#2037)
arjunsuresh Jan 21, 2025
25bba8f
Increment version to 5.0.13
arjunsuresh Jan 21, 2025
e618c31
Merge automotive reference into main branch (#2047)
rod409 Jan 21, 2025
138b794
Mixtral fix: match reference with standalone script (#2054)
pgmpablo157321 Jan 28, 2025
5091994
Add automotive submission checker (#2051)
pgmpablo157321 Jan 28, 2025
be4314b
Edit mlperf.conf for pointpainting - automotive (#2052)
anandhu-eng Jan 28, 2025
35030b8
Increment version to 5.0.14
arjunsuresh Jan 28, 2025
9c3c34c
Automotive benchmark table (#2050)
pgmpablo157321 Jan 28, 2025
9dad99d
Update backend_pytorch_native.py | Fixes #2056 (#2057)
arjunsuresh Jan 28, 2025
115dd5b
Update truncate_accuracy_log.py | Remove a wrong ERROR message in log…
arjunsuresh Jan 29, 2025
9b32a74
Changes for final report generation - PointPainting (#2063)
anandhu-eng Jan 29, 2025
10d123c
Fix mlperf.conf link for equal issue mode (#2069)
anandhu-eng Jan 31, 2025
d7bbf88
Update results.cc | Add another significant digit to percentile laten…
arjunsuresh Feb 1, 2025
d67b626
Increment version to 5.0.15
arjunsuresh Feb 1, 2025
aea55ed
Create benchmark checklist for pointpainting (#2068)
anandhu-eng Feb 1, 2025
9a2c0a1
Fix np.memmap usage, add flag to force not using memmap (#2081)
nv-alicheng Feb 3, 2025
7f5025f
Add device map - pointpainting automotive (#2087)
anandhu-eng Feb 4, 2025
54cc04d
PointPainting Documentation update (#2089)
rod409 Feb 4, 2025
c2a0117
Update verify_performance.py (#2071)
arjunsuresh Feb 4, 2025
4b7475b
Match server scenario to standalone implementation (#2086)
pgmpablo157321 Feb 4, 2025
421d20b
Add reference model details (#2084)
anandhu-eng Feb 4, 2025
4d0b358
[405B] Set max_tokens to 2k (#2088)
attafosu Feb 4, 2025
67a33a4
Add parameter and flops values (#2090)
anandhu-eng Feb 4, 2025
f000467
Update benchmark-checklist.md - PointPainting (#2083)
anandhu-eng Feb 4, 2025
32f9e84
Report improvement - support output of IDs to a json file (#2059)
arjunsuresh Feb 5, 2025
2e996ad
Fixed GPTJ accuracy checker (#2093)
nvzhihanj Feb 6, 2025
2a41918
Fix SDXL, Retinanet and GPTJ accuracy checker (#2094)
nvzhihanj Feb 6, 2025
9668b86
Update auto-update-dev.yml (#2096)
arjunsuresh Feb 6, 2025
e1d1a4f
Update submission_checker.py | Prevent empty accuracy in open divisio…
arjunsuresh Feb 6, 2025
fa5d4f7
Added information about GitHub tests currently live (#2091)
anandhu-eng Feb 7, 2025
7bf5997
Docs update, fix download links for llama models (#2055)
arjunsuresh Feb 14, 2025
a09ef11
Add MLC Automation commands (#2115)
anandhu-eng Feb 18, 2025
eaace3f
Remove llama3.1 user conf unnecessary and misleading lines (#2114)
pgmpablo157321 Feb 18, 2025
df13693
Update docs (#2118)
arjunsuresh Feb 18, 2025
2ab7e39
Update verify_performance.py | Fix compliance test for extra percenti…
arjunsuresh Feb 19, 2025
cc9a89e
Update default version of final report script (#2124)
pgmpablo157321 Feb 20, 2025
f4a195e
Update accuracy_igbh.py (#2123)
arjunsuresh Feb 21, 2025
4bf9fc6
🔄 synced file(s) with mlcommons/power-dev (#2125)
mlcommons-bot Feb 21, 2025
1873bd7
Final report cosmetic fix (#2134)
pgmpablo157321 Feb 25, 2025
9926a70
Update loadgen package name in classification_and_detection setup (#2…
annietllnd Feb 25, 2025
0e72613
Add exception for github-actions[bot] to cla.yml (#2135)
nathanwasson Feb 25, 2025
bec5714
Update CODEOWNERS
nathanwasson Feb 25, 2025
0602e21
Update GitHub Actions bot exception in cla.yml
nathanwasson Feb 25, 2025
c0b60cd
Final report cosmetic fix (#2141)
pgmpablo157321 Feb 27, 2025
e216af0
Update submission_checker.py | Fix open model unit in Results (#2144)
arjunsuresh Mar 4, 2025
5e90395
Add Llama 3.1 to special unit dict (#2150)
pgmpablo157321 Mar 10, 2025
fd021be
[Post Mortem] Log number of errors in detail log (#2164)
pgmpablo157321 Apr 15, 2025
c6d4f36
Increment version to 5.0.16
mrmhodak Apr 15, 2025
9a5867f
Docs update (#2137)
nathanwasson Apr 22, 2025
2b465fc
[Post Mortem] Check all systems and measurements folders have results…
pgmpablo157321 Apr 22, 2025
836f22c
[Post Mortem] Add calibration check to submission checker (#2185)
pgmpablo157321 Apr 29, 2025
09715a3
Add find peak performance documentation (#2186)
pgmpablo157321 May 20, 2025
654b6ec
Increment version to 5.0.17
mrmhodak May 20, 2025
cc7d6d8
Check equal issue for open division + check accuracy run covers all t…
pgmpablo157321 May 20, 2025
06b3707
add deepseek-r1 (#2198)
viraatc Jun 7, 2025
30eb03a
add deepseek-r1 (#2201)
viraatc Jun 9, 2025
341b190
update deepseek-r1 (#2202)
viraatc Jun 10, 2025
44d9192
Call existing submission generation workflow from mlperf-automations …
anandhu-eng Jun 10, 2025
a8cd212
Add deepseek configuration + v5.1 Readme (#2203)
pgmpablo157321 Jun 10, 2025
088bb82
Increment version to 5.0.18
arjunsuresh Jun 10, 2025
e0fdcea
Updated readme with mlc commands for model,dataset,accuracy and submi…
anandhu-eng Jun 10, 2025
771b789
Docs - Update disk space for reference implementation (#2159)
anandhu-eng Jun 10, 2025
3ab3b5b
Update Waymo access instructions (#2148)
nathanwasson Jun 10, 2025
9e271f9
Update verify_performance.py | Refactor the code (#2073)
arjunsuresh Jun 11, 2025
93cebda
Update mlperf.conf with final deepseek configuration (#2208)
pgmpablo157321 Jun 11, 2025
ab2b810
Increment version to 5.0.19
arjunsuresh Jun 11, 2025
d690a57
Add whisper reference implementation (#2193)
pgmpablo157321 Jun 16, 2025
248bc35
Increment version to 5.0.20
arjunsuresh Jun 16, 2025
d8764c4
Llama3.1-8b reference implementation (#2190)
pgmpablo157321 Jun 16, 2025
d286c08
Increment version to 5.0.21
arjunsuresh Jun 16, 2025
b0ce3fe
Skip imagenet calibration dataset download (#2209)
anandhu-eng Jun 16, 2025
0910c6e
Add v5.1 submission checker (#2204)
pgmpablo157321 Jun 17, 2025
5c28254
Increment version to 5.0.22
arjunsuresh Jun 17, 2025
263335c
Update Llama 3.1 model access instructions (#2149)
nathanwasson Jun 18, 2025
f9813a4
Rename audit.conf to audit.config (#2127)
arjunsuresh Jun 18, 2025
603b918
Quick fix: correct metrics (#2211)
pgmpablo157321 Jun 18, 2025
86326a6
[Whisper] Updating dataset for repacked dev-all. (#2212)
keithachorn-intel Jun 19, 2025
4cf167a
Partial fix for compliance TEST01 update (#2215)
keithachorn-intel Jun 23, 2025
ec84cab
Completion of compliance TEST01 fix (#2217)
keithachorn-intel Jun 23, 2025
537729e
fix typo (#2219)
anandhu-eng Jun 24, 2025
12e16a7
Fix Readme for inference v5.1 (#2216)
pgmpablo157321 Jun 24, 2025
75adba3
Fix small llm readme (#2220)
pgmpablo157321 Jun 24, 2025
064db01
[DLRMv2] Updating DLRMv2 dataset size (#2222)
keithachorn-intel Jun 24, 2025
e983036
Remove incorrect MLC Commands (#2218)
anandhu-eng Jun 24, 2025
0a3570e
Fix Typo in Interactive Latencies (#2147)
mrmhodak Jun 25, 2025
fdac4dc
Host speech2text artifacts in MLC storage bucket (#2223)
pgmpablo157321 Jul 2, 2025
62a0dfd
Add missing interactive configurations (#2224)
pgmpablo157321 Jul 7, 2025
20628f0
Increment version to 5.0.23
mrmhodak Jul 7, 2025
6a5fcc5
Update compliance table (#2243)
pgmpablo157321 Jul 8, 2025
7ef8b14
Add deepseek dataset sources (#2242)
pgmpablo157321 Jul 8, 2025
b25eac3
Adding mlperf.conf setting for Whisper (#2238)
keithachorn-intel Jul 8, 2025
745b9fe
Increment version to 5.0.24
mrmhodak Jul 8, 2025
7482011
Fix Docs (#2229)
arjunsuresh Jul 8, 2025
ae1320c
Incorrect Regex for RougeLSum (#2230)
hvagadia Jul 8, 2025
24767db
update eval_accuracy.py and deepseek thresholds (#2233)
viraatc Jul 8, 2025
e05fda5
Add llama3.1-8b-edge as a separated benchmark (#2231)
pgmpablo157321 Jul 8, 2025
1bc3e99
Increment version to 5.0.25
arjunsuresh Jul 8, 2025
bb0e01a
accuracy (#2259)
wu6u3tw Jul 15, 2025
bcb600e
fix regex (#2260)
wu6u3tw Jul 15, 2025
7d86e6b
Update download path for llama3.1_8b dataset (#2261)
pgmpablo157321 Jul 16, 2025
5232291
Update README.md (#2255)
keithachorn-intel Jul 16, 2025
1446b35
Update version generate_final_report.py (#2269)
pgmpablo157321 Jul 18, 2025
c57507b
Add interactive scenario in the TEST06, bump loadgen version to 5.1 (…
nvzhihanj Jul 21, 2025
fa32df9
Pinning vllm for speech-to-text reference (#2273)
keithachorn-intel Jul 22, 2025
7b9643c
Fix SingleStream llama3.1-8b typo (#2274)
pgmpablo157321 Jul 22, 2025
9a1990e
Update download path for DeepSeek-R1 Dataset (#2275)
pgmpablo157321 Jul 22, 2025
35d9836
Update documentation (#2279)
anandhu-eng Jul 23, 2025
50de991
Address issue that logger.info not captured by stdout; remove redunda…
nvzhihanj Jul 23, 2025
4435cb5
Remove TEST01 for interactive scenario; add TEST06 for them (#2281)
nvzhihanj Jul 24, 2025
a817ac5
Fix missing comma: TEST01 not required for any LLMs. (#2283)
psyhtest Jul 24, 2025
d82d7d4
Fix ds-r1 acc checker output format not captured by submission checke…
nvzhihanj Jul 24, 2025
6481ff4
Update README.md (#2288)
mrmhodak Jul 24, 2025
b9ed3c7
fix: update ds-r1 truncate max-output-len to 20k (was 32k) (#2290)
viraatc Jul 25, 2025
c54683d
Fix llama3.1-8b metric and dataset (#2300)
pgmpablo157321 Jul 29, 2025
c308e42
Add interactive scenario to llama3.1 models (#2299)
pgmpablo157321 Jul 29, 2025
4a01733
Allow more flexible datatypes in measurements file (#2298)
pgmpablo157321 Jul 29, 2025
33a0c34
Update evaluation.py (#2303)
taran2210 Jul 29, 2025
01bc11a
Only require Server or Interactive for closed (#2304)
pgmpablo157321 Jul 29, 2025
9557914
[Whisper] Adding n_token return for compliance fix (#2305)
keithachorn-intel Jul 30, 2025
091d51f
Fix checking power directory (#2306)
anandhu-eng Jul 30, 2025
47b91b9
Only check for token latency requirements for server scenario (#2313)
pgmpablo157321 Jul 31, 2025
6ad8e26
Use server SUT for SingleStream (#2314)
pgmpablo157321 Aug 1, 2025
1b64700
Update the default value for repository arg (#2317)
anandhu-eng Aug 4, 2025
e5f6208
Update preprocess_submission.py | Skip inferring offline scenario if …
arjunsuresh Aug 6, 2025
8c3c1e2
Fix: add llama3.1-8b-edge to generate_final_report (#2319)
pgmpablo157321 Aug 6, 2025
70d58ad
Allow lowercase 'interactive' as scenario name (#2315)
psyhtest Aug 7, 2025
e20506b
Use sample latency as the metric for llama3.1_8b_edge SingleStream (#…
pgmpablo157321 Aug 20, 2025
e064826
Remove rclone references and update download instructions for DeepSee…
anivar Aug 20, 2025
65e3228
hide long time untested implementations from docs (#2328)
anandhu-eng Sep 1, 2025
46beea8
Initial draft for SCC 25 documentation (#2331)
anandhu-eng Sep 10, 2025
75a2012
fix for fstring (#2332)
anandhu-eng Sep 10, 2025
938be6d
Updation of automation run commands - v5.1_dev (#2333)
anandhu-eng Sep 11, 2025
2035f3f
Fixes for docs (#2334)
anandhu-eng Sep 14, 2025
5fbf01a
Update submission_checker.py | Fixes 2325 (#2326)
arjunsuresh Sep 16, 2025
a6bd784
Fixes for scc doc (#2339)
anandhu-eng Sep 18, 2025
1d6b4c3
Update link to AMD readme for SCC (#2340)
anandhu-eng Sep 18, 2025
8c07d7c
Add additional information for dataset and model downloads (#2343)
anandhu-eng Sep 24, 2025
7c8c84d
Provide correct syntax of how batch size is provided for nvidia imple…
anandhu-eng Sep 29, 2025
b15073f
Fix lookup of required accuracy delta. (#2337)
psyhtest Sep 30, 2025
8e20008
Llama3.1-405b: Add commands for model/dataset download using R2-downl…
anandhu-eng Oct 6, 2025
ea86541
Update commands to include R2-downloader for RGAT model download (#2350)
anandhu-eng Oct 6, 2025
2a3d5cc
Update minimum disk space required for PointPainting (#2353)
anandhu-eng Oct 6, 2025
dc8ca5b
Automation Docs: Add information about launching docker in privileged…
anandhu-eng Oct 7, 2025
d9efabf
Remove Nvidia folder from compliance tree (#2354)
pgmpablo157321 Oct 7, 2025
f4d495b
Provide help - insufficient max locked memory error for Nvidia runs (…
anandhu-eng Oct 7, 2025
fbed09d
Update SUT_VLLM.py (#2349)
viraatc Oct 7, 2025
eee6352
Fix minor typos in reference_mlperf_perf.sh and reference_mlperf_accu…
naveenmiriyaluredhat Oct 7, 2025
11e469d
[LoadGen] Time to Output Token -> Time per Output Token (#2360)
wangshangsam Oct 13, 2025
0e556f1
Increment version to 5.1.1
hanyunfan Oct 13, 2025
0b8ca03
[Whisper] Add labels' in the whisper output (#2252)
wu6u3tw Oct 13, 2025
f1b7ea1
Update gptj model download command to support R2 downloader (#2368)
anandhu-eng Oct 20, 2025
3693669
Update model and dataset download commands to support R2 downloader (…
anandhu-eng Oct 20, 2025
3d06ae1
Discard duplicate information about external model download (#2370)
anandhu-eng Oct 20, 2025
0c27530
MIXTRAL - Update MLCFlow dataset and model download commands to suppo…
anandhu-eng Oct 20, 2025
a921baf
Update DLRM v2 assets download commands to support R2 downloader (#2372)
anandhu-eng Oct 20, 2025
55e41e0
SDXL: Update MLC commands to support model download through R2 (#2373)
anandhu-eng Oct 20, 2025
6b92d87
Updation for inference docs page: migration to R2 (#2374)
anandhu-eng Oct 20, 2025
d4bf062
Update MLC commands to support downloads through R2 for PointPainting…
anandhu-eng Oct 20, 2025
6776245
Replaced shell commands with Python for Windows compliance script com…
sujik18 Oct 21, 2025
808e2d7
[VLM] Offline scenario, performance-only mode of the reference implem…
wangshangsam Nov 11, 2025
ceafe86
update docs with submission instruction + hyperlinks for heading (#2390)
anandhu-eng Nov 15, 2025
32b6d66
[VLM] Server Scenario, performance and accuracy (#2388)
johncalesp Nov 18, 2025
ee809ab
Submission dir update (#2366)
pgmpablo157321 Nov 25, 2025
59c59dd
Add note on LoadGen PyPI auto-build workflow (#2398)
anandhu-eng Nov 26, 2025
06c742a
Increment version to 5.1.2
arjunsuresh Nov 26, 2025
353bfb8
Revert changes used for testing (#2401)
anandhu-eng Nov 26, 2025
fff953c
[VLM] Accuracy Evaluation (#2393)
johncalesp Dec 2, 2025
95dfd4c
[VLM] Add brand field to accuracy evaluation (#2404)
johncalesp Dec 9, 2025
8999c4d
[VLM] Update the notebook to reflect the latest version of the datase…
wangshangsam Dec 9, 2025
003f4fb
Update loadgen to 6.0; Bulk update 6.0 checker bits (#2415)
nvzhihanj Dec 18, 2025
c098f80
feat: add MTP to ds-r1 ref. impl (#2403)
viraatc Dec 19, 2025
abed98c
Increment version to 6.0.1
arjunsuresh Dec 19, 2025
ff627df
Add tests to detect failures in igbh dataset download (#2359)
anandhu-eng Dec 22, 2025
e93f59d
Text to Video Reference Implementation (#2413)
hvagadia Dec 22, 2025
f4a2cca
[VLM] Fixing request timeout error, and enable VllmDeployer to fail f…
wangshangsam Dec 22, 2025
0369dd1
Increment version to 6.0.2
mrmhodak Dec 22, 2025
7f42a83
Initial draft for Inference submission guide (#2378)
anandhu-eng Dec 23, 2025
27db053
[LoadGen] Bump version from 6.0.2 to 6.0.3 (#2417)
wangshangsam Dec 23, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
9 changes: 9 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# These owners will be the default owners for everything in the repo.
# Unless a later match takes precedence,they will be requested for review when someone opens a pull request.
* @mlcommons/wg-inference

/.github/CODEOWNERS @mlcommons/systems

/.github/workflows/cla.yml @mlcommons/systems

/LICENSE.md @mlcommons/systems
41 changes: 41 additions & 0 deletions .github/workflows/auto-update-dev.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: Auto-Update Dev Branches from Master

on:
push:
branches:
- master # Trigger workflow on commits to 'master' branch
workflow_dispatch: {}

jobs:
update-dev:
if: github.repository_owner == 'mlcommons'
strategy:
matrix:
branch: [ "dev", "docs" ]
runs-on: ubuntu-latest

permissions:
contents: write # Required to push to protected branches

steps:
- name: Checkout Main Branch
uses: actions/checkout@v4
with:
ref: ${{ matrix.branch }}
fetch-depth: 0
ssh-key: ${{ secrets.DEPLOY_KEY }}

- name: Configure Git User
run: |
git config user.name "github-actions"
git config user.email "github-actions@github.com"

- name: Merge auto-update into ${{ matrix.branch }}
run: |
git fetch origin master:master
git merge --no-ff master -m "Auto-merge updates from master branch"

- name: Push Changes to ${{ matrix.branch }}
run: |
git push origin ${{ matrix.branch }}

145 changes: 145 additions & 0 deletions .github/workflows/build_wheels.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
name: Build loadgen wheels and release them into PYPI

on:
release:
types: [published]
push:
branches:
- master
- loadgen-release
paths:
- loadgen/**

jobs:
update_version:
name: Update version only on ubuntu but used by windows and macos
if: github.repository_owner == 'mlcommons'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 2
ssh-key: ${{ secrets.DEPLOY_KEY }}

# Check if VERSION.txt file has changed in this push
- name: Check if VERSION.txt file has changed
id: version_changed
run: |
echo "version_changed=false" >> $GITHUB_ENV
echo "new_version=" >> $GITHUB_ENV # Initialize with empty value
if git diff --name-only HEAD~1 | grep -q "VERSION.txt"; then
echo "VERSION.txt file has been modified"
echo "version_changed=true" >> $GITHUB_ENV
new_version=$(cat loadgen/VERSION.txt)
echo "new_version=$new_version" >> $GITHUB_ENV
else
echo "VERSION.txt file has NOT been modified"
fi

# Step 4: Increment version if VERSION.txt was not changed
- name: Increment version if necessary
id: do_version_increment
if: env.version_changed == 'false'
run: |
cd loadgen
# Check if VERSION file exists, else initialize it
if [ ! -f VERSION.txt ]; then
echo "0.0.0" > VERSION.txt
fi

version=$(cat VERSION.txt)
IFS='.' read -r major minor patch <<< "$version"
patch=$((patch + 1))
new_version="$major.$minor.$patch"
echo $new_version > VERSION.txt
echo "New version: $new_version"
echo "new_version=$new_version" >> $GITHUB_ENV

# Step 5: Commit the updated version to the repository
- name: Commit updated version
if: env.version_changed == 'false'
run: |
cd loadgen
git config --global user.name "${{ github.actor }}"
git config --global user.email "${{ github.actor }}@users.noreply.github.com"
git add VERSION.txt
git commit -m "Increment version to $new_version"
git push

build_wheels:
name: Build wheels on ${{ matrix.os }}
if: github.repository_owner == 'mlcommons'
needs: update_version
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macos-latest, macos-15]

steps:
- uses: actions/checkout@v3

- uses: actions/setup-python@v3

- name: Install requirements
run: python -m pip install cibuildwheel twine build

- name: Build src dist
if: ${{ matrix.os == 'ubuntu-latest' }}
run: |
python -m build --sdist --outdir wheels loadgen

- name: Build wheels
run: git pull && python -m cibuildwheel loadgen/ --output-dir wheels

# Save wheels as artifacts
- name: Upload built wheels
uses: actions/upload-artifact@v4
with:
name: wheels-${{ matrix.os }}
path: wheels

publish_wheels:
if: github.repository_owner == 'mlcommons'
needs: build_wheels # Wait for the build_wheels job to complete
runs-on: ubuntu-latest # Only run this job on Linux
environment: release
permissions:
# IMPORTANT: this permission is mandatory for trusted publishing
id-token: write
steps:
- uses: actions/checkout@v3

# Download the built wheels from ubuntu
- name: Download Ubuntu wheels
uses: actions/download-artifact@v4
with:
name: wheels-ubuntu-latest
path: wheels
# Download the built wheels from macOS-latest
- name: Download macOS-latest wheels
uses: actions/download-artifact@v4
with:
name: wheels-macos-latest
path: wheels
# Download the built wheels from macOS-13 (x86)
- name: Download macOS-15 (x86) wheels
uses: actions/download-artifact@v4
with:
name: wheels-macos-15
path: wheels
# Download the built wheels from Windows
- name: Download Windows wheels
uses: actions/download-artifact@v4
with:
name: wheels-windows-latest
path: wheels
# Publish
- name: Publish
uses: pypa/gh-action-pypi-publish@release/v1
with:
verify-metadata: true
skip-existing: true
packages-dir: wheels
repository-url: https://upload.pypi.org/legacy/
verbose: true
9 changes: 5 additions & 4 deletions .github/workflows/cla.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@

name: "cla-bot"
on:
issue_comment:
Expand All @@ -7,25 +8,25 @@ on:

jobs:
cla-check:
if: github.repository_owner == 'mlcommons'
runs-on: ubuntu-latest
steps:
- name: "MLCommons CLA bot check"
if: (github.event.comment.body == 'recheck') || github.event_name == 'pull_request_target'
# Alpha Release
uses: sub-mod/github-action@v3
uses: mlcommons/cla-bot@master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# the below token should have repo scope and must be manually added by you in the repository's secret
PERSONAL_ACCESS_TOKEN : ${{ secrets.MLCOMMONS_BOT_CLA_TOKEN }}
with:
path-to-signatures: 'cla-bot/v1/cla.json'
path-to-document: 'https://github.com/mlcommons/systems/blob/main/mlcommons_cla.txt' # e.g. a CLA or a DCO document
# branch should not be protected
branch: 'main'
allowlist: user1,bot*
allowlist: user1,bot*,github-actions,github-actions[bot]
remote-organization-name: mlcommons
remote-repository-name: systems

#below are the optional inputs - If the optional inputs are not given, then default values will be taken
#remote-organization-name: enter the remote organization name where the signatures should be stored (Default is storing the signatures in the same repository)
#remote-repository-name: enter the remote repository name where the signatures should be stored (Default is storing the signatures in the same repository)
Expand Down
64 changes: 64 additions & 0 deletions .github/workflows/format.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Automatic code formatting
name: "Code formatting"
on:
push:
branches:
- "**"

env:
python_version: "3.9"

jobs:
format-code:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Set up Python ${{ env.python_version }}
uses: actions/setup-python@v3
with:
python-version: ${{ env.python_version }}

- name: Format modified Python files
env:
filter: ${{ github.event.before }}
run: |
python3 -m pip install autopep8
for FILE in $(git diff --name-only $filter | grep -E '.*\.py$')
do
# Check if the file still exists in the working tree
if [ -f "$FILE" ] && [ "$FILE" != "tools/submission/power/power_checker.py" ]; then
autopep8 --in-place -a "$FILE"
git add "$FILE"
fi
done

- name: Format modified C++ files
env:
filter: ${{ github.event.before }}
run: |
for FILE in $(git diff --name-only $filter | grep -E '.*\.(cc|cpp|h|hpp)$')
do
# Check if the file still exists in the working tree
if [ -f "$FILE" ]; then
clang-format -i -style=file $FILE
git add $FILE
fi
done

- name: Commit and push changes
run: |
HAS_CHANGES=$(git diff --staged --name-only)
if [ ${#HAS_CHANGES} -gt 0 ]; then
git config --global user.name github-actions[bot]
git config --global user.email "github-actions[bot]@users.noreply.github.com"
# Commit changes
git commit -m '[Automated Commit] Format Codebase'
# Use the GITHUB_TOKEN to push changes
git push
fi
34 changes: 34 additions & 0 deletions .github/workflows/publish.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# This is a basic workflow to help you get started with Actions

name: Publish site

on:
release:
types: [published]
push:
branches:
- master
- docs
- dev


jobs:

publish:
name: Publish the site
runs-on: ubuntu-latest

steps:
- name: Checkout repository normally
uses: actions/checkout@v3

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.11"

- name: Install Mkdocs
run: pip install -r docs/requirements.txt

- name: Run Mkdocs deploy
run: mkdocs gh-deploy --force
41 changes: 41 additions & 0 deletions .github/workflows/reset-branch.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: Reset Current Branch to Upstream After Squash Merge
on:
workflow_dispatch:
inputs:
branch:
description: 'Branch to reset (leave blank for current branch)'
required: false
default: 'dev'

jobs:
reset-branch:
runs-on: ubuntu-latest

steps:
- name: Checkout Repository
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Detect Current Branch
if: ${{ inputs.branch == '' }}
run: echo "branch=$(git rev-parse --abbrev-ref HEAD)" >> $GITHUB_ENV

- name: Use Input Branch
if: ${{ inputs.branch != '' }}
run: echo "branch=${{ inputs.branch }}" >> $GITHUB_ENV

- name: Add Upstream Remote
run: |
git remote add upstream https://github.com/mlcommons/inference.git
git fetch upstream
- name: Reset Branch to Upstream
run: |
git checkout ${{ env.branch }}
git reset --hard upstream/${{ env.branch }}
if: success()

- name: Force Push to Origin
run: |
git push origin ${{ env.branch }} --force-with-lease
if: success()
Loading