feat: complete M015 reference implementation with KPI, datasets, and edge-case test vectors#62
Conversation
…edge-case test vectors Add missing files to bring M015 (Contribution-Weighted Rewards) to parity with other mechanism implementations: - reference-impl/m015_kpi.js: KPI computation module with Gini coefficient, top earner share, stability utilization, and security invariant checks - reference-impl/README.md: documentation for score and KPI modules - datasets/schema.json: JSON Schema for M015 dataset format - datasets/README.md: dataset format documentation - 3 new test vector pairs covering early exit penalty, stability cap overflow, and zero activity edge cases - Updated m015_score.js self-test to iterate all test vectors All 4 test vectors pass in both m015_score.js and m015_kpi.js. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the M015 contribution-weighted rewards mechanism by completing its reference implementation. It introduces a dedicated module for Key Performance Indicator (KPI) computation, provides thorough documentation for both the scoring and KPI modules, and establishes a formal JSON schema for datasets. Furthermore, it expands the test suite with critical edge-case scenarios, ensuring the mechanism's accuracy and reliability across diverse conditions and bringing it to full parity with other mechanism implementations. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the m015 contribution-weighted rewards mechanism, including comprehensive documentation, a JSON schema for replay datasets, and a JavaScript reference implementation. The reference implementation provides modules for activity scoring, stability allocation, distribution, and KPI computation, along with an updated self-test harness that now iterates through various test vectors. Feedback indicates a documentation inconsistency in datasets/README.md regarding a referenced fixture file that is not included in the PR. Additionally, the m015_kpi.js self-test has misleading comments, redundant participant score validation, and lacks assertions for several key KPIs, which should be addressed for improved test focus and completeness.
| ## Files | ||
| - `schema.json` -- JSON schema for replay datasets | ||
| - `fixtures/v0_sample.json` -- single distribution period with 4 participants and 1 stability commitment | ||
| - `fixtures/v0_stability_sample.json` -- stability tier scenarios (committed, matured, early exit, cap overflow) |
There was a problem hiding this comment.
This line references fixtures/v0_stability_sample.json, but this file is not included in the pull request. To avoid confusion, please either add the file or update the documentation to indicate that it is planned for a future update.
| // Check that distribution rewards match | ||
| const scoredParticipants = input.participants.map((p) => { | ||
| const result = computeActivityScore({ activities: p.activities }); | ||
| return { address: p.address, ...result }; | ||
| }); | ||
|
|
||
| for (const exp of expected.participant_scores) { | ||
| const actual = scoredParticipants.find((p) => p.address === exp.address); | ||
| if (!actual) { | ||
| console.error(` FAIL: participant ${exp.address} not found`); | ||
| vectorFailed = true; | ||
| continue; | ||
| } | ||
| if (Math.abs(actual.total_score - exp.total_score) > 0.001) { | ||
| console.error(` FAIL: ${exp.address} score expected ${exp.total_score}, got ${actual.total_score}`); | ||
| vectorFailed = true; | ||
| } | ||
| } |
There was a problem hiding this comment.
The comment on line 185, "Check that distribution rewards match", is misleading as the following code block validates participant scores, not distribution rewards.
More importantly, this self-test is re-validating participant scores, which is already covered in the m015_score.js self-test. This introduces redundancy.
Additionally, the test is missing assertions for several key KPIs computed by computeM015KPI, such as gini_coefficient, top_earner_share, and stability_utilization. These values are logged but not validated.
To improve the test's focus and completeness:
- Remove the redundant participant score validation logic (lines 186-202).
- Update the
*.expected.jsontest vectors to include the expected values for all KPIs generated bycomputeM015KPI. - Add assertions to validate all fields in the returned KPI object against the expected values.
…remove redundant validation - Add assertions for gini_coefficient, top_earner_share, stability_utilization, total_distributed_uregen, and participant_count in m015_kpi.js self-test - Fix comment "Check that distribution rewards match" which actually validated participant scores, not distribution rewards — replaced with accurate description - Remove redundant participant score re-validation (already covered by m015_score.js self-test) — KPI self-test now focuses on KPI-level outputs only - Add expected KPI values to all 4 test vector .expected.json files Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 participant
Summary
m015_kpi.jsKPI computation module with Gini coefficient, top earner share, stability utilization, and revenue/cap constraint checksreference-impl/README.mddocumenting both score and KPI modules, inputs/outputs, and test vectorsdatasets/schema.json(JSON Schema for M015 dataset format) anddatasets/README.mdm015_score.jsself-test to iterate all test vectors instead of only the sampleAll 4 test vectors pass in both
m015_score.jsandm015_kpi.js. This brings M015 to parity with other mechanism implementations (M012 etc.) in terms of file structure and coverage.Test plan
node mechanisms/m015-contribution-weighted-rewards/reference-impl/m015_score.js-- 4 vectors passnode mechanisms/m015-contribution-weighted-rewards/reference-impl/m015_kpi.js-- 4 vectors passdatasets/schema.jsonvalidatesdatasets/fixtures/v0_sample.json🤖 Generated with Claude Code