Known Issues

[xingxing21]

README.md Inaccuracies

1. Wrong CLI binary name throughout

All examples in README use df-cli but the actual compiled binary is data-cli (package name: data-cli).

Affects: Usage section, TL;DR, Installation, all command examples.

# README says:
df-cli monitor

# Correct:
data-cli monitor

2. Wrong submit flag name

README uses --harness but the CLI defines --code.

# README says:
df-cli submit --harness ./my-harness/

# Correct:
data-cli submit --code ./my-harness/ --hotkey "..." --endpoint http://...

3. --hotkey is not a global flag

README shows --hotkey as a top-level flag before the subcommand, but it only exists on the submit subcommand.

# README says (invalid):
df-cli --hotkey 5GrwvaEF... monitor

# --hotkey is only valid on submit:
data-cli submit --hotkey "..." --code ./my-harness/

4. --rpc-url global flag does not exist

README shows df-cli --rpc-url http://localhost:8080 monitor but no such flag exists on the root Cli struct. The --endpoint flag exists per-subcommand (submit, status) but not on monitor.

# README says (invalid):
df-cli --rpc-url http://localhost:8080 monitor

# monitor has no endpoint flag — it only launches the TUI:
data-cli monitor

5. Architecture section lists non-existent files

README architecture tree includes cli/src/app.rs and cli/src/rpc.rs, neither of which exists in the codebase.

6. src/lib.rs described incorrectly

README describes the root src/lib.rs as a "HuggingFace dataset handler". It is an empty placeholder with no implementation.

7. status command uses wrong flag

README shows df-cli status --hotkey 5Abc... but the status subcommand takes --id (a submission ID), not --hotkey.

# README says (invalid):
df-cli status --hotkey 5Abc...

# Correct:
data-cli status --id <submission-id> --endpoint http://...

---

Code Bugs Fixed

8. CLI submit URL was wrong

cli/src/submit.rs was posting to /challenge/data-fabrication/submit which does not exist on the server. The platform SDK mounts /evaluate as the submission endpoint.

Fixed: URL changed to POST /evaluate.

9. CLI submit request body had wrong shape

The CLI was sending {name, code} but the SDK's /evaluate handler deserializes into EvaluationRequest {request_id, submission_id, participant_id, data, epoch}. The server's evaluate() then deserializes data into Submission {hotkey, epoch, code_hash, package}.

Fixed: CLI now builds the correct two-layer envelope:

{
  "request_id": "<uuid>",
  "submission_id": "<code_hash>",
  "participant_id": "<hotkey_hex>",
  "data": {
    "hotkey": "<hotkey_hex>",
    "epoch": 0,
    "code_hash": "<sha256>",
    "package": [...]
  },
  "epoch": 0
}

10. data-fabrication-server had no logging

The server emitted no logs on startup, validation, or evaluation. tracing was already a dependency but unused in lib.rs and main.rs.

Fixed: Added structured tracing log calls covering:

• Startup (version, challenge ID)
• Each validate() call (hotkey, epoch, package size, pass/fail, warnings)
• Each evaluate() call (hotkey, epoch, package size, code hash, rejection reasons)
• Each handle_route() call (method, path, 404s)

Control log verbosity with RUST_LOG=debug data-fabrication-server.

---

Known Limitations (not bugs)

11. data-fabrication-server evaluation is a stub

server/src/lib.rs evaluate() always returns score 0.0 with message "Evaluation accepted - full implementation pending". Real scoring runs inside the WASM module loaded by production Platform validators, not the native server.

12. Harness output format mismatch

The example harness (my-harness/generate.py) outputs {prompt, response, metadata} per line. The actual validator's WASM module parses ConversationEntry which expects:

{"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

Submissions using the {prompt, response} format will fail format validation on a real validator.

13. data-cli status hits a non-existent route

cli/src/status.rs queries GET /submission/:id which is not mounted by the SDK server. The server only exposes /health, /config, /evaluate, /validate, and the custom routes declared by handle_route() (/leaderboard, /stats).