Problem
For V2, validators should not rely on a central API to fetch scoring questions. The dataset should be public, versioned, and reproducible across validators. A central dataset service adds unnecessary dependency to the scoring path.
Dataset quality also matters. The goal is not only to have good questions, but to have questions that are actually answerable with fresh, real-time information. This is especially important for Subnet 22, where miners are expected to retrieve current information rather than answer static knowledge prompts.
Proposal
Use Hugging Face as the public source of dataset distribution and versioning. Validators should download the dataset, cache it locally, and use the same dataset version during scoring.
Question selection should be deterministic across validators. At the start of each scoring window, validators publish short seed fragments using knowledge commitments. They then derive the shared window seed and use it to deterministically choose questions from the dataset.
The dataset should be maintained separately by search type. Web Search can use existing public datasets as a bootstrap source, while X Search and AI Search need more careful curation. X Search questions should be realistically answerable through X search filters and operators. AI Search questions should focus on multi-source, real-time answering rather than generic web-search prompts.
Organic logs should help improve the dataset, but they should not be added directly into scoring. Logged queries need filtering first, since some are too long, malformed, low quality, speculative, or not realistically answerable with current data.
Scope
This issue covers moving dataset distribution to Hugging Face, validator-side download and caching, deterministic question selection from knowledge-commitment-derived seeds, search-type-specific dataset structure, and removal of the central dataset API from scoring flow.
Acceptance criteria
- Validators can load a versioned public dataset from Hugging Face and cache it locally.
- Validators no longer depend on a central API to retrieve scoring questions.
- Validators use knowledge commitments to derive a shared seed and deterministically choose questions for each scoring window.
- The dataset is structured separately for Web Search, X Search, and AI Search.
- Organic logs are only promoted into the scoring dataset through a filtered process.
Problem
For V2, validators should not rely on a central API to fetch scoring questions. The dataset should be public, versioned, and reproducible across validators. A central dataset service adds unnecessary dependency to the scoring path.
Dataset quality also matters. The goal is not only to have good questions, but to have questions that are actually answerable with fresh, real-time information. This is especially important for Subnet 22, where miners are expected to retrieve current information rather than answer static knowledge prompts.
Proposal
Use Hugging Face as the public source of dataset distribution and versioning. Validators should download the dataset, cache it locally, and use the same dataset version during scoring.
Question selection should be deterministic across validators. At the start of each scoring window, validators publish short seed fragments using knowledge commitments. They then derive the shared window seed and use it to deterministically choose questions from the dataset.
The dataset should be maintained separately by search type. Web Search can use existing public datasets as a bootstrap source, while X Search and AI Search need more careful curation. X Search questions should be realistically answerable through X search filters and operators. AI Search questions should focus on multi-source, real-time answering rather than generic web-search prompts.
Organic logs should help improve the dataset, but they should not be added directly into scoring. Logged queries need filtering first, since some are too long, malformed, low quality, speculative, or not realistically answerable with current data.
Scope
This issue covers moving dataset distribution to Hugging Face, validator-side download and caching, deterministic question selection from knowledge-commitment-derived seeds, search-type-specific dataset structure, and removal of the central dataset API from scoring flow.
Acceptance criteria