Demand forecasting

What the task is

Every item is a typed decision: a state (the facts, as text or JSON), a question and options. The model answers in one forward pass; the validator reads the probabilities of the option letters and scores them. The three primitives are choice (pick one option), noul (a yes/no judgement with a probability) and score (an ordinal rating). Items are fresh every round and sealed, so memorising the public data does not help.

Score S = 0.75 × competence + 0.25 × calibration, from 0 to 100. Competence is chance-corrected correctness; calibration rewards honest probabilities and punishes confident wrong answers.

One real example (public dev item)

DEMAND SIGNAL DEFINITIONS v2.6
'Sustained Interest' = at least 2 consecutive qualifying weeks after the baseline window.
Baseline = average of the first 4 weeks whose feed is OK. Later weeks qualify when the index is more than 20% above the baseline.
Any week with an index above 3x the baseline is treated as a bot spike: ignored, neither counts nor interrupts a run.
An OUTAGE week interrupts a run (the run restarts from zero) and does not count.

Series for Rugged Speaker
week,interest_index,feed
W01,96,OK
W02,92,OK
W03,n/a,OUTAGE
W04,99,OK
W05,101,OK
W06,101,OK
W07,97,OK
W08,99,OK
W09,342,OK
W10,95,OK
W11,148,OK

Region: Ghent; channel: search

Baseline to beat

The reference is the s1-fast model (Plumb-4B, the model inference miners serve) on the sealed pool. A submission must reach the entry floor, the baseline plus 1.5 points, to be ranked at all.

Scores run from 0 to 100. A champion appears in the chart once a submission has been confirmed.

Data

Public train and dev splits, licence CC-BY-4.0. For demand_forecasting that is 4,058 train items and 513 dev items, in demand_forecasting/train.jsonl and demand_forecasting/dev.jsonl. The sealed split used for ranking stays private.

huggingface-cli download decisionmodels/sn92-specialists --repo-type dataset --local-dir sn92-specialists

• decisionmodels/sn92-specialists: training and dev data for all 20 categories.
• decisionmodels/reax-sn92-network: public network data.

How to submit

The fastest route is the agent box above. By hand:

1. Prerequisites. A hotkey registered on netuid 92 (a separate UID if you also serve inference), a Hugging Face account, and a GPU for training and the local check (about 25 GB free, Docker with the NVIDIA runtime). A track needs no serving hardware and no traffic.
2. Get code and data. git clone https://github.com/reaxlabs/reax-subnet, install it with pip install -e '.[spec]', and download the dataset (above).
3. Fine-tune an allowed base model on demand_forecasting/train.jsonl, with the fixed prompt and the standard turn layout of the model family. Train the option-letter distribution and keep probabilities calibrated.
4. Check locally on the public dev split with the validator's own evaluator:

   reax-spec-eval ./my-model --dev ./sn92-specialists --category demand_forecasting --hotkey <hotkey ss58> --no-manifest-check

   The number is a preview, not a ranking.
5. Prepare the repo: *.safetensors only, config.json, tokenizer files, a LICENSE file and this manifest as reax-specialist.json:

   {
    "schema": "reax-specialist/1",
    "category": "demand_forecasting",
    "hotkey": "<hotkey ss58>",
    "base_model": "<hf id>@<revision>",
    "licence": "apache-2.0"
   }

   Then run the check again without --no-manifest-check.
6. Upload to Hugging Face and note the full 40-character commit hash of the revision.
7. Commit on chain with the hotkey. The text is

   reax-spec/1:demand_forecasting:<hf_repo>@<40-hex revision>

   at most 128 bytes, lower-case hash, never a branch name.
8. Make the repo public within 7,200 blocks (about 24 hours) of the commitment and keep it public.

Never share a recovery phrase, private key or coldkey file with anyone, including your agent.

Rules

Leaderboard

Pending means scored once, confirmed means it also cleared the second draw, champion is the current best. Validators run about 6 model evaluations per hour in commitment order, so a queue can take a while.

Payout mechanics

• The track has 15% of miner emission. Inside it, rank 1 gets 75%, rank 2 17% and rank 3 8%. Ranks 2 and 3 are the most recent former champions that are still committed.
• A challenger must beat the incumbent by max(1.5 points, 2 standard errors) on a fresh draw, then beat it again by at least half that margin on a second disjoint draw at least 12 hours later.
• A submission below the entry floor earns nothing from the track; an unclaimed share goes back to inference miners.

Rewards are a protocol mechanism, set by validators from benchmark results. They are variable and can be zero. Rules can change; any change is announced in advance. Full rules: SPECIALIST-TRACKS.md · all competitions · miner leaderboard.