Risk assessment

What the task is

Every item is a typed decision: a state (the facts, as text or JSON), a question and options. The model answers in one forward pass; the validator reads the probabilities of the option letters and scores them. The three primitives are choice (pick one option), noul (a yes/no judgement with a probability) and score (an ordinal rating). Items are fresh every round and sealed, so memorising the public data does not help.

Score S = 0.75 × competence + 0.25 × calibration, from 0 to 100. Competence is chance-corrected correctness; calibration rewards honest probabilities and punishes confident wrong answers.

One real example (public dev item)

"Ref","IL","II","Ctrl","Eff","TestDT","Stat"
"CR-99","3","4","KYC/AML STP Auto-Check","75%","2022-01-10","OD"

Appetite Limit: Residual score (L x I) must be <= 8.

Control Validation Rule: Effectiveness percentages are applied to reduce inherent likelihood only when the control has a passing test status. If the test is overdue (OD), the effectiveness rating is voided and 0% reduction applies.

Baseline to beat

The reference is the s1-fast model (Plumb-4B, the model inference miners serve) on the sealed pool. A submission must reach the entry floor, the baseline plus 1.5 points, to be ranked at all.

Scores run from 0 to 100. A champion appears in the chart once a submission has been confirmed.

Data

Public train and dev splits, licence CC-BY-4.0. For risk_assessment that is 3,049 train items and 449 dev items, in risk_assessment/train.jsonl and risk_assessment/dev.jsonl. The sealed split used for ranking stays private.

huggingface-cli download decisionmodels/sn92-specialists --repo-type dataset --local-dir sn92-specialists

• decisionmodels/sn92-specialists: training and dev data for all 20 categories.
• decisionmodels/reax-sn92-network: public network data.

How to submit

The fastest route is the agent box above. By hand:

1. Prerequisites. A hotkey registered on netuid 92 (a separate UID if you also serve inference), a Hugging Face account, and a GPU for training and the local check (about 25 GB free, Docker with the NVIDIA runtime). A track needs no serving hardware and no traffic.
2. Get code and data. git clone https://github.com/reaxlabs/reax-subnet, install it with pip install -e '.[spec]', and download the dataset (above).
3. Fine-tune an allowed base model on risk_assessment/train.jsonl, with the fixed prompt and the standard turn layout of the model family. Train the option-letter distribution and keep probabilities calibrated.
4. Check locally on the public dev split with the validator's own evaluator:

   reax-spec-eval ./my-model --dev ./sn92-specialists --category risk_assessment --hotkey <hotkey ss58> --no-manifest-check

   The number is a preview, not a ranking.
5. Prepare the repo: *.safetensors only, config.json, tokenizer files, a LICENSE file and this manifest as reax-specialist.json:

   {
    "schema": "reax-specialist/1",
    "category": "risk_assessment",
    "hotkey": "<hotkey ss58>",
    "base_model": "<hf id>@<revision>",
    "licence": "apache-2.0"
   }

   Then run the check again without --no-manifest-check.
6. Upload to Hugging Face and note the full 40-character commit hash of the revision.
7. Commit on chain with the hotkey. The text is

   reax-spec/1:risk_assessment:<hf_repo>@<40-hex revision>

   at most 128 bytes, lower-case hash, never a branch name.
8. Make the repo public within 7,200 blocks (about 24 hours) of the commitment and keep it public.

Never share a recovery phrase, private key or coldkey file with anyone, including your agent.

Rules

Leaderboard

Pending means scored once, confirmed means it also cleared the second draw, champion is the current best. Validators run about 6 model evaluations per hour in commitment order, so a queue can take a while.

Payout mechanics

• The track has 15% of miner emission. Inside it, rank 1 gets 75%, rank 2 17% and rank 3 8%. Ranks 2 and 3 are the most recent former champions that are still committed.
• A challenger must beat the incumbent by max(1.5 points, 2 standard errors) on a fresh draw, then beat it again by at least half that margin on a second disjoint draw at least 12 hours later.
• A submission below the entry floor earns nothing from the track; an unclaimed share goes back to inference miners.

Rewards are a protocol mechanism, set by validators from benchmark results. They are variable and can be zero. Rules can change; any change is announced in advance. Full rules: SPECIALIST-TRACKS.md · all competitions · miner leaderboard.