Live evaluation queue

How to read this page

• Cycles. The worker runs an evaluation cycle about once an hour. A cycle evaluates up to 8 models per hour across tracks on one fresh 300-item draw per track; the champion is one of them. New entries wait for the next cycle, in the order of their on-chain commitment.
• Status. queued: waits for its first draw. waiting for repo: the Hugging Face repo or revision is not reachable yet. waiting for base model: the declared base model is not available here yet. evaluating: the worker fetched it and the draw is running. pending confirmation: beat the champion on the first draw and needs a second, fresh draw after the time shown. not better, rejected: finished.
• Numbers. S_eff is the draw score after the generalisation penalty (Evaluation v3); the penalty itself is not shown because, with the raw score, it would reveal the score on the private items. The first draw passes when the gain over the champion clears the required margin. Scores of different draws are not directly comparable; only the champion and challengers on the same draw are.
• Times are estimates. A track waits while the cycle budget is used by the other track or when the worker pauses for an update. Items, draw seeds and the scores on the private items are never published.

Rules and leaderboards: Competitions, Demand forecasting, Risk assessment. Data file: https://reax.co/data/miners/eval-live.json.