Hugging Face's logo

doofz
/
THX-01

Text Classification
Safetensors
thx01
decision-model
calibration
reinforcement-learning
proper-scoring-rules
routing
extraction
citation
azerbaijani
thx-01
Eval Results
Model card Files
xet
 Community
2
doofz commited on
Commit
bef2e24
·
verified ·
1 Parent(s): f47cc50

THX-01 1.0.0: THX architecture name, NOTICE, demos

Browse files
Files changed (10) hide show
  1. NOTICE +3 -5
  2. PYPI.md +0 -7
  3. README.md +0 -7
  4. bench/v10_eval.py +1 -1
  5. config.json +4 -4
  6. doom/index.html +4 -4
  7. results/decision_index/benchmark-summary.json +0 -1425
  8. results/decision_index/index.json +0 -506
  9. results/decision_index/scores.json +0 -1428
  10. snake/index.html +4 -4
NOTICE CHANGED
@@ -1,7 +1,5 @@
1
  THX-01
2
- Copyright 2026 HAL-X
3
 
4
- This product includes software and model weights derived from works licensed under the
5
- Apache License, Version 2.0 (Laya, Copyright Convai Innovations), and the mmBERT encoder
6
- (jhu-clsp/mmBERT-base). Post-training data, post-trained weights, the batched runtime,
7
- tournament decoding, the REST server, the Doom demo and all benchmarks are by HAL-X.
 
1
  THX-01
2
+ Copyright 2026 HAL-X AI
3
 
4
+ This product includes the mmBERT encoder (jhu-clsp/mmBERT-base). Post-training data, post-trained weights, the batched
5
+ runtime, tournament decoding, the REST server, the Doom demo and all benchmarks are by HAL-X AI.
 
 
PYPI.md CHANGED
@@ -271,13 +271,6 @@ The state may be a string or an object with a `document` field; the question tex
271
 
272
  ## Limitations
273
 
274
- THX-01 is a 322M-parameter encoder built for business decisions: classification, routing, triage, guardrails, and
275
- locating numbers and passages in documents. It has no generative world model and does not reason step by step.
276
-
277
- - It is not a general reasoning model. On the public Decision Index 0.2.1 suite (150,246 requests, run with the suite's
278
- own kit, HLE not included) THX-01 scores **10.6**, with knowledge and reasoning benchmarks (MMLU, GPQA, ARC, HellaSwag,
279
- GSM8K, BBH) at chance level. Use a large language model for questions that need world knowledge, mathematics or code.
280
- Full per-benchmark results: [`results/decision_index/`](https://huggingface.co/doofz/THX-01/tree/main/results/decision_index).
281
  - Fine-grained intent sets with many near-synonymous labels remain hard: Banking77 (77 intents) reaches 65.9%.
282
  - Emotion with six overlapping classes (DAIR Emotion) reaches 44.6%; the model reports correspondingly low confidence.
283
  - `number` and `excerpt` locate text that is present in the document; they do not perform arithmetic, unit conversion or
 
271
 
272
  ## Limitations
273
 
 
 
 
 
 
 
 
274
  - Fine-grained intent sets with many near-synonymous labels remain hard: Banking77 (77 intents) reaches 65.9%.
275
  - Emotion with six overlapping classes (DAIR Emotion) reaches 44.6%; the model reports correspondingly low confidence.
276
  - `number` and `excerpt` locate text that is present in the document; they do not perform arithmetic, unit conversion or
README.md CHANGED
@@ -280,13 +280,6 @@ The state may be a string or an object with a `document` field; the question tex
280
 
281
  ## Limitations
282
 
283
- THX-01 is a 322M-parameter encoder built for business decisions: classification, routing, triage, guardrails, and
284
- locating numbers and passages in documents. It has no generative world model and does not reason step by step.
285
-
286
- - It is not a general reasoning model. On the public Decision Index 0.2.1 suite (150,246 requests, run with the suite's
287
- own kit, HLE not included) THX-01 scores **10.6**, with knowledge and reasoning benchmarks (MMLU, GPQA, ARC, HellaSwag,
288
- GSM8K, BBH) at chance level. Use a large language model for questions that need world knowledge, mathematics or code.
289
- Full per-benchmark results: `results/decision_index/`.
290
  - Fine-grained intent sets with many near-synonymous labels remain hard: Banking77 (77 intents) reaches 65.9%.
291
  - Emotion with six overlapping classes (DAIR Emotion) reaches 44.6%; the model reports correspondingly low confidence.
292
  - `number` and `excerpt` locate text that is present in the document; they do not perform arithmetic, unit conversion or
 
280
 
281
  ## Limitations
282
 
 
 
 
 
 
 
 
283
  - Fine-grained intent sets with many near-synonymous labels remain hard: Banking77 (77 intents) reaches 65.9%.
284
  - Emotion with six overlapping classes (DAIR Emotion) reaches 44.6%; the model reports correspondingly low confidence.
285
  - `number` and `excerpt` locate text that is present in the document; they do not perform arithmetic, unit conversion or
bench/v10_eval.py CHANGED
@@ -25,7 +25,7 @@ def thx_backend(base):
25
  def ask(items):
26
  out = []
27
  for i in range(0, len(items), 64):
28
- r = c.post(base + "/v1/systemone/batch", headers={"Authorization": "Bearer " + KEY["SYSTEM01_API_KEY"]},
29
  json={"items": [{"state": s, "questions": q} for s, q in items[i:i + 64]]}).json()
30
  out += [x["answers"] for x in r["results"]]
31
  return out
 
25
  def ask(items):
26
  out = []
27
  for i in range(0, len(items), 64):
28
+ r = c.post(base + "/v1/systemone/batch", headers={"Authorization": "Bearer " + KEY["THX01_API_KEY"]},
29
  json={"items": [{"state": s, "questions": q} for s, q in items[i:i + 64]]}).json()
30
  out += [x["answers"] for x in r["results"]]
31
  return out
config.json CHANGED
@@ -1,9 +1,9 @@
1
  {
2
- "model_type": "system01",
3
  "architectures": [
4
- "System01DecisionModel"
5
  ],
6
- "library": "system01",
7
  "encoder": "jhu-clsp/mmBERT-base",
8
  "head_layers": 2,
9
  "max_len": 1024,
@@ -13,4 +13,4 @@
13
  "score"
14
  ],
15
  "decision_config": "rl_agent_config.json"
16
- }
 
1
  {
2
+ "model_type": "thx01",
3
  "architectures": [
4
+ "THXDecisionModel"
5
  ],
6
+ "library": "thx01",
7
  "encoder": "jhu-clsp/mmBERT-base",
8
  "head_layers": 2,
9
  "max_len": 1024,
 
13
  "score"
14
  ],
15
  "decision_config": "rl_agent_config.json"
16
+ }
doom/index.html CHANGED
@@ -3,7 +3,7 @@
3
  <head>
4
  <meta charset="utf-8">
5
  <meta name="viewport" content="width=device-width, initial-scale=1">
6
- <title>System01 plays Doom</title>
7
  <style>
8
  :root {
9
  --bg: #0e0e0d; --panel: #171716; --line: #2a2a28; --ink: #f4f3ee; --ink2: #b7b5ab; --muted: #7d7b73;
@@ -51,14 +51,14 @@
51
  </head>
52
  <body>
53
  <header>
54
- <h1>System01 plays Doom</h1>
55
  <span class="sub">no vision · no text generation · two typed questions answered in one forward pass per step</span>
56
  <span class="sp"></span>
57
  <span class="scen" id="scen"></span>
58
  </header>
59
  <main>
60
  <section>
61
- <div class="game"><img src="/stream.mjpg" alt="Live Doom game played by System01"><span class="tag" id="scenTag">—</span></div>
62
  <div class="row">
63
  <div class="stat"><div class="k">Key pressed</div><div class="v" id="key" style="font-size:16px">–</div></div>
64
  <div class="stat"><div class="k">Latency / decision</div><div class="v" id="lat">–</div></div>
@@ -90,7 +90,7 @@
90
  </div>
91
  </aside>
92
  </main>
93
- <footer>Key pressed = ATTACK when P(crosshair on enemy) ≥ 0.5, otherwise the top choice · System01 · RTX 4090 · output tokens: 0 · no Doom data in training — the model reads the state text, the rules and the option names.</footer>
94
  <script>
95
  const ORDER = ["ATTACK","TURN_LEFT","TURN_RIGHT","MOVE_FORWARD","MOVE_LEFT","MOVE_RIGHT","MOVE_BACKWARD"];
96
  const SCEN = ["defend_the_center","deadly_corridor","defend_the_line","basic"];
 
3
  <head>
4
  <meta charset="utf-8">
5
  <meta name="viewport" content="width=device-width, initial-scale=1">
6
+ <title>THX-01 plays Doom</title>
7
  <style>
8
  :root {
9
  --bg: #0e0e0d; --panel: #171716; --line: #2a2a28; --ink: #f4f3ee; --ink2: #b7b5ab; --muted: #7d7b73;
 
51
  </head>
52
  <body>
53
  <header>
54
+ <h1>THX-01 plays Doom</h1>
55
  <span class="sub">no vision · no text generation · two typed questions answered in one forward pass per step</span>
56
  <span class="sp"></span>
57
  <span class="scen" id="scen"></span>
58
  </header>
59
  <main>
60
  <section>
61
+ <div class="game"><img src="/stream.mjpg" alt="Live Doom game played by THX-01"><span class="tag" id="scenTag">—</span></div>
62
  <div class="row">
63
  <div class="stat"><div class="k">Key pressed</div><div class="v" id="key" style="font-size:16px">–</div></div>
64
  <div class="stat"><div class="k">Latency / decision</div><div class="v" id="lat">–</div></div>
 
90
  </div>
91
  </aside>
92
  </main>
93
+ <footer>Key pressed = ATTACK when P(crosshair on enemy) ≥ 0.5, otherwise the top choice · THX-01 · RTX 4090 · output tokens: 0 · no Doom data in training — the model reads the state text, the rules and the option names.</footer>
94
  <script>
95
  const ORDER = ["ATTACK","TURN_LEFT","TURN_RIGHT","MOVE_FORWARD","MOVE_LEFT","MOVE_RIGHT","MOVE_BACKWARD"];
96
  const SCEN = ["defend_the_center","deadly_corridor","defend_the_line","basic"];
results/decision_index/benchmark-summary.json DELETED
@@ -1,1425 +0,0 @@
1
- {
2
- "engine": "thx01",
3
- "counts": {
4
- "ok": 150246
5
- },
6
- "successful_request_latency_ms": {
7
- "median": 26.845910935662687,
8
- "p95": 169.2261470016092,
9
- "mean": 51.947523622932074
10
- },
11
- "benchmarks": [
12
- {
13
- "catalog_id": 1,
14
- "dataset": "BFCL",
15
- "requests": 1694,
16
- "answered": 1694,
17
- "unsupported": 0,
18
- "errors": 0,
19
- "abstained": 0,
20
- "pending": 0,
21
- "scored_requests": 1694,
22
- "metric": "case exact accuracy",
23
- "score": 0.38488783943329397,
24
- "reference_same_cases": null,
25
- "median_ms": 121.45544704981148,
26
- "detail": {
27
- "field_accuracy": 0.35927013422818793,
28
- "case_exact_accuracy": 0.38488783943329397,
29
- "custom_metrics": {},
30
- "positive_micro_f1": 0.5239208352812841
31
- }
32
- },
33
- {
34
- "catalog_id": 2,
35
- "dataset": "ToolRet",
36
- "requests": 685,
37
- "answered": 685,
38
- "unsupported": 0,
39
- "errors": 0,
40
- "abstained": 0,
41
- "pending": 0,
42
- "scored_requests": 685,
43
- "metric": "nDCG@10",
44
- "score": 0.324200342897765,
45
- "reference_same_cases": null,
46
- "median_ms": 338.77906086854637,
47
- "detail": {
48
- "field_accuracy": null,
49
- "case_exact_accuracy": null,
50
- "custom_metrics": {
51
- "ndcg_at_10": 0.324200342897765,
52
- "mrr": 0.32936583261921787,
53
- "recall_at_10": 0.5099252693778241,
54
- "candidate_recall": 0.8401355578727842,
55
- "scorable_candidate_recall": 0.8401355578727842,
56
- "candidates_scored": 31.9985401459854,
57
- "candidates_retrieved": 32,
58
- "bm25_ndcg_at_10": 0.5430661225020075
59
- }
60
- }
61
- },
62
- {
63
- "catalog_id": 3,
64
- "dataset": "API-Bank",
65
- "requests": 508,
66
- "answered": 508,
67
- "unsupported": 0,
68
- "errors": 0,
69
- "abstained": 0,
70
- "pending": 0,
71
- "scored_requests": 508,
72
- "metric": "accuracy",
73
- "score": 0.42913385826771655,
74
- "reference_same_cases": null,
75
- "median_ms": 234.53721939586103,
76
- "detail": {
77
- "field_accuracy": 0.42913385826771655,
78
- "case_exact_accuracy": 0.42913385826771655,
79
- "custom_metrics": {}
80
- }
81
- },
82
- {
83
- "catalog_id": 4,
84
- "dataset": "BANKING77",
85
- "requests": 3080,
86
- "answered": 3080,
87
- "unsupported": 0,
88
- "errors": 0,
89
- "abstained": 0,
90
- "pending": 0,
91
- "scored_requests": 3080,
92
- "metric": "macro-F1",
93
- "score": 0.5766682740779784,
94
- "reference_same_cases": null,
95
- "median_ms": 64.00237395428121,
96
- "detail": {
97
- "field_accuracy": 0.5805194805194805,
98
- "case_exact_accuracy": 0.5805194805194805,
99
- "macro_f1": 0.5766682740779784,
100
- "custom_metrics": {}
101
- }
102
- },
103
- {
104
- "catalog_id": 5,
105
- "dataset": "CLINC150+OOS",
106
- "requests": 5500,
107
- "answered": 5500,
108
- "unsupported": 0,
109
- "errors": 0,
110
- "abstained": 0,
111
- "pending": 0,
112
- "scored_requests": 5500,
113
- "metric": "macro-F1",
114
- "score": 0.6715865319260599,
115
- "reference_same_cases": null,
116
- "median_ms": 95.62165406532586,
117
- "detail": {
118
- "field_accuracy": 0.6083636363636363,
119
- "case_exact_accuracy": 0.6083636363636363,
120
- "macro_f1": 0.6715865319260599,
121
- "custom_metrics": {}
122
- }
123
- },
124
- {
125
- "catalog_id": 6,
126
- "dataset": "RouterBench",
127
- "requests": 10000,
128
- "answered": 10000,
129
- "unsupported": 0,
130
- "errors": 0,
131
- "abstained": 0,
132
- "pending": 0,
133
- "scored_requests": 10000,
134
- "metric": "selected quality (quality objective)",
135
- "score": 0.56812,
136
- "reference_same_cases": null,
137
- "median_ms": 79.34242649935186,
138
- "tracks": {
139
- "RouterBench-0shot": {
140
- "metric": "selected quality (quality objective)",
141
- "score": 0.5445932440535679,
142
- "scored_requests": 5003
143
- },
144
- "RouterBench-5shot": {
145
- "metric": "selected quality (quality objective)",
146
- "score": 0.5916750050030019,
147
- "scored_requests": 4997
148
- }
149
- },
150
- "detail": {
151
- "field_accuracy": null,
152
- "case_exact_accuracy": null,
153
- "custom_metrics": {
154
- "quality_quality": 0.56812,
155
- "quality_cost_usd": 0.0008766388728,
156
- "quality_utility": 0.56812,
157
- "quality_oracle_optimal": 0.5935,
158
- "quality_utility_regret": 0.35094,
159
- "cost_aware_quality": 0.59414,
160
- "cost_aware_cost_usd": 0.000866311654,
161
- "cost_aware_utility": 0.58547688346,
162
- "cost_aware_oracle_optimal": 0.1439,
163
- "cost_aware_utility_regret": 0.330052459168
164
- }
165
- }
166
- },
167
- {
168
- "catalog_id": 9,
169
- "dataset": "Home appliance simulator",
170
- "requests": 88,
171
- "answered": 88,
172
- "unsupported": 0,
173
- "errors": 0,
174
- "abstained": 0,
175
- "pending": 0,
176
- "scored_requests": 88,
177
- "metric": "case exact accuracy",
178
- "score": 0,
179
- "reference_same_cases": null,
180
- "median_ms": 185.2301029721275,
181
- "detail": {
182
- "field_accuracy": 0.6849056603773584,
183
- "case_exact_accuracy": 0,
184
- "custom_metrics": {},
185
- "subgroups": {
186
- "outcome": {
187
- "n": 88,
188
- "mean": 0.13636363636363635
189
- },
190
- "post_0": {
191
- "n": 69,
192
- "mean": 0.08695652173913043
193
- },
194
- "post_1": {
195
- "n": 25,
196
- "mean": 0.16
197
- },
198
- "resolution": {
199
- "n": 88,
200
- "mean": 0.03409090909090909
201
- },
202
- "state_read": {
203
- "n": 88,
204
- "mean": 0.6136363636363636
205
- },
206
- "target_member": {
207
- "n": 1232,
208
- "mean": 0.8198051948051948
209
- }
210
- }
211
- }
212
- },
213
- {
214
- "catalog_id": 10,
215
- "dataset": "SGD/SGD-X",
216
- "requests": 2500,
217
- "answered": 2500,
218
- "unsupported": 0,
219
- "errors": 0,
220
- "abstained": 0,
221
- "pending": 0,
222
- "scored_requests": 2500,
223
- "metric": "macro-F1",
224
- "score": 0.508498753933838,
225
- "reference_same_cases": null,
226
- "median_ms": 51.02665442973375,
227
- "detail": {
228
- "field_accuracy": 0.5192,
229
- "case_exact_accuracy": 0.5192,
230
- "macro_f1": 0.508498753933838,
231
- "custom_metrics": {}
232
- }
233
- },
234
- {
235
- "catalog_id": 11,
236
- "dataset": "ContractNLI",
237
- "requests": 123,
238
- "answered": 123,
239
- "unsupported": 0,
240
- "errors": 0,
241
- "abstained": 0,
242
- "pending": 0,
243
- "scored_requests": 123,
244
- "metric": "macro-F1",
245
- "score": 0.24766866504459276,
246
- "reference_same_cases": null,
247
- "median_ms": 246.63001485168934,
248
- "detail": {
249
- "field_accuracy": 0.41224294595887134,
250
- "case_exact_accuracy": 0,
251
- "macro_f1": 0.24766866504459276,
252
- "custom_metrics": {}
253
- }
254
- },
255
- {
256
- "catalog_id": 12,
257
- "dataset": "ANLI",
258
- "requests": 3200,
259
- "answered": 3200,
260
- "unsupported": 0,
261
- "errors": 0,
262
- "abstained": 0,
263
- "pending": 0,
264
- "scored_requests": 3200,
265
- "metric": "macro-F1",
266
- "score": 0.39232867385891407,
267
- "reference_same_cases": null,
268
- "median_ms": 20.80285653937608,
269
- "detail": {
270
- "field_accuracy": 0.4059375,
271
- "case_exact_accuracy": 0.4059375,
272
- "macro_f1": 0.39232867385891407,
273
- "custom_metrics": {}
274
- }
275
- },
276
- {
277
- "catalog_id": 20,
278
- "dataset": "BPoMP",
279
- "requests": 5000,
280
- "answered": 5000,
281
- "unsupported": 0,
282
- "errors": 0,
283
- "abstained": 0,
284
- "pending": 0,
285
- "scored_requests": 5000,
286
- "metric": "accuracy",
287
- "score": 0.4924,
288
- "reference_same_cases": null,
289
- "median_ms": 20.98024997394532,
290
- "detail": {
291
- "field_accuracy": 0.4924,
292
- "case_exact_accuracy": 0.2601726263871763,
293
- "custom_metrics": {},
294
- "subgroups": {
295
- "beam search": {
296
- "n": 401,
297
- "mean": 0.5336658354114713
298
- },
299
- "corrupted rhyme": {
300
- "n": 376,
301
- "mean": 0.5132978723404256
302
- },
303
- "delayed beam search": {
304
- "n": 137,
305
- "mean": 0.39416058394160586
306
- },
307
- "delete any 1 nonrhyming word": {
308
- "n": 376,
309
- "mean": 0.5132978723404256
310
- },
311
- "delete any 1 rhyming EOL word": {
312
- "n": 376,
313
- "mean": 0.4787234042553192
314
- },
315
- "delete any 1 word": {
316
- "n": 376,
317
- "mean": 0.47074468085106386
318
- },
319
- "delete any 2 nonrhyming words": {
320
- "n": 376,
321
- "mean": 0.5159574468085106
322
- },
323
- "delete any 2 rhyming EOL words": {
324
- "n": 376,
325
- "mean": 0.47074468085106386
326
- },
327
- "delete any 2 words": {
328
- "n": 376,
329
- "mean": 0.5053191489361702
330
- },
331
- "delete any 3 nonrhyming words": {
332
- "n": 376,
333
- "mean": 0.5053191489361702
334
- },
335
- "delete any 3 rhyming words": {
336
- "n": 376,
337
- "mean": 0.5079787234042553
338
- },
339
- "delete any 3 words": {
340
- "n": 376,
341
- "mean": 0.48138297872340424
342
- },
343
- "shuffled rhyming lines": {
344
- "n": 336,
345
- "mean": 0.4732142857142857
346
- },
347
- "shuffled rhyming words": {
348
- "n": 366,
349
- "mean": 0.46174863387978143
350
- }
351
- }
352
- }
353
- },
354
- {
355
- "catalog_id": 21,
356
- "dataset": "Humicroedit",
357
- "requests": 2628,
358
- "answered": 2628,
359
- "unsupported": 0,
360
- "errors": 0,
361
- "abstained": 0,
362
- "pending": 0,
363
- "scored_requests": 2628,
364
- "metric": "accuracy",
365
- "score": 0.4927701674277017,
366
- "reference_same_cases": null,
367
- "median_ms": 19.66916397213936,
368
- "detail": {
369
- "field_accuracy": 0.4927701674277017,
370
- "case_exact_accuracy": 0.4927701674277017,
371
- "custom_metrics": {}
372
- }
373
- },
374
- {
375
- "catalog_id": 22,
376
- "dataset": "POP909-CL",
377
- "requests": 2000,
378
- "answered": 2000,
379
- "unsupported": 0,
380
- "errors": 0,
381
- "abstained": 0,
382
- "pending": 0,
383
- "scored_requests": 2000,
384
- "metric": "accuracy",
385
- "score": 0.0015,
386
- "reference_same_cases": null,
387
- "median_ms": 283.798721851781,
388
- "detail": {
389
- "field_accuracy": 0.0015,
390
- "case_exact_accuracy": 0.0015,
391
- "cluster_macro_accuracy": 0.001936026936026936,
392
- "custom_metrics": {},
393
- "subgroups": {
394
- "A#7": {
395
- "n": 1,
396
- "mean": 0
397
- },
398
- "A#M": {
399
- "n": 88,
400
- "mean": 0
401
- },
402
- "A#m": {
403
- "n": 43,
404
- "mean": 0
405
- },
406
- "A#m7": {
407
- "n": 7,
408
- "mean": 0
409
- },
410
- "A#maj7": {
411
- "n": 3,
412
- "mean": 0
413
- },
414
- "A7": {
415
- "n": 4,
416
- "mean": 0
417
- },
418
- "AM": {
419
- "n": 73,
420
- "mean": 0
421
- },
422
- "Am": {
423
- "n": 69,
424
- "mean": 0
425
- },
426
- "Am7": {
427
- "n": 8,
428
- "mean": 0
429
- },
430
- "Amaj7": {
431
- "n": 3,
432
- "mean": 0
433
- },
434
- "B7": {
435
- "n": 2,
436
- "mean": 0
437
- },
438
- "BM": {
439
- "n": 88,
440
- "mean": 0
441
- },
442
- "Bdim": {
443
- "n": 1,
444
- "mean": 0
445
- },
446
- "Bhalf-dim7": {
447
- "n": 1,
448
- "mean": 0
449
- },
450
- "Bm": {
451
- "n": 65,
452
- "mean": 0
453
- },
454
- "Bm7": {
455
- "n": 6,
456
- "mean": 0
457
- },
458
- "Bmaj7": {
459
- "n": 3,
460
- "mean": 0
461
- },
462
- "C#7": {
463
- "n": 1,
464
- "mean": 0
465
- },
466
- "C#M": {
467
- "n": 78,
468
- "mean": 0
469
- },
470
- "C#aug / Faug / Aaug": {
471
- "n": 1,
472
- "mean": 0
473
- },
474
- "C#dim": {
475
- "n": 1,
476
- "mean": 0
477
- },
478
- "C#m": {
479
- "n": 51,
480
- "mean": 0
481
- },
482
- "C#m7": {
483
- "n": 8,
484
- "mean": 0
485
- },
486
- "C#maj7": {
487
- "n": 6,
488
- "mean": 0
489
- },
490
- "C#sus2 / G#sus4": {
491
- "n": 4,
492
- "mean": 0
493
- },
494
- "C#sus4 / F#sus2": {
495
- "n": 6,
496
- "mean": 0
497
- },
498
- "C7": {
499
- "n": 2,
500
- "mean": 0
501
- },
502
- "CM": {
503
- "n": 99,
504
- "mean": 0
505
- },
506
- "Cdim": {
507
- "n": 1,
508
- "mean": 0
509
- },
510
- "Cm": {
511
- "n": 67,
512
- "mean": 0
513
- },
514
- "Cm7": {
515
- "n": 4,
516
- "mean": 0
517
- },
518
- "Cmaj7": {
519
- "n": 1,
520
- "mean": 0
521
- },
522
- "Csus2 / Gsus4": {
523
- "n": 9,
524
- "mean": 0
525
- },
526
- "Csus4 / Fsus2": {
527
- "n": 6,
528
- "mean": 0
529
- },
530
- "D#7": {
531
- "n": 7,
532
- "mean": 0
533
- },
534
- "D#M": {
535
- "n": 86,
536
- "mean": 0
537
- },
538
- "D#aug / Gaug / Baug": {
539
- "n": 1,
540
- "mean": 0
541
- },
542
- "D#dim": {
543
- "n": 1,
544
- "mean": 0
545
- },
546
- "D#m": {
547
- "n": 43,
548
- "mean": 0
549
- },
550
- "D#m7": {
551
- "n": 3,
552
- "mean": 0
553
- },
554
- "D#maj7": {
555
- "n": 6,
556
- "mean": 0
557
- },
558
- "D#sus2 / A#sus4": {
559
- "n": 7,
560
- "mean": 0
561
- },
562
- "D#sus4 / G#sus2": {
563
- "n": 4,
564
- "mean": 0
565
- },
566
- "D7": {
567
- "n": 5,
568
- "mean": 0
569
- },
570
- "DM": {
571
- "n": 77,
572
- "mean": 0
573
- },
574
- "Dm": {
575
- "n": 64,
576
- "mean": 0
577
- },
578
- "Dm7": {
579
- "n": 8,
580
- "mean": 0
581
- },
582
- "DmMaj7": {
583
- "n": 1,
584
- "mean": 0
585
- },
586
- "Dmaj7": {
587
- "n": 3,
588
- "mean": 0
589
- },
590
- "Dsus2 / Asus4": {
591
- "n": 12,
592
- "mean": 0
593
- },
594
- "Dsus4 / Gsus2": {
595
- "n": 5,
596
- "mean": 0
597
- },
598
- "E7": {
599
- "n": 2,
600
- "mean": 0
601
- },
602
- "EM": {
603
- "n": 80,
604
- "mean": 0
605
- },
606
- "Edim": {
607
- "n": 1,
608
- "mean": 0
609
- },
610
- "Em": {
611
- "n": 74,
612
- "mean": 0
613
- },
614
- "Em7": {
615
- "n": 6,
616
- "mean": 0
617
- },
618
- "Emaj7": {
619
- "n": 1,
620
- "mean": 0
621
- },
622
- "Esus2 / Bsus4": {
623
- "n": 6,
624
- "mean": 0
625
- },
626
- "Esus4 / Asus2": {
627
- "n": 8,
628
- "mean": 0
629
- },
630
- "F#7": {
631
- "n": 2,
632
- "mean": 0
633
- },
634
- "F#M": {
635
- "n": 62,
636
- "mean": 0
637
- },
638
- "F#dim": {
639
- "n": 1,
640
- "mean": 0
641
- },
642
- "F#half-dim7": {
643
- "n": 1,
644
- "mean": 0
645
- },
646
- "F#m": {
647
- "n": 58,
648
- "mean": 0
649
- },
650
- "F#m7": {
651
- "n": 9,
652
- "mean": 0
653
- },
654
- "F#mMaj7": {
655
- "n": 1,
656
- "mean": 0
657
- },
658
- "F#maj7": {
659
- "n": 4,
660
- "mean": 0
661
- },
662
- "F#sus4 / Bsus2": {
663
- "n": 5,
664
- "mean": 0
665
- },
666
- "F7": {
667
- "n": 2,
668
- "mean": 0
669
- },
670
- "FM": {
671
- "n": 93,
672
- "mean": 0
673
- },
674
- "Fm": {
675
- "n": 55,
676
- "mean": 0
677
- },
678
- "Fm7": {
679
- "n": 7,
680
- "mean": 0
681
- },
682
- "Fmaj7": {
683
- "n": 4,
684
- "mean": 0
685
- },
686
- "Fsus4 / A#sus2": {
687
- "n": 9,
688
- "mean": 0
689
- },
690
- "G#7": {
691
- "n": 6,
692
- "mean": 0
693
- },
694
- "G#M": {
695
- "n": 88,
696
- "mean": 0
697
- },
698
- "G#m": {
699
- "n": 50,
700
- "mean": 0
701
- },
702
- "G#m7": {
703
- "n": 4,
704
- "mean": 0
705
- },
706
- "G#maj7": {
707
- "n": 6,
708
- "mean": 0
709
- },
710
- "G7": {
711
- "n": 2,
712
- "mean": 0
713
- },
714
- "GM": {
715
- "n": 102,
716
- "mean": 0
717
- },
718
- "Gdim": {
719
- "n": 1,
720
- "mean": 0
721
- },
722
- "Gm": {
723
- "n": 66,
724
- "mean": 0
725
- },
726
- "Gm7": {
727
- "n": 7,
728
- "mean": 0
729
- },
730
- "Gmaj7": {
731
- "n": 2,
732
- "mean": 0
733
- },
734
- "NoChord": {
735
- "n": 29,
736
- "mean": 0
737
- },
738
- "Other": {
739
- "n": 3,
740
- "mean": 1
741
- }
742
- }
743
- }
744
- },
745
- {
746
- "catalog_id": 23,
747
- "dataset": "cfcolor",
748
- "requests": 5000,
749
- "answered": 5000,
750
- "unsupported": 0,
751
- "errors": 0,
752
- "abstained": 0,
753
- "pending": 0,
754
- "scored_requests": 5000,
755
- "metric": "accuracy",
756
- "score": 0.4794,
757
- "reference_same_cases": null,
758
- "median_ms": 23.61105615273118,
759
- "detail": {
760
- "field_accuracy": 0.4794,
761
- "case_exact_accuracy": 0.4794,
762
- "cluster_macro_accuracy": 0.49865244052586905,
763
- "custom_metrics": {}
764
- }
765
- },
766
- {
767
- "catalog_id": 24,
768
- "dataset": "MMLU",
769
- "requests": 14033,
770
- "answered": 14033,
771
- "unsupported": 0,
772
- "errors": 0,
773
- "abstained": 0,
774
- "pending": 0,
775
- "scored_requests": 14033,
776
- "metric": "accuracy",
777
- "score": 0.2727855768545571,
778
- "reference_same_cases": null,
779
- "median_ms": 20.243939943611622,
780
- "detail": {
781
- "field_accuracy": 0.2727855768545571,
782
- "case_exact_accuracy": 0.2727855768545571,
783
- "custom_metrics": {}
784
- }
785
- },
786
- {
787
- "catalog_id": 25,
788
- "dataset": "GPQA Diamond",
789
- "requests": 196,
790
- "answered": 196,
791
- "unsupported": 0,
792
- "errors": 0,
793
- "abstained": 0,
794
- "pending": 0,
795
- "scored_requests": 196,
796
- "metric": "accuracy",
797
- "score": 0.23469387755102042,
798
- "reference_same_cases": null,
799
- "median_ms": 19.5722229545936,
800
- "detail": {
801
- "field_accuracy": 0.23469387755102042,
802
- "case_exact_accuracy": 0.23469387755102042,
803
- "custom_metrics": {}
804
- }
805
- },
806
- {
807
- "catalog_id": 26,
808
- "dataset": "ARC-Easy",
809
- "requests": 2376,
810
- "answered": 2376,
811
- "unsupported": 0,
812
- "errors": 0,
813
- "abstained": 0,
814
- "pending": 0,
815
- "scored_requests": 2376,
816
- "metric": "accuracy",
817
- "score": 0.40824915824915825,
818
- "reference_same_cases": null,
819
- "median_ms": 20.53705498110503,
820
- "detail": {
821
- "field_accuracy": 0.40824915824915825,
822
- "case_exact_accuracy": 0.40824915824915825,
823
- "custom_metrics": {}
824
- }
825
- },
826
- {
827
- "catalog_id": 27,
828
- "dataset": "ARC-Challenge",
829
- "requests": 1172,
830
- "answered": 1172,
831
- "unsupported": 0,
832
- "errors": 0,
833
- "abstained": 0,
834
- "pending": 0,
835
- "scored_requests": 1172,
836
- "metric": "accuracy",
837
- "score": 0.2713310580204778,
838
- "reference_same_cases": null,
839
- "median_ms": 21.03443897794932,
840
- "detail": {
841
- "field_accuracy": 0.2713310580204778,
842
- "case_exact_accuracy": 0.2713310580204778,
843
- "custom_metrics": {}
844
- }
845
- },
846
- {
847
- "catalog_id": 28,
848
- "dataset": "WinoGrande",
849
- "requests": 1267,
850
- "answered": 1267,
851
- "unsupported": 0,
852
- "errors": 0,
853
- "abstained": 0,
854
- "pending": 0,
855
- "scored_requests": 1267,
856
- "metric": "accuracy",
857
- "score": 0.5130228887134964,
858
- "reference_same_cases": null,
859
- "median_ms": 19.667087122797966,
860
- "detail": {
861
- "field_accuracy": 0.5130228887134964,
862
- "case_exact_accuracy": 0.5130228887134964,
863
- "custom_metrics": {}
864
- }
865
- },
866
- {
867
- "catalog_id": 29,
868
- "dataset": "HellaSwag",
869
- "requests": 10042,
870
- "answered": 10042,
871
- "unsupported": 0,
872
- "errors": 0,
873
- "abstained": 0,
874
- "pending": 0,
875
- "scored_requests": 10042,
876
- "metric": "accuracy",
877
- "score": 0.2909778928500299,
878
- "reference_same_cases": null,
879
- "median_ms": 21.53277595061809,
880
- "detail": {
881
- "field_accuracy": 0.2909778928500299,
882
- "case_exact_accuracy": 0.2909778928500299,
883
- "custom_metrics": {}
884
- }
885
- },
886
- {
887
- "catalog_id": 30,
888
- "dataset": "GSM8K",
889
- "requests": 2638,
890
- "answered": 2638,
891
- "unsupported": 0,
892
- "errors": 0,
893
- "abstained": 0,
894
- "pending": 0,
895
- "scored_requests": 2638,
896
- "metric": "accuracy",
897
- "score": 0.10803639120545869,
898
- "reference_same_cases": null,
899
- "median_ms": 21.464205579832196,
900
- "tracks": {
901
- "GSM8K-10choice": {
902
- "metric": "accuracy",
903
- "score": 0.043214556482183475,
904
- "scored_requests": 1319
905
- },
906
- "GSM8K-4choice": {
907
- "metric": "accuracy",
908
- "score": 0.1728582259287339,
909
- "scored_requests": 1319
910
- }
911
- },
912
- "detail": {
913
- "field_accuracy": 0.10803639120545869,
914
- "case_exact_accuracy": 0.02122820318423048,
915
- "custom_metrics": {}
916
- }
917
- },
918
- {
919
- "catalog_id": 31,
920
- "dataset": "ChessBench",
921
- "requests": 5000,
922
- "answered": 5000,
923
- "unsupported": 0,
924
- "errors": 0,
925
- "abstained": 0,
926
- "pending": 0,
927
- "scored_requests": 5000,
928
- "metric": "accuracy",
929
- "score": 0.0888,
930
- "reference_same_cases": null,
931
- "median_ms": 62.117030494846404,
932
- "detail": {
933
- "field_accuracy": 0.0888,
934
- "case_exact_accuracy": 0.0888,
935
- "custom_metrics": {
936
- "value_regret": 0.1940995647560189
937
- }
938
- }
939
- },
940
- {
941
- "catalog_id": 32,
942
- "dataset": "MuSR",
943
- "requests": 752,
944
- "answered": 752,
945
- "unsupported": 0,
946
- "errors": 0,
947
- "abstained": 0,
948
- "pending": 0,
949
- "scored_requests": 752,
950
- "metric": "accuracy",
951
- "score": 0.37101063829787234,
952
- "reference_same_cases": null,
953
- "median_ms": 51.88089143484831,
954
- "detail": {
955
- "field_accuracy": 0.37101063829787234,
956
- "case_exact_accuracy": 0.37101063829787234,
957
- "custom_metrics": {}
958
- }
959
- },
960
- {
961
- "catalog_id": 33,
962
- "dataset": "SATA-Bench",
963
- "requests": 1650,
964
- "answered": 1650,
965
- "unsupported": 0,
966
- "errors": 0,
967
- "abstained": 0,
968
- "pending": 0,
969
- "scored_requests": 1650,
970
- "metric": "case exact accuracy",
971
- "score": 0.005454545454545455,
972
- "reference_same_cases": null,
973
- "median_ms": 202.8815409867093,
974
- "detail": {
975
- "field_accuracy": 0.47238512599084875,
976
- "case_exact_accuracy": 0.005454545454545455,
977
- "custom_metrics": {},
978
- "positive_micro_f1": 0.5733937783335938
979
- }
980
- },
981
- {
982
- "catalog_id": 34,
983
- "dataset": "SimpleBench",
984
- "requests": 10,
985
- "answered": 10,
986
- "unsupported": 0,
987
- "errors": 0,
988
- "abstained": 0,
989
- "pending": 0,
990
- "scored_requests": 10,
991
- "metric": "accuracy",
992
- "score": 0.2,
993
- "reference_same_cases": null,
994
- "median_ms": 398.93044938798994,
995
- "detail": {
996
- "field_accuracy": 0.2,
997
- "case_exact_accuracy": 0.2,
998
- "custom_metrics": {}
999
- }
1000
- },
1001
- {
1002
- "catalog_id": 36,
1003
- "dataset": "BRIGHT",
1004
- "requests": 220,
1005
- "answered": 220,
1006
- "unsupported": 0,
1007
- "errors": 0,
1008
- "abstained": 0,
1009
- "pending": 0,
1010
- "scored_requests": 220,
1011
- "metric": "nDCG@10",
1012
- "score": 0.13219727065760292,
1013
- "reference_same_cases": null,
1014
- "median_ms": 390.8218175638467,
1015
- "detail": {
1016
- "field_accuracy": null,
1017
- "case_exact_accuracy": null,
1018
- "custom_metrics": {
1019
- "ndcg_at_10": 0.13219727065760292,
1020
- "mrr": 0.18970302711015216,
1021
- "recall_at_10": 0.19709139163115386,
1022
- "candidate_recall": 0.5445170600851817,
1023
- "scorable_candidate_recall": 0.5445170600851817,
1024
- "candidates_scored": 32,
1025
- "candidates_retrieved": 32,
1026
- "bm25_ndcg_at_10": 0.2761244257399229
1027
- }
1028
- }
1029
- },
1030
- {
1031
- "catalog_id": 37,
1032
- "dataset": "Amazon ESCI",
1033
- "requests": 5000,
1034
- "answered": 5000,
1035
- "unsupported": 0,
1036
- "errors": 0,
1037
- "abstained": 0,
1038
- "pending": 0,
1039
- "scored_requests": 5000,
1040
- "metric": "macro-F1",
1041
- "score": 0.2719711634365288,
1042
- "reference_same_cases": null,
1043
- "median_ms": 37.29241050314158,
1044
- "detail": {
1045
- "field_accuracy": 0.4506,
1046
- "case_exact_accuracy": 0.4506,
1047
- "macro_f1": 0.2719711634365288,
1048
- "custom_metrics": {}
1049
- }
1050
- },
1051
- {
1052
- "catalog_id": 38,
1053
- "dataset": "ACOS",
1054
- "requests": 1565,
1055
- "answered": 1565,
1056
- "unsupported": 0,
1057
- "errors": 0,
1058
- "abstained": 0,
1059
- "pending": 0,
1060
- "scored_requests": 1565,
1061
- "metric": "per-review F1",
1062
- "score": 0.03106206357482472,
1063
- "reference_same_cases": null,
1064
- "median_ms": 211.83185908012092,
1065
- "detail": {
1066
- "field_accuracy": 0.038258024853993766,
1067
- "case_exact_accuracy": 0,
1068
- "custom_metrics": {},
1069
- "positive_micro_f1": 0.011352736054529244
1070
- }
1071
- },
1072
- {
1073
- "catalog_id": 39,
1074
- "dataset": "FinEntity",
1075
- "requests": 979,
1076
- "answered": 979,
1077
- "unsupported": 0,
1078
- "errors": 0,
1079
- "abstained": 0,
1080
- "pending": 0,
1081
- "scored_requests": 979,
1082
- "metric": "macro-F1",
1083
- "score": 0.5997414798643262,
1084
- "reference_same_cases": null,
1085
- "median_ms": 24.536699056625366,
1086
- "detail": {
1087
- "field_accuracy": 0.6533583842179427,
1088
- "case_exact_accuracy": 0.550561797752809,
1089
- "macro_f1": 0.5997414798643262,
1090
- "custom_metrics": {}
1091
- }
1092
- },
1093
- {
1094
- "catalog_id": 40,
1095
- "dataset": "iSarcasmEval",
1096
- "requests": 4600,
1097
- "answered": 4600,
1098
- "unsupported": 0,
1099
- "errors": 0,
1100
- "abstained": 0,
1101
- "pending": 0,
1102
- "scored_requests": 4600,
1103
- "metric": "see separate subtask tracks",
1104
- "score": null,
1105
- "reference_same_cases": null,
1106
- "median_ms": 22.290082648396492,
1107
- "tracks": {
1108
- "iSarcasmEval-A-Ar": {
1109
- "metric": "see separate subtask tracks",
1110
- "score": null,
1111
- "scored_requests": 1400
1112
- },
1113
- "iSarcasmEval-A-En": {
1114
- "metric": "see separate subtask tracks",
1115
- "score": null,
1116
- "scored_requests": 1400
1117
- },
1118
- "iSarcasmEval-B-En": {
1119
- "metric": "see separate subtask tracks",
1120
- "score": null,
1121
- "scored_requests": 1400
1122
- },
1123
- "iSarcasmEval-C-Ar": {
1124
- "metric": "see separate subtask tracks",
1125
- "score": null,
1126
- "scored_requests": 200
1127
- },
1128
- "iSarcasmEval-C-En": {
1129
- "metric": "see separate subtask tracks",
1130
- "score": null,
1131
- "scored_requests": 200
1132
- }
1133
- },
1134
- "detail": {
1135
- "field_accuracy": 0.7002586206896552,
1136
- "case_exact_accuracy": 0.5406521739130434,
1137
- "macro_f1": 0.3610221062920707,
1138
- "custom_metrics": {},
1139
- "positive_f1_by_field": {
1140
- "sarcastic": 0.2578754578754579,
1141
- "sarcasm": 0.28869047619047616,
1142
- "irony": 0.008350730688935281,
1143
- "satire": 0.04155844155844156,
1144
- "understatement": 0.0,
1145
- "overstatement": 0.029585798816568046,
1146
- "rhetorical_question": 0.0392156862745098
1147
- },
1148
- "category_macro_f1": 0.09503951305776982
1149
- }
1150
- },
1151
- {
1152
- "catalog_id": 41,
1153
- "dataset": "VAST",
1154
- "requests": 3006,
1155
- "answered": 3006,
1156
- "unsupported": 0,
1157
- "errors": 0,
1158
- "abstained": 0,
1159
- "pending": 0,
1160
- "scored_requests": 3006,
1161
- "metric": "macro-F1",
1162
- "score": 0.3439452167186462,
1163
- "reference_same_cases": null,
1164
- "median_ms": 20.197101053781807,
1165
- "detail": {
1166
- "field_accuracy": 0.3852295409181637,
1167
- "case_exact_accuracy": 0.3852295409181637,
1168
- "macro_f1": 0.3439452167186462,
1169
- "custom_metrics": {}
1170
- }
1171
- },
1172
- {
1173
- "catalog_id": 42,
1174
- "dataset": "NLI4CT",
1175
- "requests": 5500,
1176
- "answered": 5500,
1177
- "unsupported": 0,
1178
- "errors": 0,
1179
- "abstained": 0,
1180
- "pending": 0,
1181
- "scored_requests": 5500,
1182
- "metric": "macro-F1",
1183
- "score": 0.45260271469971447,
1184
- "reference_same_cases": null,
1185
- "median_ms": 39.66621856670827,
1186
- "detail": {
1187
- "field_accuracy": 0.4536363636363636,
1188
- "case_exact_accuracy": 0.4536363636363636,
1189
- "macro_f1": 0.45260271469971447,
1190
- "custom_metrics": {}
1191
- }
1192
- },
1193
- {
1194
- "catalog_id": 43,
1195
- "dataset": "CRUXEval",
1196
- "requests": 570,
1197
- "answered": 570,
1198
- "unsupported": 0,
1199
- "errors": 0,
1200
- "abstained": 0,
1201
- "pending": 0,
1202
- "scored_requests": 570,
1203
- "metric": "accuracy",
1204
- "score": 0.44912280701754387,
1205
- "reference_same_cases": null,
1206
- "median_ms": 23.366250563412905,
1207
- "detail": {
1208
- "field_accuracy": 0.44912280701754387,
1209
- "case_exact_accuracy": 0.44912280701754387,
1210
- "custom_metrics": {}
1211
- }
1212
- },
1213
- {
1214
- "catalog_id": 44,
1215
- "dataset": "CLadder",
1216
- "requests": 5000,
1217
- "answered": 5000,
1218
- "unsupported": 0,
1219
- "errors": 0,
1220
- "abstained": 0,
1221
- "pending": 0,
1222
- "scored_requests": 5000,
1223
- "metric": "accuracy",
1224
- "score": 0.4988,
1225
- "reference_same_cases": null,
1226
- "median_ms": 19.946140586398542,
1227
- "detail": {
1228
- "field_accuracy": 0.4988,
1229
- "case_exact_accuracy": 0.4988,
1230
- "custom_metrics": {}
1231
- }
1232
- },
1233
- {
1234
- "catalog_id": 48,
1235
- "dataset": "ForecastBench",
1236
- "requests": 10139,
1237
- "answered": 10139,
1238
- "unsupported": 0,
1239
- "errors": 0,
1240
- "abstained": 0,
1241
- "pending": 0,
1242
- "scored_requests": 10139,
1243
- "metric": "Brier (lower is better)",
1244
- "score": 0.4411365806943486,
1245
- "reference_same_cases": null,
1246
- "median_ms": 42.0946329832077,
1247
- "detail": {
1248
- "field_accuracy": null,
1249
- "case_exact_accuracy": null,
1250
- "custom_metrics": {
1251
- "brier": 0.4411365806943486,
1252
- "log_loss": 1.2082081705104082,
1253
- "accuracy": 0.3527961337410001
1254
- },
1255
- "subgroups": {
1256
- "joint": {
1257
- "n": 3242,
1258
- "mean": 0.5354781387877853
1259
- },
1260
- "scalar": {
1261
- "n": 6897,
1262
- "mean": 0.3967904401493403
1263
- }
1264
- }
1265
- }
1266
- },
1267
- {
1268
- "catalog_id": 50,
1269
- "dataset": "Habermas Machine",
1270
- "requests": 1676,
1271
- "answered": 1676,
1272
- "unsupported": 0,
1273
- "errors": 0,
1274
- "abstained": 0,
1275
- "pending": 0,
1276
- "scored_requests": 1676,
1277
- "metric": "accuracy",
1278
- "score": 0.31861575178997614,
1279
- "reference_same_cases": null,
1280
- "median_ms": 40.231638005934656,
1281
- "detail": {
1282
- "field_accuracy": 0.31861575178997614,
1283
- "case_exact_accuracy": 0.31861575178997614,
1284
- "custom_metrics": {
1285
- "normalized_rank_regret": 0.2711515513126492
1286
- }
1287
- }
1288
- },
1289
- {
1290
- "catalog_id": 56,
1291
- "dataset": "PhishNChips phishing decisions",
1292
- "requests": 2000,
1293
- "answered": 2000,
1294
- "unsupported": 0,
1295
- "errors": 0,
1296
- "abstained": 0,
1297
- "pending": 0,
1298
- "metric": "accuracy",
1299
- "score": 0.535,
1300
- "median_ms": 82.68452051561326,
1301
- "scored_requests": 2000,
1302
- "detail": {
1303
- "field_accuracy": 0.535,
1304
- "scored_fields": 2000,
1305
- "chance_on_rows": 0.5
1306
- }
1307
- },
1308
- {
1309
- "catalog_id": 57,
1310
- "dataset": "MMLU-Pro",
1311
- "requests": 12032,
1312
- "answered": 12032,
1313
- "unsupported": 0,
1314
- "errors": 0,
1315
- "abstained": 0,
1316
- "pending": 0,
1317
- "metric": "accuracy",
1318
- "score": 0.10314162234042554,
1319
- "median_ms": 22.42864842992276,
1320
- "scored_requests": 12032,
1321
- "detail": {
1322
- "field_accuracy": 0.10314162234042554,
1323
- "scored_fields": 12032,
1324
- "chance_on_rows": 0.11087694718844986
1325
- }
1326
- },
1327
- {
1328
- "catalog_id": 58,
1329
- "dataset": "BBH fixed-option tasks",
1330
- "requests": 5507,
1331
- "answered": 5507,
1332
- "unsupported": 0,
1333
- "errors": 0,
1334
- "abstained": 0,
1335
- "pending": 0,
1336
- "metric": "accuracy",
1337
- "score": 0.3244960958779735,
1338
- "median_ms": 21.44559402950108,
1339
- "scored_requests": 5507,
1340
- "detail": {
1341
- "field_accuracy": 0.3244960958779735,
1342
- "scored_fields": 5507,
1343
- "chance_on_rows": 0.31010949680343713
1344
- }
1345
- },
1346
- {
1347
- "catalog_id": 59,
1348
- "dataset": "RAGTruth response-level hallucination",
1349
- "requests": 2700,
1350
- "answered": 2700,
1351
- "unsupported": 0,
1352
- "errors": 0,
1353
- "abstained": 0,
1354
- "pending": 0,
1355
- "metric": "F1 on hallucinated class",
1356
- "score": 0.4989928057553957,
1357
- "median_ms": 46.31052748300135,
1358
- "scored_requests": 2700,
1359
- "detail": {
1360
- "field_accuracy": 0.35518518518518516,
1361
- "scored_fields": 2700,
1362
- "chance_on_rows": 0.41125163541212384
1363
- }
1364
- },
1365
- {
1366
- "catalog_id": 61,
1367
- "dataset": "HoVer claim verification",
1368
- "requests": 4000,
1369
- "answered": 4000,
1370
- "unsupported": 0,
1371
- "errors": 0,
1372
- "abstained": 0,
1373
- "pending": 0,
1374
- "metric": "accuracy",
1375
- "score": 0.5375,
1376
- "median_ms": 32.56508696358651,
1377
- "scored_requests": 4000,
1378
- "detail": {
1379
- "field_accuracy": 0.5375,
1380
- "scored_fields": 4000,
1381
- "chance_on_rows": 0.5
1382
- }
1383
- },
1384
- {
1385
- "catalog_id": 62,
1386
- "dataset": "When2Call MCQ",
1387
- "requests": 3652,
1388
- "answered": 3652,
1389
- "unsupported": 0,
1390
- "errors": 0,
1391
- "abstained": 0,
1392
- "pending": 0,
1393
- "metric": "accuracy",
1394
- "score": 0.14156626506024098,
1395
- "median_ms": 43.8731734175235,
1396
- "scored_requests": 3652,
1397
- "detail": {
1398
- "field_accuracy": 0.14156626506024098,
1399
- "scored_fields": 3652,
1400
- "chance_on_rows": 0.25
1401
- }
1402
- },
1403
- {
1404
- "catalog_id": 64,
1405
- "dataset": "New Yorker caption matching",
1406
- "requests": 528,
1407
- "answered": 528,
1408
- "unsupported": 0,
1409
- "errors": 0,
1410
- "abstained": 0,
1411
- "pending": 0,
1412
- "metric": "accuracy",
1413
- "score": 0.25757575757575757,
1414
- "median_ms": 41.93732305429876,
1415
- "scored_requests": 528,
1416
- "detail": {
1417
- "field_accuracy": 0.25757575757575757,
1418
- "scored_fields": 528,
1419
- "chance_on_rows": 0.2
1420
- }
1421
- }
1422
- ],
1423
- "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage.",
1424
- "edition": "0.2.1"
1425
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
results/decision_index/index.json DELETED
@@ -1,506 +0,0 @@
1
- {
2
- "edition": "0.2.1",
3
- "panel_id": "decision-index-0.2.1",
4
- "index": 10.63,
5
- "raw_index": 31.87,
6
- "scores": {
7
- "balanced_skill": 10.63,
8
- "balanced_raw": 31.87,
9
- "breadth_skill": 8.68
10
- },
11
- "areas": [
12
- {
13
- "id": "knowledge",
14
- "label": "Knowledge & Reasoning",
15
- "raw": 0.2144,
16
- "skill": 0.0147,
17
- "coverage": 1.0,
18
- "n": 10,
19
- "benchmarks": [
20
- 25,
21
- 30,
22
- 31,
23
- 32,
24
- 33,
25
- 43,
26
- 44,
27
- 45,
28
- 57,
29
- 58
30
- ]
31
- },
32
- {
33
- "id": "language",
34
- "label": "Language Understanding",
35
- "raw": 0.3667,
36
- "skill": 0.0663,
37
- "coverage": 1.0,
38
- "n": 10,
39
- "benchmarks": [
40
- 11,
41
- 12,
42
- 28,
43
- 29,
44
- 38,
45
- 39,
46
- 40,
47
- 41,
48
- 42,
49
- 59
50
- ]
51
- },
52
- {
53
- "id": "retrieval",
54
- "label": "Retrieval & Classification",
55
- "raw": 0.4547,
56
- "skill": 0.2641,
57
- "coverage": 1.0,
58
- "n": 6,
59
- "benchmarks": [
60
- 4,
61
- 5,
62
- 36,
63
- 37,
64
- 56,
65
- 61
66
- ]
67
- },
68
- {
69
- "id": "tools",
70
- "label": "Tools & Automation",
71
- "raw": 0.2671,
72
- "skill": 0.1713,
73
- "coverage": 1.0,
74
- "n": 5,
75
- "benchmarks": [
76
- 1,
77
- 2,
78
- 3,
79
- 9,
80
- 62
81
- ]
82
- },
83
- {
84
- "id": "arts",
85
- "label": "Arts & Human Taste",
86
- "raw": 0.2857,
87
- "skill": 0.0115,
88
- "coverage": 1.0,
89
- "n": 7,
90
- "benchmarks": [
91
- 20,
92
- 21,
93
- 22,
94
- 23,
95
- 48,
96
- 50,
97
- 64
98
- ]
99
- }
100
- ],
101
- "benchmarks": {
102
- "1": {
103
- "raw": 0.3849,
104
- "skill": 0.1696,
105
- "coverage": 1.0,
106
- "random": 0.2592,
107
- "rule": "track",
108
- "in_index": true,
109
- "tracks": []
110
- },
111
- "2": {
112
- "raw": 0.3242,
113
- "skill": 0.2196,
114
- "coverage": 1.0,
115
- "random": 0.1341,
116
- "rule": "track",
117
- "in_index": true,
118
- "tracks": []
119
- },
120
- "3": {
121
- "raw": 0.4291,
122
- "skill": 0.4181,
123
- "coverage": 1.0,
124
- "random": 0.0189,
125
- "rule": "chance",
126
- "in_index": true
127
- },
128
- "4": {
129
- "raw": 0.5767,
130
- "skill": 0.5713,
131
- "coverage": 1.0,
132
- "random": 0.0127,
133
- "rule": "chance",
134
- "in_index": true
135
- },
136
- "5": {
137
- "raw": 0.6716,
138
- "skill": 0.6696,
139
- "coverage": 1.0,
140
- "random": 0.006,
141
- "rule": "chance",
142
- "in_index": true
143
- },
144
- "9": {
145
- "raw": 0.0,
146
- "skill": 0.0,
147
- "coverage": 1.0,
148
- "random": 0.0,
149
- "rule": "chance",
150
- "in_index": true
151
- },
152
- "11": {
153
- "raw": 0.2477,
154
- "skill": 0.0,
155
- "coverage": 1.0,
156
- "random": 0.3085,
157
- "rule": "track",
158
- "in_index": true,
159
- "tracks": []
160
- },
161
- "12": {
162
- "raw": 0.3923,
163
- "skill": 0.0897,
164
- "coverage": 1.0,
165
- "random": 0.3324,
166
- "rule": "chance",
167
- "in_index": true
168
- },
169
- "20": {
170
- "raw": 0.4875,
171
- "skill": 0.0,
172
- "coverage": 1.0,
173
- "random": 0.5,
174
- "rule": "track",
175
- "in_index": true,
176
- "tracks": []
177
- },
178
- "21": {
179
- "raw": 0.4928,
180
- "skill": 0.0,
181
- "coverage": 1.0,
182
- "random": 0.5,
183
- "rule": "track",
184
- "in_index": true,
185
- "tracks": []
186
- },
187
- "22": {
188
- "raw": 0.0019,
189
- "skill": 0.0,
190
- "coverage": 1.0,
191
- "random": 0.0078,
192
- "rule": "track",
193
- "in_index": true,
194
- "tracks": []
195
- },
196
- "23": {
197
- "raw": 0.4987,
198
- "skill": 0.0,
199
- "coverage": 1.0,
200
- "random": 0.5,
201
- "rule": "track",
202
- "in_index": true,
203
- "tracks": []
204
- },
205
- "25": {
206
- "raw": 0.2347,
207
- "skill": 0.0,
208
- "coverage": 1.0,
209
- "random": 0.25,
210
- "rule": "track",
211
- "in_index": true,
212
- "tracks": []
213
- },
214
- "28": {
215
- "raw": 0.513,
216
- "skill": 0.026,
217
- "coverage": 1.0,
218
- "random": 0.5,
219
- "rule": "chance",
220
- "in_index": true
221
- },
222
- "29": {
223
- "raw": 0.291,
224
- "skill": 0.0547,
225
- "coverage": 1.0,
226
- "random": 0.25,
227
- "rule": "chance",
228
- "in_index": true
229
- },
230
- "30": {
231
- "raw": 0.108,
232
- "skill": 0.0,
233
- "coverage": 1.0,
234
- "random": 0.25,
235
- "rule": "track",
236
- "in_index": true,
237
- "tracks": [
238
- {
239
- "track": "GSM8K-4choice",
240
- "score": 0.1729,
241
- "headline": false
242
- },
243
- {
244
- "track": "GSM8K-10choice",
245
- "score": 0.0432,
246
- "headline": false
247
- }
248
- ]
249
- },
250
- "31": {
251
- "raw": 0.0888,
252
- "skill": 0.0076,
253
- "coverage": 1.0,
254
- "random": 0.0819,
255
- "rule": "track",
256
- "in_index": true,
257
- "tracks": []
258
- },
259
- "32": {
260
- "raw": 0.371,
261
- "skill": 0.0,
262
- "coverage": 1.0,
263
- "random": 0.371,
264
- "rule": "chance",
265
- "in_index": true
266
- },
267
- "33": {
268
- "raw": 0.0055,
269
- "skill": 0.0,
270
- "coverage": 1.0,
271
- "random": 0.0131,
272
- "rule": "chance",
273
- "in_index": true
274
- },
275
- "36": {
276
- "raw": 0.1322,
277
- "skill": 0.0183,
278
- "coverage": 1.0,
279
- "random": 0.116,
280
- "rule": "track",
281
- "in_index": true,
282
- "tracks": []
283
- },
284
- "37": {
285
- "raw": 0.272,
286
- "skill": 0.0868,
287
- "coverage": 1.0,
288
- "random": 0.2027,
289
- "rule": "track",
290
- "in_index": true,
291
- "tracks": []
292
- },
293
- "38": {
294
- "raw": 0.0311,
295
- "skill": 0.0001,
296
- "coverage": 1.0,
297
- "random": 0.031,
298
- "rule": "chance",
299
- "in_index": true
300
- },
301
- "39": {
302
- "raw": 0.5997,
303
- "skill": 0.4112,
304
- "coverage": 1.0,
305
- "random": 0.3201,
306
- "rule": "chance",
307
- "in_index": true
308
- },
309
- "40": {
310
- "raw": 0.2778,
311
- "skill": 0.0709,
312
- "coverage": 1.0,
313
- "random": 0.2227,
314
- "rule": "track",
315
- "in_index": true,
316
- "tracks": [
317
- {
318
- "track": "A · Arabic",
319
- "score": 0.2379,
320
- "headline": false
321
- },
322
- {
323
- "track": "A · English",
324
- "score": 0.2778,
325
- "headline": true
326
- },
327
- {
328
- "track": "C · Arabic pairs",
329
- "score": 0.55,
330
- "headline": false
331
- },
332
- {
333
- "track": "C · English pairs",
334
- "score": 0.51,
335
- "headline": false
336
- }
337
- ]
338
- },
339
- "41": {
340
- "raw": 0.3439,
341
- "skill": 0.0159,
342
- "coverage": 1.0,
343
- "random": 0.3333,
344
- "rule": "track",
345
- "in_index": true,
346
- "tracks": []
347
- },
348
- "42": {
349
- "raw": 0.4526,
350
- "skill": 0.0,
351
- "coverage": 1.0,
352
- "random": 0.486,
353
- "rule": "chance",
354
- "in_index": true
355
- },
356
- "43": {
357
- "raw": 0.4491,
358
- "skill": 0.126,
359
- "coverage": 1.0,
360
- "random": 0.3697,
361
- "rule": "track",
362
- "in_index": true,
363
- "tracks": []
364
- },
365
- "44": {
366
- "raw": 0.4988,
367
- "skill": 0.0,
368
- "coverage": 1.0,
369
- "random": 0.5,
370
- "rule": "track",
371
- "in_index": true,
372
- "tracks": []
373
- },
374
- "45": {
375
- "raw": 0.0,
376
- "skill": 0.0,
377
- "coverage": 1.0,
378
- "random": 0.1641,
379
- "rule": "chance",
380
- "in_index": true
381
- },
382
- "48": {
383
- "raw": 0.0,
384
- "skill": 0.0,
385
- "coverage": 1.0,
386
- "random": 0.25,
387
- "rule": "vs baseline",
388
- "in_index": true
389
- },
390
- "50": {
391
- "raw": 0.3186,
392
- "skill": 0.0111,
393
- "coverage": 1.0,
394
- "random": 0.311,
395
- "rule": "track",
396
- "in_index": true,
397
- "tracks": []
398
- },
399
- "56": {
400
- "raw": 0.535,
401
- "skill": 0.07,
402
- "coverage": 1.0,
403
- "random": 0.5,
404
- "rule": "chance",
405
- "in_index": true
406
- },
407
- "57": {
408
- "raw": 0.1031,
409
- "skill": 0.0,
410
- "coverage": 1.0,
411
- "random": 0.1109,
412
- "rule": "chance",
413
- "in_index": true
414
- },
415
- "58": {
416
- "raw": 0.3245,
417
- "skill": 0.0209,
418
- "coverage": 1.0,
419
- "random": 0.3101,
420
- "rule": "chance",
421
- "in_index": true
422
- },
423
- "59": {
424
- "raw": 0.499,
425
- "skill": 0.0,
426
- "coverage": 1.0,
427
- "random": 0.5177,
428
- "rule": "chance",
429
- "in_index": true
430
- },
431
- "61": {
432
- "raw": 0.5375,
433
- "skill": 0.075,
434
- "coverage": 1.0,
435
- "random": 0.5,
436
- "rule": "chance",
437
- "in_index": true
438
- },
439
- "62": {
440
- "raw": 0.1416,
441
- "skill": 0.0,
442
- "coverage": 1.0,
443
- "random": 0.25,
444
- "rule": "chance",
445
- "in_index": true
446
- },
447
- "64": {
448
- "raw": 0.2576,
449
- "skill": 0.072,
450
- "coverage": 1.0,
451
- "random": 0.2,
452
- "rule": "chance",
453
- "in_index": true
454
- },
455
- "6": {
456
- "raw": 0.5681,
457
- "skill": 0.0,
458
- "coverage": 1.0,
459
- "random": 0.5735,
460
- "rule": "shown, not counted",
461
- "in_index": false
462
- },
463
- "10": {
464
- "raw": 0.5085,
465
- "skill": 0.1822,
466
- "coverage": 1.0,
467
- "random": 0.399,
468
- "rule": "shown, not counted",
469
- "in_index": false
470
- },
471
- "24": {
472
- "raw": 0.2728,
473
- "skill": 0.0304,
474
- "coverage": 1.0,
475
- "random": 0.25,
476
- "rule": "shown, not counted",
477
- "in_index": false
478
- },
479
- "26": {
480
- "raw": 0.4082,
481
- "skill": 0.2107,
482
- "coverage": 1.0,
483
- "random": 0.2502,
484
- "rule": "shown, not counted",
485
- "in_index": false
486
- },
487
- "27": {
488
- "raw": 0.2713,
489
- "skill": 0.0281,
490
- "coverage": 1.0,
491
- "random": 0.2502,
492
- "rule": "shown, not counted",
493
- "in_index": false
494
- },
495
- "34": {
496
- "raw": 0.2,
497
- "skill": 0.04,
498
- "coverage": 1.0,
499
- "random": 0.1667,
500
- "rule": "shown, not counted",
501
- "in_index": false
502
- }
503
- },
504
- "coverage": 1.0,
505
- "note": "Decision Index 0.2.1 averages 38 benchmarks in five areas. Arts & Human Taste weighs 10%; the other four share 90% in proportion to the square root of their benchmark count (knowledge 25.8%, language 25.8%, retrieval 20.0%, tools 18.3%). Inside an area, gold ★ benchmarks weigh 1.2 and the rest 1.0; the index is 100 x the weighted mean of the five areas. Each benchmark is chance-corrected first, (score - chance) / (1 - chance) clipped to 0-1, so 0 means random guessing and 100 means perfect. Every score is coverage-adjusted, so an unanswered or unsupported request counts as wrong. ForecastBench enters against its baseline: clip((0.25 - Brier) / 0.25) x coverage, so always predicting 0.5 scores zero. MMLU, ARC-Easy, ARC-Challenge, RouterBench, SGD stay on the board as non-index benchmarks. The six interactive environments are still unrun and stay out. Every entrant on the board has results on all 38 index benchmarks. Point estimates only, no uncertainty intervals yet."
506
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
results/decision_index/scores.json DELETED
@@ -1,1428 +0,0 @@
1
- {
2
- "engine": "thx01",
3
- "edition": "0.2.1",
4
- "generated_utc": "2026-10-06T09:46:24+00:00",
5
- "suite": {
6
- "edition": "release-v2.1",
7
- "requests": 120340,
8
- "scoreable": 119898,
9
- "excluded": 442,
10
- "added_requests": 30419,
11
- "benchmarks": 44,
12
- "rows_sha256": "b2b56d6fb636837ca469e689087bdbf373dda8de7638aa2da6793e6eda0792d5",
13
- "added_sha256": "7429f3c9cdddb772c1cfc42bb2a45e8516b0032152b746e6929f1c8b52f4ce89"
14
- },
15
- "completed": 149816,
16
- "complete": false,
17
- "counts": {
18
- "ok": 149816
19
- },
20
- "latency_ms": {
21
- "median": 26.8,
22
- "p95": 169.2,
23
- "mean": 51.9
24
- },
25
- "decision_index": 10.63,
26
- "raw_index": 31.87,
27
- "scores": {
28
- "balanced_skill": 10.63,
29
- "balanced_raw": 31.87,
30
- "breadth_skill": 8.68
31
- },
32
- "areas": [
33
- {
34
- "id": "knowledge",
35
- "label": "Knowledge & Reasoning",
36
- "raw": 0.2144,
37
- "skill": 0.0147,
38
- "coverage": 1.0,
39
- "n": 10,
40
- "benchmarks": [
41
- 25,
42
- 30,
43
- 31,
44
- 32,
45
- 33,
46
- 43,
47
- 44,
48
- 45,
49
- 57,
50
- 58
51
- ]
52
- },
53
- {
54
- "id": "language",
55
- "label": "Language Understanding",
56
- "raw": 0.3667,
57
- "skill": 0.0663,
58
- "coverage": 1.0,
59
- "n": 10,
60
- "benchmarks": [
61
- 11,
62
- 12,
63
- 28,
64
- 29,
65
- 38,
66
- 39,
67
- 40,
68
- 41,
69
- 42,
70
- 59
71
- ]
72
- },
73
- {
74
- "id": "retrieval",
75
- "label": "Retrieval & Classification",
76
- "raw": 0.4547,
77
- "skill": 0.2641,
78
- "coverage": 1.0,
79
- "n": 6,
80
- "benchmarks": [
81
- 4,
82
- 5,
83
- 36,
84
- 37,
85
- 56,
86
- 61
87
- ]
88
- },
89
- {
90
- "id": "tools",
91
- "label": "Tools & Automation",
92
- "raw": 0.2671,
93
- "skill": 0.1713,
94
- "coverage": 1.0,
95
- "n": 5,
96
- "benchmarks": [
97
- 1,
98
- 2,
99
- 3,
100
- 9,
101
- 62
102
- ]
103
- },
104
- {
105
- "id": "arts",
106
- "label": "Arts & Human Taste",
107
- "raw": 0.2857,
108
- "skill": 0.0115,
109
- "coverage": 1.0,
110
- "n": 7,
111
- "benchmarks": [
112
- 20,
113
- 21,
114
- 22,
115
- 23,
116
- 48,
117
- 50,
118
- 64
119
- ]
120
- }
121
- ],
122
- "index_benchmarks": {
123
- "1": {
124
- "raw": 0.3849,
125
- "skill": 0.1696,
126
- "coverage": 1.0,
127
- "random": 0.2592,
128
- "rule": "track",
129
- "in_index": true,
130
- "tracks": []
131
- },
132
- "2": {
133
- "raw": 0.3242,
134
- "skill": 0.2196,
135
- "coverage": 1.0,
136
- "random": 0.1341,
137
- "rule": "track",
138
- "in_index": true,
139
- "tracks": []
140
- },
141
- "3": {
142
- "raw": 0.4291,
143
- "skill": 0.4181,
144
- "coverage": 1.0,
145
- "random": 0.0189,
146
- "rule": "chance",
147
- "in_index": true
148
- },
149
- "4": {
150
- "raw": 0.5767,
151
- "skill": 0.5713,
152
- "coverage": 1.0,
153
- "random": 0.0127,
154
- "rule": "chance",
155
- "in_index": true
156
- },
157
- "5": {
158
- "raw": 0.6716,
159
- "skill": 0.6696,
160
- "coverage": 1.0,
161
- "random": 0.006,
162
- "rule": "chance",
163
- "in_index": true
164
- },
165
- "9": {
166
- "raw": 0.0,
167
- "skill": 0.0,
168
- "coverage": 1.0,
169
- "random": 0.0,
170
- "rule": "chance",
171
- "in_index": true
172
- },
173
- "11": {
174
- "raw": 0.2477,
175
- "skill": 0.0,
176
- "coverage": 1.0,
177
- "random": 0.3085,
178
- "rule": "track",
179
- "in_index": true,
180
- "tracks": []
181
- },
182
- "12": {
183
- "raw": 0.3923,
184
- "skill": 0.0897,
185
- "coverage": 1.0,
186
- "random": 0.3324,
187
- "rule": "chance",
188
- "in_index": true
189
- },
190
- "20": {
191
- "raw": 0.4875,
192
- "skill": 0.0,
193
- "coverage": 1.0,
194
- "random": 0.5,
195
- "rule": "track",
196
- "in_index": true,
197
- "tracks": []
198
- },
199
- "21": {
200
- "raw": 0.4928,
201
- "skill": 0.0,
202
- "coverage": 1.0,
203
- "random": 0.5,
204
- "rule": "track",
205
- "in_index": true,
206
- "tracks": []
207
- },
208
- "22": {
209
- "raw": 0.0019,
210
- "skill": 0.0,
211
- "coverage": 1.0,
212
- "random": 0.0078,
213
- "rule": "track",
214
- "in_index": true,
215
- "tracks": []
216
- },
217
- "23": {
218
- "raw": 0.4987,
219
- "skill": 0.0,
220
- "coverage": 1.0,
221
- "random": 0.5,
222
- "rule": "track",
223
- "in_index": true,
224
- "tracks": []
225
- },
226
- "25": {
227
- "raw": 0.2347,
228
- "skill": 0.0,
229
- "coverage": 1.0,
230
- "random": 0.25,
231
- "rule": "track",
232
- "in_index": true,
233
- "tracks": []
234
- },
235
- "28": {
236
- "raw": 0.513,
237
- "skill": 0.026,
238
- "coverage": 1.0,
239
- "random": 0.5,
240
- "rule": "chance",
241
- "in_index": true
242
- },
243
- "29": {
244
- "raw": 0.291,
245
- "skill": 0.0547,
246
- "coverage": 1.0,
247
- "random": 0.25,
248
- "rule": "chance",
249
- "in_index": true
250
- },
251
- "30": {
252
- "raw": 0.108,
253
- "skill": 0.0,
254
- "coverage": 1.0,
255
- "random": 0.25,
256
- "rule": "track",
257
- "in_index": true,
258
- "tracks": [
259
- {
260
- "track": "GSM8K-4choice",
261
- "score": 0.1729,
262
- "headline": false
263
- },
264
- {
265
- "track": "GSM8K-10choice",
266
- "score": 0.0432,
267
- "headline": false
268
- }
269
- ]
270
- },
271
- "31": {
272
- "raw": 0.0888,
273
- "skill": 0.0076,
274
- "coverage": 1.0,
275
- "random": 0.0819,
276
- "rule": "track",
277
- "in_index": true,
278
- "tracks": []
279
- },
280
- "32": {
281
- "raw": 0.371,
282
- "skill": 0.0,
283
- "coverage": 1.0,
284
- "random": 0.371,
285
- "rule": "chance",
286
- "in_index": true
287
- },
288
- "33": {
289
- "raw": 0.0055,
290
- "skill": 0.0,
291
- "coverage": 1.0,
292
- "random": 0.0131,
293
- "rule": "chance",
294
- "in_index": true
295
- },
296
- "36": {
297
- "raw": 0.1322,
298
- "skill": 0.0183,
299
- "coverage": 1.0,
300
- "random": 0.116,
301
- "rule": "track",
302
- "in_index": true,
303
- "tracks": []
304
- },
305
- "37": {
306
- "raw": 0.272,
307
- "skill": 0.0868,
308
- "coverage": 1.0,
309
- "random": 0.2027,
310
- "rule": "track",
311
- "in_index": true,
312
- "tracks": []
313
- },
314
- "38": {
315
- "raw": 0.0311,
316
- "skill": 0.0001,
317
- "coverage": 1.0,
318
- "random": 0.031,
319
- "rule": "chance",
320
- "in_index": true
321
- },
322
- "39": {
323
- "raw": 0.5997,
324
- "skill": 0.4112,
325
- "coverage": 1.0,
326
- "random": 0.3201,
327
- "rule": "chance",
328
- "in_index": true
329
- },
330
- "40": {
331
- "raw": 0.2778,
332
- "skill": 0.0709,
333
- "coverage": 1.0,
334
- "random": 0.2227,
335
- "rule": "track",
336
- "in_index": true,
337
- "tracks": [
338
- {
339
- "track": "A · Arabic",
340
- "score": 0.2379,
341
- "headline": false
342
- },
343
- {
344
- "track": "A · English",
345
- "score": 0.2778,
346
- "headline": true
347
- },
348
- {
349
- "track": "C · Arabic pairs",
350
- "score": 0.55,
351
- "headline": false
352
- },
353
- {
354
- "track": "C · English pairs",
355
- "score": 0.51,
356
- "headline": false
357
- }
358
- ]
359
- },
360
- "41": {
361
- "raw": 0.3439,
362
- "skill": 0.0159,
363
- "coverage": 1.0,
364
- "random": 0.3333,
365
- "rule": "track",
366
- "in_index": true,
367
- "tracks": []
368
- },
369
- "42": {
370
- "raw": 0.4526,
371
- "skill": 0.0,
372
- "coverage": 1.0,
373
- "random": 0.486,
374
- "rule": "chance",
375
- "in_index": true
376
- },
377
- "43": {
378
- "raw": 0.4491,
379
- "skill": 0.126,
380
- "coverage": 1.0,
381
- "random": 0.3697,
382
- "rule": "track",
383
- "in_index": true,
384
- "tracks": []
385
- },
386
- "44": {
387
- "raw": 0.4988,
388
- "skill": 0.0,
389
- "coverage": 1.0,
390
- "random": 0.5,
391
- "rule": "track",
392
- "in_index": true,
393
- "tracks": []
394
- },
395
- "45": {
396
- "raw": 0.0,
397
- "skill": 0.0,
398
- "coverage": 1.0,
399
- "random": 0.1641,
400
- "rule": "chance",
401
- "in_index": true
402
- },
403
- "48": {
404
- "raw": 0.0,
405
- "skill": 0.0,
406
- "coverage": 1.0,
407
- "random": 0.25,
408
- "rule": "vs baseline",
409
- "in_index": true
410
- },
411
- "50": {
412
- "raw": 0.3186,
413
- "skill": 0.0111,
414
- "coverage": 1.0,
415
- "random": 0.311,
416
- "rule": "track",
417
- "in_index": true,
418
- "tracks": []
419
- },
420
- "56": {
421
- "raw": 0.535,
422
- "skill": 0.07,
423
- "coverage": 1.0,
424
- "random": 0.5,
425
- "rule": "chance",
426
- "in_index": true
427
- },
428
- "57": {
429
- "raw": 0.1031,
430
- "skill": 0.0,
431
- "coverage": 1.0,
432
- "random": 0.1109,
433
- "rule": "chance",
434
- "in_index": true
435
- },
436
- "58": {
437
- "raw": 0.3245,
438
- "skill": 0.0209,
439
- "coverage": 1.0,
440
- "random": 0.3101,
441
- "rule": "chance",
442
- "in_index": true
443
- },
444
- "59": {
445
- "raw": 0.499,
446
- "skill": 0.0,
447
- "coverage": 1.0,
448
- "random": 0.5177,
449
- "rule": "chance",
450
- "in_index": true
451
- },
452
- "61": {
453
- "raw": 0.5375,
454
- "skill": 0.075,
455
- "coverage": 1.0,
456
- "random": 0.5,
457
- "rule": "chance",
458
- "in_index": true
459
- },
460
- "62": {
461
- "raw": 0.1416,
462
- "skill": 0.0,
463
- "coverage": 1.0,
464
- "random": 0.25,
465
- "rule": "chance",
466
- "in_index": true
467
- },
468
- "64": {
469
- "raw": 0.2576,
470
- "skill": 0.072,
471
- "coverage": 1.0,
472
- "random": 0.2,
473
- "rule": "chance",
474
- "in_index": true
475
- },
476
- "6": {
477
- "raw": 0.5681,
478
- "skill": 0.0,
479
- "coverage": 1.0,
480
- "random": 0.5735,
481
- "rule": "shown, not counted",
482
- "in_index": false
483
- },
484
- "10": {
485
- "raw": 0.5085,
486
- "skill": 0.1822,
487
- "coverage": 1.0,
488
- "random": 0.399,
489
- "rule": "shown, not counted",
490
- "in_index": false
491
- },
492
- "24": {
493
- "raw": 0.2728,
494
- "skill": 0.0304,
495
- "coverage": 1.0,
496
- "random": 0.25,
497
- "rule": "shown, not counted",
498
- "in_index": false
499
- },
500
- "26": {
501
- "raw": 0.4082,
502
- "skill": 0.2107,
503
- "coverage": 1.0,
504
- "random": 0.2502,
505
- "rule": "shown, not counted",
506
- "in_index": false
507
- },
508
- "27": {
509
- "raw": 0.2713,
510
- "skill": 0.0281,
511
- "coverage": 1.0,
512
- "random": 0.2502,
513
- "rule": "shown, not counted",
514
- "in_index": false
515
- },
516
- "34": {
517
- "raw": 0.2,
518
- "skill": 0.04,
519
- "coverage": 1.0,
520
- "random": 0.1667,
521
- "rule": "shown, not counted",
522
- "in_index": false
523
- }
524
- },
525
- "benchmarks": {
526
- "1": {
527
- "catalog_id": 1,
528
- "dataset": "BFCL",
529
- "requests": 1694,
530
- "answered": 1694,
531
- "unsupported": 0,
532
- "errors": 0,
533
- "abstained": 0,
534
- "pending": 0,
535
- "scored_requests": 1694,
536
- "metric": "case exact accuracy",
537
- "score": 0.3849,
538
- "reference_same_cases": null,
539
- "median_ms": 121.5,
540
- "index_raw": 0.3849,
541
- "index_skill": 0.1696,
542
- "coverage": 1.0,
543
- "chance": 0.2592,
544
- "in_index": true
545
- },
546
- "2": {
547
- "catalog_id": 2,
548
- "dataset": "ToolRet",
549
- "requests": 685,
550
- "answered": 685,
551
- "unsupported": 0,
552
- "errors": 0,
553
- "abstained": 0,
554
- "pending": 0,
555
- "scored_requests": 685,
556
- "metric": "nDCG@10",
557
- "score": 0.3242,
558
- "reference_same_cases": null,
559
- "median_ms": 338.8,
560
- "index_raw": 0.3242,
561
- "index_skill": 0.2196,
562
- "coverage": 1.0,
563
- "chance": 0.1341,
564
- "in_index": true
565
- },
566
- "3": {
567
- "catalog_id": 3,
568
- "dataset": "API-Bank",
569
- "requests": 508,
570
- "answered": 508,
571
- "unsupported": 0,
572
- "errors": 0,
573
- "abstained": 0,
574
- "pending": 0,
575
- "scored_requests": 508,
576
- "metric": "accuracy",
577
- "score": 0.4291,
578
- "reference_same_cases": null,
579
- "median_ms": 234.5,
580
- "index_raw": 0.4291,
581
- "index_skill": 0.4181,
582
- "coverage": 1.0,
583
- "chance": 0.0189,
584
- "in_index": true
585
- },
586
- "4": {
587
- "catalog_id": 4,
588
- "dataset": "BANKING77",
589
- "requests": 3080,
590
- "answered": 3080,
591
- "unsupported": 0,
592
- "errors": 0,
593
- "abstained": 0,
594
- "pending": 0,
595
- "scored_requests": 3080,
596
- "metric": "macro-F1",
597
- "score": 0.5767,
598
- "reference_same_cases": null,
599
- "median_ms": 64.0,
600
- "index_raw": 0.5767,
601
- "index_skill": 0.5713,
602
- "coverage": 1.0,
603
- "chance": 0.0127,
604
- "in_index": true
605
- },
606
- "5": {
607
- "catalog_id": 5,
608
- "dataset": "CLINC150+OOS",
609
- "requests": 5500,
610
- "answered": 5500,
611
- "unsupported": 0,
612
- "errors": 0,
613
- "abstained": 0,
614
- "pending": 0,
615
- "scored_requests": 5500,
616
- "metric": "macro-F1",
617
- "score": 0.6716,
618
- "reference_same_cases": null,
619
- "median_ms": 95.6,
620
- "index_raw": 0.6716,
621
- "index_skill": 0.6696,
622
- "coverage": 1.0,
623
- "chance": 0.006,
624
- "in_index": true
625
- },
626
- "6": {
627
- "catalog_id": 6,
628
- "dataset": "RouterBench",
629
- "requests": 10000,
630
- "answered": 10000,
631
- "unsupported": 0,
632
- "errors": 0,
633
- "abstained": 0,
634
- "pending": 0,
635
- "scored_requests": 10000,
636
- "metric": "selected quality (quality objective)",
637
- "score": 0.5681,
638
- "reference_same_cases": null,
639
- "median_ms": 79.3,
640
- "tracks": {
641
- "RouterBench-0shot": {
642
- "metric": "selected quality (quality objective)",
643
- "score": 0.5445932440535679,
644
- "scored_requests": 5003
645
- },
646
- "RouterBench-5shot": {
647
- "metric": "selected quality (quality objective)",
648
- "score": 0.5916750050030019,
649
- "scored_requests": 4997
650
- }
651
- },
652
- "index_raw": 0.5681,
653
- "index_skill": 0.0,
654
- "coverage": 1.0,
655
- "chance": 0.5735,
656
- "in_index": false
657
- },
658
- "9": {
659
- "catalog_id": 9,
660
- "dataset": "Home appliance simulator",
661
- "requests": 88,
662
- "answered": 88,
663
- "unsupported": 0,
664
- "errors": 0,
665
- "abstained": 0,
666
- "pending": 0,
667
- "scored_requests": 88,
668
- "metric": "case exact accuracy",
669
- "score": 0,
670
- "reference_same_cases": null,
671
- "median_ms": 185.2,
672
- "index_raw": 0.0,
673
- "index_skill": 0.0,
674
- "coverage": 1.0,
675
- "chance": 0.0,
676
- "in_index": true
677
- },
678
- "10": {
679
- "catalog_id": 10,
680
- "dataset": "SGD/SGD-X",
681
- "requests": 2500,
682
- "answered": 2500,
683
- "unsupported": 0,
684
- "errors": 0,
685
- "abstained": 0,
686
- "pending": 0,
687
- "scored_requests": 2500,
688
- "metric": "macro-F1",
689
- "score": 0.5085,
690
- "reference_same_cases": null,
691
- "median_ms": 51.0,
692
- "index_raw": 0.5085,
693
- "index_skill": 0.1822,
694
- "coverage": 1.0,
695
- "chance": 0.399,
696
- "in_index": false
697
- },
698
- "11": {
699
- "catalog_id": 11,
700
- "dataset": "ContractNLI",
701
- "requests": 123,
702
- "answered": 123,
703
- "unsupported": 0,
704
- "errors": 0,
705
- "abstained": 0,
706
- "pending": 0,
707
- "scored_requests": 123,
708
- "metric": "macro-F1",
709
- "score": 0.2477,
710
- "reference_same_cases": null,
711
- "median_ms": 246.6,
712
- "index_raw": 0.2477,
713
- "index_skill": 0.0,
714
- "coverage": 1.0,
715
- "chance": 0.3085,
716
- "in_index": true
717
- },
718
- "12": {
719
- "catalog_id": 12,
720
- "dataset": "ANLI",
721
- "requests": 3200,
722
- "answered": 3200,
723
- "unsupported": 0,
724
- "errors": 0,
725
- "abstained": 0,
726
- "pending": 0,
727
- "scored_requests": 3200,
728
- "metric": "macro-F1",
729
- "score": 0.3923,
730
- "reference_same_cases": null,
731
- "median_ms": 20.8,
732
- "index_raw": 0.3923,
733
- "index_skill": 0.0897,
734
- "coverage": 1.0,
735
- "chance": 0.3324,
736
- "in_index": true
737
- },
738
- "20": {
739
- "catalog_id": 20,
740
- "dataset": "BPoMP",
741
- "requests": 5000,
742
- "answered": 5000,
743
- "unsupported": 0,
744
- "errors": 0,
745
- "abstained": 0,
746
- "pending": 0,
747
- "scored_requests": 5000,
748
- "metric": "accuracy",
749
- "score": 0.4924,
750
- "reference_same_cases": null,
751
- "median_ms": 21.0,
752
- "index_raw": 0.4875,
753
- "index_skill": 0.0,
754
- "coverage": 1.0,
755
- "chance": 0.5,
756
- "in_index": true
757
- },
758
- "21": {
759
- "catalog_id": 21,
760
- "dataset": "Humicroedit",
761
- "requests": 2628,
762
- "answered": 2628,
763
- "unsupported": 0,
764
- "errors": 0,
765
- "abstained": 0,
766
- "pending": 0,
767
- "scored_requests": 2628,
768
- "metric": "accuracy",
769
- "score": 0.4928,
770
- "reference_same_cases": null,
771
- "median_ms": 19.7,
772
- "index_raw": 0.4928,
773
- "index_skill": 0.0,
774
- "coverage": 1.0,
775
- "chance": 0.5,
776
- "in_index": true
777
- },
778
- "22": {
779
- "catalog_id": 22,
780
- "dataset": "POP909-CL",
781
- "requests": 2000,
782
- "answered": 2000,
783
- "unsupported": 0,
784
- "errors": 0,
785
- "abstained": 0,
786
- "pending": 0,
787
- "scored_requests": 2000,
788
- "metric": "accuracy",
789
- "score": 0.0015,
790
- "reference_same_cases": null,
791
- "median_ms": 283.8,
792
- "index_raw": 0.0019,
793
- "index_skill": 0.0,
794
- "coverage": 1.0,
795
- "chance": 0.0078,
796
- "in_index": true
797
- },
798
- "23": {
799
- "catalog_id": 23,
800
- "dataset": "cfcolor",
801
- "requests": 5000,
802
- "answered": 5000,
803
- "unsupported": 0,
804
- "errors": 0,
805
- "abstained": 0,
806
- "pending": 0,
807
- "scored_requests": 5000,
808
- "metric": "accuracy",
809
- "score": 0.4794,
810
- "reference_same_cases": null,
811
- "median_ms": 23.6,
812
- "index_raw": 0.4987,
813
- "index_skill": 0.0,
814
- "coverage": 1.0,
815
- "chance": 0.5,
816
- "in_index": true
817
- },
818
- "24": {
819
- "catalog_id": 24,
820
- "dataset": "MMLU",
821
- "requests": 14033,
822
- "answered": 14033,
823
- "unsupported": 0,
824
- "errors": 0,
825
- "abstained": 0,
826
- "pending": 0,
827
- "scored_requests": 14033,
828
- "metric": "accuracy",
829
- "score": 0.2728,
830
- "reference_same_cases": null,
831
- "median_ms": 20.2,
832
- "index_raw": 0.2728,
833
- "index_skill": 0.0304,
834
- "coverage": 1.0,
835
- "chance": 0.25,
836
- "in_index": false
837
- },
838
- "25": {
839
- "catalog_id": 25,
840
- "dataset": "GPQA Diamond",
841
- "requests": 196,
842
- "answered": 196,
843
- "unsupported": 0,
844
- "errors": 0,
845
- "abstained": 0,
846
- "pending": 0,
847
- "scored_requests": 196,
848
- "metric": "accuracy",
849
- "score": 0.2347,
850
- "reference_same_cases": null,
851
- "median_ms": 19.6,
852
- "index_raw": 0.2347,
853
- "index_skill": 0.0,
854
- "coverage": 1.0,
855
- "chance": 0.25,
856
- "in_index": true
857
- },
858
- "26": {
859
- "catalog_id": 26,
860
- "dataset": "ARC-Easy",
861
- "requests": 2376,
862
- "answered": 2376,
863
- "unsupported": 0,
864
- "errors": 0,
865
- "abstained": 0,
866
- "pending": 0,
867
- "scored_requests": 2376,
868
- "metric": "accuracy",
869
- "score": 0.4082,
870
- "reference_same_cases": null,
871
- "median_ms": 20.5,
872
- "index_raw": 0.4082,
873
- "index_skill": 0.2107,
874
- "coverage": 1.0,
875
- "chance": 0.2502,
876
- "in_index": false
877
- },
878
- "27": {
879
- "catalog_id": 27,
880
- "dataset": "ARC-Challenge",
881
- "requests": 1172,
882
- "answered": 1172,
883
- "unsupported": 0,
884
- "errors": 0,
885
- "abstained": 0,
886
- "pending": 0,
887
- "scored_requests": 1172,
888
- "metric": "accuracy",
889
- "score": 0.2713,
890
- "reference_same_cases": null,
891
- "median_ms": 21.0,
892
- "index_raw": 0.2713,
893
- "index_skill": 0.0281,
894
- "coverage": 1.0,
895
- "chance": 0.2502,
896
- "in_index": false
897
- },
898
- "28": {
899
- "catalog_id": 28,
900
- "dataset": "WinoGrande",
901
- "requests": 1267,
902
- "answered": 1267,
903
- "unsupported": 0,
904
- "errors": 0,
905
- "abstained": 0,
906
- "pending": 0,
907
- "scored_requests": 1267,
908
- "metric": "accuracy",
909
- "score": 0.513,
910
- "reference_same_cases": null,
911
- "median_ms": 19.7,
912
- "index_raw": 0.513,
913
- "index_skill": 0.026,
914
- "coverage": 1.0,
915
- "chance": 0.5,
916
- "in_index": true
917
- },
918
- "29": {
919
- "catalog_id": 29,
920
- "dataset": "HellaSwag",
921
- "requests": 10042,
922
- "answered": 10042,
923
- "unsupported": 0,
924
- "errors": 0,
925
- "abstained": 0,
926
- "pending": 0,
927
- "scored_requests": 10042,
928
- "metric": "accuracy",
929
- "score": 0.291,
930
- "reference_same_cases": null,
931
- "median_ms": 21.5,
932
- "index_raw": 0.291,
933
- "index_skill": 0.0547,
934
- "coverage": 1.0,
935
- "chance": 0.25,
936
- "in_index": true
937
- },
938
- "30": {
939
- "catalog_id": 30,
940
- "dataset": "GSM8K",
941
- "requests": 2638,
942
- "answered": 2638,
943
- "unsupported": 0,
944
- "errors": 0,
945
- "abstained": 0,
946
- "pending": 0,
947
- "scored_requests": 2638,
948
- "metric": "accuracy",
949
- "score": 0.108,
950
- "reference_same_cases": null,
951
- "median_ms": 21.5,
952
- "tracks": [
953
- {
954
- "track": "GSM8K-4choice",
955
- "score": 0.1729,
956
- "headline": false
957
- },
958
- {
959
- "track": "GSM8K-10choice",
960
- "score": 0.0432,
961
- "headline": false
962
- }
963
- ],
964
- "index_raw": 0.108,
965
- "index_skill": 0.0,
966
- "coverage": 1.0,
967
- "chance": 0.25,
968
- "in_index": true
969
- },
970
- "31": {
971
- "catalog_id": 31,
972
- "dataset": "ChessBench",
973
- "requests": 5000,
974
- "answered": 5000,
975
- "unsupported": 0,
976
- "errors": 0,
977
- "abstained": 0,
978
- "pending": 0,
979
- "scored_requests": 5000,
980
- "metric": "accuracy",
981
- "score": 0.0888,
982
- "reference_same_cases": null,
983
- "median_ms": 62.1,
984
- "index_raw": 0.0888,
985
- "index_skill": 0.0076,
986
- "coverage": 1.0,
987
- "chance": 0.0819,
988
- "in_index": true
989
- },
990
- "32": {
991
- "catalog_id": 32,
992
- "dataset": "MuSR",
993
- "requests": 752,
994
- "answered": 752,
995
- "unsupported": 0,
996
- "errors": 0,
997
- "abstained": 0,
998
- "pending": 0,
999
- "scored_requests": 752,
1000
- "metric": "accuracy",
1001
- "score": 0.371,
1002
- "reference_same_cases": null,
1003
- "median_ms": 51.9,
1004
- "index_raw": 0.371,
1005
- "index_skill": 0.0,
1006
- "coverage": 1.0,
1007
- "chance": 0.371,
1008
- "in_index": true
1009
- },
1010
- "33": {
1011
- "catalog_id": 33,
1012
- "dataset": "SATA-Bench",
1013
- "requests": 1650,
1014
- "answered": 1650,
1015
- "unsupported": 0,
1016
- "errors": 0,
1017
- "abstained": 0,
1018
- "pending": 0,
1019
- "scored_requests": 1650,
1020
- "metric": "case exact accuracy",
1021
- "score": 0.0055,
1022
- "reference_same_cases": null,
1023
- "median_ms": 202.9,
1024
- "index_raw": 0.0055,
1025
- "index_skill": 0.0,
1026
- "coverage": 1.0,
1027
- "chance": 0.0131,
1028
- "in_index": true
1029
- },
1030
- "34": {
1031
- "catalog_id": 34,
1032
- "dataset": "SimpleBench",
1033
- "requests": 10,
1034
- "answered": 10,
1035
- "unsupported": 0,
1036
- "errors": 0,
1037
- "abstained": 0,
1038
- "pending": 0,
1039
- "scored_requests": 10,
1040
- "metric": "accuracy",
1041
- "score": 0.2,
1042
- "reference_same_cases": null,
1043
- "median_ms": 398.9,
1044
- "index_raw": 0.2,
1045
- "index_skill": 0.04,
1046
- "coverage": 1.0,
1047
- "chance": 0.1667,
1048
- "in_index": false
1049
- },
1050
- "36": {
1051
- "catalog_id": 36,
1052
- "dataset": "BRIGHT",
1053
- "requests": 220,
1054
- "answered": 220,
1055
- "unsupported": 0,
1056
- "errors": 0,
1057
- "abstained": 0,
1058
- "pending": 0,
1059
- "scored_requests": 220,
1060
- "metric": "nDCG@10",
1061
- "score": 0.1322,
1062
- "reference_same_cases": null,
1063
- "median_ms": 390.8,
1064
- "index_raw": 0.1322,
1065
- "index_skill": 0.0183,
1066
- "coverage": 1.0,
1067
- "chance": 0.116,
1068
- "in_index": true
1069
- },
1070
- "37": {
1071
- "catalog_id": 37,
1072
- "dataset": "Amazon ESCI",
1073
- "requests": 5000,
1074
- "answered": 5000,
1075
- "unsupported": 0,
1076
- "errors": 0,
1077
- "abstained": 0,
1078
- "pending": 0,
1079
- "scored_requests": 5000,
1080
- "metric": "macro-F1",
1081
- "score": 0.272,
1082
- "reference_same_cases": null,
1083
- "median_ms": 37.3,
1084
- "index_raw": 0.272,
1085
- "index_skill": 0.0868,
1086
- "coverage": 1.0,
1087
- "chance": 0.2027,
1088
- "in_index": true
1089
- },
1090
- "38": {
1091
- "catalog_id": 38,
1092
- "dataset": "ACOS",
1093
- "requests": 1565,
1094
- "answered": 1565,
1095
- "unsupported": 0,
1096
- "errors": 0,
1097
- "abstained": 0,
1098
- "pending": 0,
1099
- "scored_requests": 1565,
1100
- "metric": "per-review F1",
1101
- "score": 0.0311,
1102
- "reference_same_cases": null,
1103
- "median_ms": 211.8,
1104
- "index_raw": 0.0311,
1105
- "index_skill": 0.0001,
1106
- "coverage": 1.0,
1107
- "chance": 0.031,
1108
- "in_index": true
1109
- },
1110
- "39": {
1111
- "catalog_id": 39,
1112
- "dataset": "FinEntity",
1113
- "requests": 979,
1114
- "answered": 979,
1115
- "unsupported": 0,
1116
- "errors": 0,
1117
- "abstained": 0,
1118
- "pending": 0,
1119
- "scored_requests": 979,
1120
- "metric": "macro-F1",
1121
- "score": 0.5997,
1122
- "reference_same_cases": null,
1123
- "median_ms": 24.5,
1124
- "index_raw": 0.5997,
1125
- "index_skill": 0.4112,
1126
- "coverage": 1.0,
1127
- "chance": 0.3201,
1128
- "in_index": true
1129
- },
1130
- "40": {
1131
- "catalog_id": 40,
1132
- "dataset": "iSarcasmEval",
1133
- "requests": 4600,
1134
- "answered": 4600,
1135
- "unsupported": 0,
1136
- "errors": 0,
1137
- "abstained": 0,
1138
- "pending": 0,
1139
- "scored_requests": 4600,
1140
- "metric": "Sarcasm F1 · track A, English",
1141
- "score": 0.2778,
1142
- "reference_same_cases": null,
1143
- "median_ms": 22.3,
1144
- "tracks": [
1145
- {
1146
- "track": "A · Arabic",
1147
- "score": 0.2379,
1148
- "headline": false
1149
- },
1150
- {
1151
- "track": "A · English",
1152
- "score": 0.2778,
1153
- "headline": true
1154
- },
1155
- {
1156
- "track": "C · Arabic pairs",
1157
- "score": 0.55,
1158
- "headline": false
1159
- },
1160
- {
1161
- "track": "C · English pairs",
1162
- "score": 0.51,
1163
- "headline": false
1164
- }
1165
- ],
1166
- "index_raw": 0.2778,
1167
- "index_skill": 0.0709,
1168
- "coverage": 1.0,
1169
- "chance": 0.2227,
1170
- "in_index": true
1171
- },
1172
- "41": {
1173
- "catalog_id": 41,
1174
- "dataset": "VAST",
1175
- "requests": 3006,
1176
- "answered": 3006,
1177
- "unsupported": 0,
1178
- "errors": 0,
1179
- "abstained": 0,
1180
- "pending": 0,
1181
- "scored_requests": 3006,
1182
- "metric": "macro-F1",
1183
- "score": 0.3439,
1184
- "reference_same_cases": null,
1185
- "median_ms": 20.2,
1186
- "index_raw": 0.3439,
1187
- "index_skill": 0.0159,
1188
- "coverage": 1.0,
1189
- "chance": 0.3333,
1190
- "in_index": true
1191
- },
1192
- "42": {
1193
- "catalog_id": 42,
1194
- "dataset": "NLI4CT",
1195
- "requests": 5500,
1196
- "answered": 5500,
1197
- "unsupported": 0,
1198
- "errors": 0,
1199
- "abstained": 0,
1200
- "pending": 0,
1201
- "scored_requests": 5500,
1202
- "metric": "macro-F1",
1203
- "score": 0.4526,
1204
- "reference_same_cases": null,
1205
- "median_ms": 39.7,
1206
- "index_raw": 0.4526,
1207
- "index_skill": 0.0,
1208
- "coverage": 1.0,
1209
- "chance": 0.486,
1210
- "in_index": true
1211
- },
1212
- "43": {
1213
- "catalog_id": 43,
1214
- "dataset": "CRUXEval",
1215
- "requests": 570,
1216
- "answered": 570,
1217
- "unsupported": 0,
1218
- "errors": 0,
1219
- "abstained": 0,
1220
- "pending": 0,
1221
- "scored_requests": 570,
1222
- "metric": "accuracy",
1223
- "score": 0.4491,
1224
- "reference_same_cases": null,
1225
- "median_ms": 23.4,
1226
- "index_raw": 0.4491,
1227
- "index_skill": 0.126,
1228
- "coverage": 1.0,
1229
- "chance": 0.3697,
1230
- "in_index": true
1231
- },
1232
- "44": {
1233
- "catalog_id": 44,
1234
- "dataset": "CLadder",
1235
- "requests": 5000,
1236
- "answered": 5000,
1237
- "unsupported": 0,
1238
- "errors": 0,
1239
- "abstained": 0,
1240
- "pending": 0,
1241
- "scored_requests": 5000,
1242
- "metric": "accuracy",
1243
- "score": 0.4988,
1244
- "reference_same_cases": null,
1245
- "median_ms": 19.9,
1246
- "index_raw": 0.4988,
1247
- "index_skill": 0.0,
1248
- "coverage": 1.0,
1249
- "chance": 0.5,
1250
- "in_index": true
1251
- },
1252
- "48": {
1253
- "catalog_id": 48,
1254
- "dataset": "ForecastBench",
1255
- "requests": 10139,
1256
- "answered": 10139,
1257
- "unsupported": 0,
1258
- "errors": 0,
1259
- "abstained": 0,
1260
- "pending": 0,
1261
- "scored_requests": 10139,
1262
- "metric": "Brier (lower is better)",
1263
- "score": 0.4411,
1264
- "reference_same_cases": null,
1265
- "median_ms": 42.1,
1266
- "index_raw": 0.0,
1267
- "index_skill": 0.0,
1268
- "coverage": 1.0,
1269
- "chance": 0.25,
1270
- "in_index": true
1271
- },
1272
- "50": {
1273
- "catalog_id": 50,
1274
- "dataset": "Habermas Machine",
1275
- "requests": 1676,
1276
- "answered": 1676,
1277
- "unsupported": 0,
1278
- "errors": 0,
1279
- "abstained": 0,
1280
- "pending": 0,
1281
- "scored_requests": 1676,
1282
- "metric": "accuracy",
1283
- "score": 0.3186,
1284
- "reference_same_cases": null,
1285
- "median_ms": 40.2,
1286
- "index_raw": 0.3186,
1287
- "index_skill": 0.0111,
1288
- "coverage": 1.0,
1289
- "chance": 0.311,
1290
- "in_index": true
1291
- },
1292
- "56": {
1293
- "catalog_id": 56,
1294
- "dataset": "PhishNChips phishing decisions",
1295
- "requests": 2000,
1296
- "answered": 2000,
1297
- "unsupported": 0,
1298
- "errors": 0,
1299
- "abstained": 0,
1300
- "pending": 0,
1301
- "metric": "accuracy",
1302
- "score": 0.535,
1303
- "median_ms": 82.7,
1304
- "scored_requests": 2000,
1305
- "index_raw": 0.535,
1306
- "index_skill": 0.07,
1307
- "coverage": 1.0,
1308
- "chance": 0.5,
1309
- "in_index": true
1310
- },
1311
- "57": {
1312
- "catalog_id": 57,
1313
- "dataset": "MMLU-Pro",
1314
- "requests": 12032,
1315
- "answered": 12032,
1316
- "unsupported": 0,
1317
- "errors": 0,
1318
- "abstained": 0,
1319
- "pending": 0,
1320
- "metric": "accuracy",
1321
- "score": 0.1031,
1322
- "median_ms": 22.4,
1323
- "scored_requests": 12032,
1324
- "index_raw": 0.1031,
1325
- "index_skill": 0.0,
1326
- "coverage": 1.0,
1327
- "chance": 0.1109,
1328
- "in_index": true
1329
- },
1330
- "58": {
1331
- "catalog_id": 58,
1332
- "dataset": "BBH fixed-option tasks",
1333
- "requests": 5507,
1334
- "answered": 5507,
1335
- "unsupported": 0,
1336
- "errors": 0,
1337
- "abstained": 0,
1338
- "pending": 0,
1339
- "metric": "accuracy",
1340
- "score": 0.3245,
1341
- "median_ms": 21.4,
1342
- "scored_requests": 5507,
1343
- "index_raw": 0.3245,
1344
- "index_skill": 0.0209,
1345
- "coverage": 1.0,
1346
- "chance": 0.3101,
1347
- "in_index": true
1348
- },
1349
- "59": {
1350
- "catalog_id": 59,
1351
- "dataset": "RAGTruth response-level hallucination",
1352
- "requests": 2700,
1353
- "answered": 2700,
1354
- "unsupported": 0,
1355
- "errors": 0,
1356
- "abstained": 0,
1357
- "pending": 0,
1358
- "metric": "F1 on hallucinated class",
1359
- "score": 0.499,
1360
- "median_ms": 46.3,
1361
- "scored_requests": 2700,
1362
- "index_raw": 0.499,
1363
- "index_skill": 0.0,
1364
- "coverage": 1.0,
1365
- "chance": 0.5177,
1366
- "in_index": true
1367
- },
1368
- "61": {
1369
- "catalog_id": 61,
1370
- "dataset": "HoVer claim verification",
1371
- "requests": 4000,
1372
- "answered": 4000,
1373
- "unsupported": 0,
1374
- "errors": 0,
1375
- "abstained": 0,
1376
- "pending": 0,
1377
- "metric": "accuracy",
1378
- "score": 0.5375,
1379
- "median_ms": 32.6,
1380
- "scored_requests": 4000,
1381
- "index_raw": 0.5375,
1382
- "index_skill": 0.075,
1383
- "coverage": 1.0,
1384
- "chance": 0.5,
1385
- "in_index": true
1386
- },
1387
- "62": {
1388
- "catalog_id": 62,
1389
- "dataset": "When2Call MCQ",
1390
- "requests": 3652,
1391
- "answered": 3652,
1392
- "unsupported": 0,
1393
- "errors": 0,
1394
- "abstained": 0,
1395
- "pending": 0,
1396
- "metric": "accuracy",
1397
- "score": 0.1416,
1398
- "median_ms": 43.9,
1399
- "scored_requests": 3652,
1400
- "index_raw": 0.1416,
1401
- "index_skill": 0.0,
1402
- "coverage": 1.0,
1403
- "chance": 0.25,
1404
- "in_index": true
1405
- },
1406
- "64": {
1407
- "catalog_id": 64,
1408
- "dataset": "New Yorker caption matching",
1409
- "requests": 528,
1410
- "answered": 528,
1411
- "unsupported": 0,
1412
- "errors": 0,
1413
- "abstained": 0,
1414
- "pending": 0,
1415
- "metric": "accuracy",
1416
- "score": 0.2576,
1417
- "median_ms": 41.9,
1418
- "scored_requests": 528,
1419
- "index_raw": 0.2576,
1420
- "index_skill": 0.072,
1421
- "coverage": 1.0,
1422
- "chance": 0.2,
1423
- "in_index": true
1424
- }
1425
- },
1426
- "panel_id": "decision-index-0.2.1",
1427
- "note": "Decision Index 0.2.1 averages 38 benchmarks in five areas. Arts & Human Taste weighs 10%; the other four share 90% in proportion to the square root of their benchmark count (knowledge 25.8%, language 25.8%, retrieval 20.0%, tools 18.3%). Inside an area, gold ★ benchmarks weigh 1.2 and the rest 1.0; the index is 100 x the weighted mean of the five areas. Each benchmark is chance-corrected first, (score - chance) / (1 - chance) clipped to 0-1, so 0 means random guessing and 100 means perfect. Every score is coverage-adjusted, so an unanswered or unsupported request counts as wrong. ForecastBench enters against its baseline: clip((0.25 - Brier) / 0.25) x coverage, so always predicting 0.5 scores zero. MMLU, ARC-Easy, ARC-Challenge, RouterBench, SGD stay on the board as non-index benchmarks. The six interactive environments are still unrun and stay out. Every entrant on the board has results on all 38 index benchmarks. Point estimates only, no uncertainty intervals yet."
1428
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
snake/index.html CHANGED
@@ -3,7 +3,7 @@
3
  <head>
4
  <meta charset="utf-8">
5
  <meta name="viewport" content="width=device-width, initial-scale=1">
6
- <title>System01 plays Snake</title>
7
  <style>
8
  :root {
9
  --bg: #0e0e0d; --panel: #171716; --line: #2a2a28; --ink: #f4f3ee; --ink2: #b7b5ab; --muted: #7d7b73;
@@ -43,7 +43,7 @@
43
  </head>
44
  <body>
45
  <header>
46
- <h1>System01 plays Snake</h1>
47
  <span class="sub">no vision · no text generation · 8 yes/no questions answered in one forward pass per move</span>
48
  <span class="sp"></span>
49
  <span class="sub">speed</span>
@@ -51,7 +51,7 @@
51
  </header>
52
  <main>
53
  <section>
54
- <div class="board"><canvas id="c" width="640" height="640" aria-label="Snake board played by System01"></canvas></div>
55
  <div class="row">
56
  <div class="stat"><div class="k">Score</div><div class="v" id="score">0</div></div>
57
  <div class="stat"><div class="k">Best</div><div class="v" id="best">0</div></div>
@@ -78,7 +78,7 @@
78
  </div>
79
  </aside>
80
  </main>
81
- <footer>System01 · RTX 4090 · 0 output tokens · no Snake data in training — the model only reads the state text and answers typed yes/no questions.</footer>
82
  <script>
83
  const D = ["UP", "DOWN", "LEFT", "RIGHT"];
84
  const $ = id => document.getElementById(id);
 
3
  <head>
4
  <meta charset="utf-8">
5
  <meta name="viewport" content="width=device-width, initial-scale=1">
6
+ <title>THX-01 plays Snake</title>
7
  <style>
8
  :root {
9
  --bg: #0e0e0d; --panel: #171716; --line: #2a2a28; --ink: #f4f3ee; --ink2: #b7b5ab; --muted: #7d7b73;
 
43
  </head>
44
  <body>
45
  <header>
46
+ <h1>THX-01 plays Snake</h1>
47
  <span class="sub">no vision · no text generation · 8 yes/no questions answered in one forward pass per move</span>
48
  <span class="sp"></span>
49
  <span class="sub">speed</span>
 
51
  </header>
52
  <main>
53
  <section>
54
+ <div class="board"><canvas id="c" width="640" height="640" aria-label="Snake board played by THX-01"></canvas></div>
55
  <div class="row">
56
  <div class="stat"><div class="k">Score</div><div class="v" id="score">0</div></div>
57
  <div class="stat"><div class="k">Best</div><div class="v" id="best">0</div></div>
 
78
  </div>
79
  </aside>
80
  </main>
81
+ <footer>THX-01 · RTX 4090 · 0 output tokens · no Snake data in training — the model only reads the state text and answers typed yes/no questions.</footer>
82
  <script>
83
  const D = ["UP", "DOWN", "LEFT", "RIGHT"];
84
  const $ = id => document.getElementById(id);