SO-Arena evaluation results for H2O Lightning

#6
by Lichen2003 - opened

We recently released SO-Arena, a benchmark for System One decision models, and H2O Lightning is among the 21 models we evaluated. We'd like to share the results here, as they may be useful to you and to others using the model.

SO-Arena asks whether models are right, and reliably right: it spans 11 major domains with difficulty tiers (the leaderboard is scored on the hardest tier of each), and tests whether decisions stay consistent under reordered options or a missing correct answer.

s1-teaser

SO-Arena at a glance: verified labels, difficulty tiers, decision reliability, and interactive evaluation.

H2O Lightning pairs speed with strong accuracy: it sits on our performance–latency Pareto frontier and outscores OpenAI Decisions at a fraction of its latency.

Metric Result Standing
Mean latency 26.5 ms on the performance–latency Pareto frontier
Leaderboard score 54.9 above OpenAI Decisions (53.6), more than 10× faster
Among 4B models — 3rd of 8 4B configurations

s1-latency
Performance vs. latency on SO-Arena. H2O Lightning is on the Pareto frontier.

Evaluated release: h2oai/h2o-lightning-4b @ 08e1da5. Rankings are among the 19 System One configurations in the paper (reliability: the 18 without StartLux Auto). Open-weight latency is the mean per task on one H100; closed APIs are timed end to end.

Full per-domain and per-tier results are on our leaderboard, with details in the paper. We'd be delighted if SO-Arena could help showcase H2O Lightning's strengths. Everything needed to run it is in our GitHub repo, and we're glad to help.

Sign up or log in to comment