S1CogBench: What System One Decision Models Can and Cannot Do on Cognitive Experiments
Abstract
S1CogBench evaluates System One decision models on 24 cognitive tasks, revealing that fast decisions often lack evidence-based reasoning and vary widely in reliability.
Decision-making in humans and AI involves multiple cognitive abilities. System One decision models produce fast, structured choices. Their cognitive strengths and limitations, however, remain poorly understood. We introduce S1CogBench, a benchmark of 24 established cognitive tasks and 27 evaluation modes, evaluating seven open models and the API model Jev. It combines human-response prediction, autonomous execution, and controlled task manipulations to reveal strengths and failures that aggregate scores miss. We find distinct behavioral profiles across the evaluated models. In these models, strong performance on some learning tasks coexists with failures to gather information or adapt to feedback, while high accuracy does not guarantee human-like responses to changing task demands. Shared readout controls show that model rankings depend partly on how choices are extracted. Separate latency comparisons with generative language models show that long responses are much slower than direct decisions, whereas matched one-token generation adds little overhead. To our knowledge, S1CogBench is the first benchmark of System One decision models on established cognitive experiments. It provides a common empirical basis for identifying their capabilities and limitations before integrating them into automated decision systems.
Community
S1CogBench tests System One decision models, which return one fast typed decision over the legal options, on 24 established cognitive experiments (27 modes), with seven open models and the Jev API. Acting well does not imply predicting humans. In WCST, Nox has the highest autonomous accuracy among the evaluated models (32.1%) yet predicts held-out human choices worse than uniform choice. When more choices remain in the Horizon task, humans pick the less-observed arm 10.00 points more often, while the models predict changes of only -0.49 to 2.62 points. Code, results, a community leaderboard and an extension track for new tasks are at https://dengzhe-hou.github.io/S1CogBench/
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- System1-Arena: Which Decisions Can Be Left to System 1? (2026)
- MMDM: Measuring Massive Multitask Decision Making (2026)
- Better Behavioral Prediction, More Faithful Model Ablations? Evidence from Sequential Choice (2026)
- Large Decision Models: Repurposing Large Language Models as Single-Pass Decision Makers (2026)
- Beyond Successful Play: Diagnosing Opponent Modeling in Large Language Models (2026)
- JevOut: Natural Context Can Flip Decision Models (2026)
- Decision 1.0: Towards Open Decision Foundation Models (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.00911 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper