Add tasksource/tasksource-jev-nano-v0 (#1 Sub-200M Decision Model)
#39
by sileod - opened
Add to the Jev Decision Index tracker.
Model Overview
- Repository: https://huggingface.co/tasksource/tasksource-jev-nano-v0
- Parameters: ~149M (ModernBERT-base backbone)
- Architecture: Multi-vector late interaction (ColBERT MaxSim) with reusable state caching and exact candidate permutation equivariance.
Decision Index 0.2.1 Benchmark Results
- Balanced Skill Index (Chance-Corrected): 43.50 (#1 Sub-200M SOTA)
- Balanced Raw Accuracy: 57.18%
- Tools & Automation: 100.00%
- Arts & Taste: 65.42%
- Retrieval & Classification: 54.33%
- Knowledge & Reasoning: 19.58%
- Language Understanding: 10.43%
Contamination Firewall
Zero contamination: all 21 Decision Index benchmarks were strictly excluded from the training stream with zero train/test overlap.
Update: Grand Cross-Run Model Soup & Calibrated Inference Results
We have updated the repository weights on with the cross-run model soup (Skill: 44.21, JevBench Composite: 68.11).
Key Benchmark Highlights (Edition 0.2.1 Protocol)
- Balanced Skill Index (Chance-Corrected): 44.21 / 100.0 (#1 SOTA Sub-200M)
- Balanced Raw Accuracy: 57.55%
- Breadth Skill Index: 34.58 / 100.0
- Tools & Automation: 100.00% (Flawless across BFCL, API-Bank, When2Call)
- Retrieval & Classification: 57.90% Skill (64.12% Raw)
- Arts & Human Taste: 65.42% Skill (73.33% Raw)
- JevBench Composite Score: 68.11 / 100.0
- Calibration (ECE): 0.1690 (exceptional calibration)
- Classifier-Benchmark v2: 57.14% Macro Accuracy across 49 production tasks
Zero-Contamination Firewall Guarantee
All 21 Decision Index benchmarks (, , , , , , , , etc.) were strictly blacklisted and excluded from the candidate training stream with zero train/test overlap.
Closing pull request.
sileod changed pull request status to closed