ChordFormer baseline for SheetSage2

Updated 2026-09-15. ChordFormer replaces Jiang et al. (2019) as the chord specialist in the current comparison. SheetSage2 predictions and scores are unchanged.

Dataset	Tracks	ChordFormer maj/min (%)	Released-checkpoint inference
osu2017	142	86.54974535	Average five models' component probabilities, then HMM
Chords1217	1,217	83.94339229	One held-out fold model per track, then HMM

Both use the official default submission chord vocabulary, whole-song CQT, and the upstream HMM. The setting was not selected by testing alternative vocabularies on osu2017. No models were trained or tuned on osu2017.

Scoring protocol

We use the same frozen references and scorer as the SheetSage2 comparison: sanitize intervals, crop predictions to each reference's earliest/latest time, compute per-track mir_eval.chord.evaluate maj/min recall, and average those scores weighted by each reference annotation span. All 1,359 reference-file SHA-256 values match the archived cohorts. Scores are percentages.

The official ChordFormer corpus scorer instead pools correct and eligible reference durations for each metric. Our separate Chords1217 reproduction with that scorer gives 84.350128% using full, matching the accepted paper's 84.35%, and 84.428217% using submission. The table above uses submission with the SheetSage2 scorer, giving 83.943392%. These configurations and aggregation rules must not be mixed. The old 84.09% literature entry is retired.

The held-out Chords1217 evaluation uses a different model per fold. SheetSage2 uses one fixed checkpoint for the collection; aligned scoring does not imply identical training data. No audio-fingerprint-based training-overlap exclusion is claimed here.

Reproduction

Official code, pinned to 3da11c078c802b3f5a2ff0f6f8ec184a41537844.
Five released chordformer_head16(1.0,1.0)_s0.best.sdict through s4.best.sdict checkpoints.
On osu2017, the official chord_recognition.py audio.mp3 output.lab entry uses the five-model ensemble and default vocabulary.
For Chords1217, use the fold in the per-track record; averaging all five models would include models trained on that track and is invalid for this evaluation.
Inference used dgx-071 GPU 7 (NVIDIA H800). Compatibility changes selected logical GPU 0 after external device masking and explicitly loaded trusted upstream checkpoints with weights_only=False.
Two osu2017 songs, including the longest, were checked against the official entry point: identical chord labels and segment counts, with boundary differences below 5.1e-10 seconds from decimal serialization.

With one <sample_id>.lab per track in the reference and prediction directories:

python -m pip install numpy==1.26.4 scipy==1.13.1 mir_eval==0.7
python score_chordformer.py --dataset osu2017 --reference /path/to/reference --predictions /path/to/predictions
python score_chordformer.py --dataset Chords1217 --reference /path/to/chordlab --predictions /path/to/held_out_predictions

The scorer verifies the frozen reference hashes and complete cohort. Add --verify-archived-predictions when using the archived LAB predictions to verify their hashes too. Machine-readable results and per-track scores, folds, and hashes support independent checks.

Attribution

Please cite ChordFormer and its upstream Large-Vocabulary Chord Transcription via Chord Structure Decomposition, as requested by the ChordFormer repository. Keeping this upstream attribution does not retain the Jiang model as a current benchmark entry.