# ChordFormer baseline for SheetSage2

Updated 2026-09-15. ChordFormer replaces Jiang et al. (2019) as the chord
specialist in the current comparison. SheetSage2 predictions and scores are unchanged.

| Dataset | Tracks | ChordFormer maj/min (%) | Released-checkpoint inference |
| --- | ---: | ---: | --- |
| osu2017 | 142 | 86.54974535 | Average five models' component probabilities, then HMM |
| Chords1217 | 1,217 | 83.94339229 | One held-out fold model per track, then HMM |

Both use the official default `submission` chord vocabulary, whole-song CQT,
and the upstream HMM. The setting was not selected by testing alternative
vocabularies on osu2017. No models were trained or tuned on osu2017.

## Scoring protocol

We use the same frozen references and scorer as the SheetSage2 comparison:
sanitize intervals, crop predictions to each reference's earliest/latest time,
compute per-track `mir_eval.chord.evaluate` maj/min recall, and average those
scores weighted by each reference annotation span. All 1,359 reference-file
SHA-256 values match the archived cohorts. Scores are percentages.

The official ChordFormer corpus scorer instead pools correct and eligible
reference durations for each metric. Our separate Chords1217 reproduction with
that scorer gives **84.350128%** using `full`, matching the accepted paper's
84.35%, and **84.428217%** using `submission`. The table above uses `submission`
with the SheetSage2 scorer, giving **83.943392%**. These configurations and
aggregation rules must not be mixed. The old 84.09% literature entry is retired.

The held-out Chords1217 evaluation uses a different model per fold. SheetSage2
uses one fixed checkpoint for the collection; aligned scoring does not imply
identical training data. No audio-fingerprint-based training-overlap exclusion
is claimed here.

## Reproduction

- [Official code](https://github.com/mwaseemrandhawa/ChordFormer/tree/3da11c078c802b3f5a2ff0f6f8ec184a41537844), pinned to `3da11c078c802b3f5a2ff0f6f8ec184a41537844`.
- Five released `chordformer_head16(1.0,1.0)_s0.best.sdict` through `s4.best.sdict` checkpoints.
- On osu2017, the official `chord_recognition.py audio.mp3 output.lab` entry uses the five-model ensemble and default vocabulary.
- For Chords1217, use the fold in [the per-track record](chordformer_per_song.csv); averaging all five models would include models trained on that track and is invalid for this evaluation.
- Inference used dgx-071 GPU 7 (NVIDIA H800). Compatibility changes selected logical GPU 0 after external device masking and explicitly loaded trusted upstream checkpoints with `weights_only=False`.
- Two osu2017 songs, including the longest, were checked against the official entry point: identical chord labels and segment counts, with boundary differences below 5.1e-10 seconds from decimal serialization.

With one `<sample_id>.lab` per track in the reference and prediction directories:

```bash
python -m pip install numpy==1.26.4 scipy==1.13.1 mir_eval==0.7
python score_chordformer.py --dataset osu2017 --reference /path/to/reference --predictions /path/to/predictions
python score_chordformer.py --dataset Chords1217 --reference /path/to/chordlab --predictions /path/to/held_out_predictions
```

The scorer verifies the frozen reference hashes and complete cohort. Add
`--verify-archived-predictions` when using the archived LAB predictions to verify
their hashes too. [Machine-readable results](chordformer_results.json) and
[per-track scores, folds, and hashes](chordformer_per_song.csv) support independent checks.

## Attribution

Please cite [ChordFormer](https://arxiv.org/abs/2502.11840) and its upstream
[Large-Vocabulary Chord Transcription via Chord Structure Decomposition](https://archives.ismir.net/ismir2019/paper/000078.pdf),
as requested by the ChordFormer repository. Keeping this upstream attribution
does not retain the Jiang model as a current benchmark entry.