JAAH chord benchmark

Updated 2026-09-16. All 113 recordings in the Jazz Audio-Aligned Harmony (JAAH) dataset are evaluated with fixed models and the same reference intervals. Scores are percentages; higher is better. No model is trained or tuned on JAAH for this comparison.

Results

Model	Maj/min
ChordFormer	59.45
SheetSage2-Prober	62.94
SheetSage2-AR	64.50

AR improves over ChordFormer by 5.06 percentage points (95% CI: 1.29–7.97). The AR-minus-Prober difference is 1.56 points (95% CI: −0.44–4.15), so that comparison remains uncertain. Intervals use a paired song-level bias-corrected and accelerated (BCa) bootstrap, 20,000 resamples and seed 20260916. Each resample recomputes the reference-span-weighted statistic with the same songs and weights for both models. Bold denotes the highest point estimate for these fixed models and recordings, not a claim of JAAH state of the art.

Evaluation protocol

Cohort: all 113 original recordings; no track or chorus selection, reference-beat inputs, test-time pitch augmentation, global timing shift, or JAAH parameter tuning. The audio is from the supplied recordings, not reconstructed spectrograms.
References: byte-preserved official chord labels at JAAH commit 6fec3f6, intersected with each WAV's actual time span. Shorter annotations are not extended. This removes 9.146654 seconds of trailing N labels beyond the audio, leaving 24,991.969346 seconds. Five tracks have overruns above 20 ms and 37 have smaller rounding differences. The original labels are preserved.
Metric: per-track mir_eval.chord.majmin recall. Reference segments outside the metric's vocabulary receive -1 and are excluded from that track's metric denominator; they are never converted to N. Eligible reference duration, including N, is 93.87%.
Aggregation: weight each track's recall by its complete evaluated reference span. This is the same aggregation as the other SheetSage2 chord benchmarks. Macro averages and recall pooled by metric-eligible duration are retained in the JSON under separate names and are not the main-table values.
ChordFormer: five released Chords1217 fold models, component-probability ensemble followed by the official HMM, and the predeclared submission vocabulary, matching osu2017. On Chords1217 itself, each track instead uses only its held-out fold model. Source: official ChordFormer commit 3da11c0.
SheetSage2-Prober: fixed checkpoint, 24-kHz mono audio, 300-second windows, BF16, batch size one, native predicted beats with a fixed zero global offset; melody decoding disabled.
SheetSage2-AR: fixed paper checkpoint, 24-kHz mono audio, 300-second windows with 100-second overlap and predicted-prefix carry, BF16 autocast with FP32 weights, SDPA, greedy decoding, and a 5,120-token limit. Prompts: timestamp, downbeat_meter, chord_full. Two of 132 windows (blues_for_alice, lady_bird) reach the token limit and use the existing decoder's recovery for incomplete events. All 113 tracks are retained.

Using the untrimmed official annotation spans changes each main score by less than 0.03 percentage points. ChordFormer's other fixed vocabulary, full, is included only as an auxiliary result (59.11 Maj/min); the primary vocabulary was fixed before seeing JAAH scores. Independent rescoring with mir_eval 0.7 and 0.8.2 agreed on all 904 model/reference/track combinations.

A title-and-performer audit found no matches between JAAH and Chords1217. This does not establish absence from foundation-model pretraining or constitute an audio-fingerprint audit. Published JAAH results on selected choruses, rewritten labels, different vocabularies, or different train/test splits are not directly comparable to this full-collection evaluation. Only chord results are reported here.

Machine-readable results

Full-precision scores, reference variants, metric coverage, and checkpoint hashes
Per-track scores and prediction/reference SHA-256 hashes: 904 rows = 113 tracks × four model/vocabulary variants × two reference policies. Main results use reference_variant=audio_span and exclude ChordFormer_full_auxiliary.
Bootstrap intervals and paired differences
SheetSage2 paper preset and dataset prompts

Dataset citation

Vsevolod Eremenko, Emir Demirel, Baris Bozkurt, and Xavier Serra. “Audio-aligned jazz harmony dataset for automatic chord transcription and corpus-based research.” Proceedings of the 19th International Society for Music Information Retrieval Conference, 2018, pp. 483–490. DOI: 10.5281/zenodo.1291834.