---
license: other
library_name: pytorch
tags:
- audio
- representation-learning
- jepa
- vital
datasets:
- duckking032/vital
---

# Vital Audio JEPA V2

Audio JEPA encoder trained from scratch on 74,685 parameter-aware renders made
with the Vital 1.6.4 standalone renderer.

## Input

- 22,050 Hz mono PCM audio
- exactly 6.0 seconds
- 128-bin log-mel spectrogram
- 1024-sample FFT and 256-sample hop
- 16x16 spectrogram patches: an 8x32 grid with 256 tokens

## Training data

The training corpus is published under
`duckking032/vital/augmented_v1_22050_mono_6s`. It contains 74,685 accepted
renders derived from 7,394 unchanged parent presets. Variants include
category-aware MIDI notes and deterministic mutations of macros, filters,
amplitude envelopes, oscillator timbre, unison, active modulation, and active
effects.

All variants of a parent preset are assigned to the same split. The model saw
70,974 clips from 7,022 parents during training; validation used 3,711 clips
from 370 entirely unseen parents.

## Objective and training

The online encoder receives 40% context patches. A predictor reconstructs 40%
target patches in the latent space of an exponential-moving-average target
encoder.

- 35,690,112 total parameters; 21,493,632 trainable
- batch size 64, bfloat16, AdamW
- 30 epochs / 33,240 optimizer steps
- RTX PRO 4500 Blackwell
- training time: 2,792.6 seconds (46.5 minutes)

The selected checkpoint is step 15,000. Later checkpoints achieved lower
masked loss but weaker semantic retrieval, so selection used cross-parent
same-category retrieval rather than loss alone.

## Controlled evaluation

Evaluation uses 3,711 variants from 370 held-out parents. A query is forbidden
from retrieving another variant of its own parent, preventing augmented
siblings from inflating the scores.

| Model | Masked val loss | Top-1 | Top-5 | MRR@10 |
|---|---:|---:|---:|---:|
| Published V1 | 0.10759 | 32.47% | 54.11% | 0.4220 |
| V2 step 15,000 | 0.08507 | 35.65% | 56.53% | 0.4494 |

The validation set was also used for checkpoint selection, so these metrics
should not be treated as a final untouched-test estimate. A future release
should reserve a third parent-held-out test split.

## Intended use

This model is a retrieval and initialization baseline for editable Vital sound
design. It is not an exact audio-to-preset reconstruction system, and category
retrieval is only a coarse proxy for perceptual usefulness.

`checkpoints/jepa_v2/selected.pt` is the optimizer-free selected checkpoint.
See `eval_runs/jepa_v2/` for the full comparison, controlled V1 baseline, V2
evaluation report, and validation embeddings.