Separated Stems on Aligned Timelines

In the case of aligned timelines, usually there is only one track audible at a time, since each track may carry its own timeline. It is possible to configure the player to still provide simultaneous playback for tracks that live on the same timeline. In Source Separation for example, several stems belong to one timeline, and can be listened to together.

The player below therefore offers two levels of hierarchy. The first level picks the aligned performance, the second mixes the stems inside it. Each performance of Gefrorne Tränen was decomposed into a harmonic, a percussive and a residual stem; the three of them sum back to the recording.

Audio data: Schubert Winterreise Dataset, Gefrorne Tränen, D. 911, No. 3, performances HU33 and SC06.

Two levels of selection

soloGroup decides what a trackList contributes to the choice:

The first level holds every selectable timeline the lists contribute. Whatever is selected there sounds; everything else is silent.

• Both lists in a soloGroup — the player has one level. Every row of every list is a timeline, and exactly one of them plays. This is what the Aligned Timelines example does.
• One list in a soloGroup, one not — the first level holds that list’s N rows plus one entry for the whole second list. Selecting the second list makes its stems audible together, and its rows become the second level.
• Neither list in a soloGroup — the first level holds one entry per list, and each list carries its own stem mix. That is the player above.

Because alignment already selects one timeline at a time across the whole player, the lists that do declare a soloGroup all share one number; a second number would name the same selection.

A list that declares no soloGroup gets an extra row above its tracks. That row is the list’s own entry on the first level; its tracks are indented underneath it as the second level. Selecting a stem of an unselected list switches the player to that list. Its rows then toggle like an ordinary mixer, except that the last remaining stem stays audible — playback needs a timeline to run on. Switching away and back restores the mix the list was left with.

One timeline per list

Stems only mix if they really are one recording taken apart, so a trackList without a soloGroup requires all of its tracks to name the same alignment column and to carry the same startOffsetMs and endOffsetMs:

"alignment": {
  "src": "../../assets/alignment/alignment.csv",
  "referenceTimeline": "reference",
  "timelines": {
    "reference": "time_sync_reference",
    "score": "measure_Schubert_D911-03_2",
    "hu33Harmonic": "time_Schubert_D911-03_HU33",
    "hu33Percussive": "time_Schubert_D911-03_HU33",
    "hu33Residual": "time_Schubert_D911-03_HU33",
    "sc06Harmonic": "time_Schubert_D911-03_SC06",
    "sc06Percussive": "time_Schubert_D911-03_SC06",
    "sc06Residual": "time_Schubert_D911-03_SC06"
  }
}

The three HU33 stems share the HU33 column, and the three SC06 stems share the SC06 column. Both perform the same piece at their own tempo, and the alignment maps between them as it would for the undecomposed recordings — the sheet music above follows either performance.

The lists themselves declare nothing but their tracks and a title for the selection row. Leaving soloGroup out is what makes their stems mix:

{
  "type": "trackList",
  "title": "Performance HU33",
  "tracks": ["hu33Harmonic", "hu33Percussive", "hu33Residual"],
  "trackVolumeControls": true
}

The waveform view above the lists uses "tracks": "audible", so it draws whatever currently sounds: the stems of the selected performance, on that performance’s own time axis.

How the stems were made

The stems come from librosa’s harmonic/percussive source separation, applied to each recording’s spectrogram. What the two components leave over is kept as a third, residual stem, so the three of them add back up to the input:

import librosa

y, sr = librosa.load(path, sr=None, mono=True)
stft = librosa.stft(y, n_fft=2048, hop_length=512)

harmonic, percussive = librosa.decompose.hpss(stft, margin=(2.0, 2.0))
residual = stft - (harmonic + percussive)

for name, spectrum in {"h": harmonic, "p": percussive, "r": residual}.items():
    signal = librosa.istft(spectrum, hop_length=512, length=len(y))

A margin above 1.0 makes the separation stricter and pushes everything the masks disagree about into the residual, which is what makes that third stem worth listening to.