Music Analysis

Press Play, or step a note with the arrow keys. Drag or use turn and tilt; Center puts the view back. Version 1 bends pitch around a 200-degree arc. Version 2 puts the notes on the chromatic wheel, C on top, so octaves share a spoke. Version 3 shows every note being played as a ball, joins notes struck together into one form, and chains each triplet and melody note to its neighbours. Version 4 draws the recorded sound the way Lucio Arese's Seeing Birdsong draws a blackcap: every 23 ms of audio is a ball placed by its harmony or its timbre, joined in time order, with the spectrogram, chromagram and cepstrogram scrolling below. Version 5 puts Version 3's balls on a vertical scale like a piano roll: higher notes sit higher, one rung per semitone, and brighter-sounding notes sit nearer to you. Talk pins your spoken words to the music you talk over, and Stop and talk pins them to the last four seconds you heard.

View

turntilt

Notes

glowsizeopacityrelease

Attention

widthstays forshows ahead

Current

widthbrightnessstays connected

Star

sizemove timecurvetoward strongesttrailtrail width

Wheel

brightness

Bubble

strengthsize

Show

Live

trailshell

Notes

sizefade

Axes

time sizepitch sizewidth size

Balls

sharpnesssizenote fade

Tails

tail fade

Layout

width by

Timing

spreadspeedsync

Details

Time runs along the straight diagonal, one cross-section every triplet note (three per beat, 834 in all, bars 1 to 69), with a faint ring at each one and a brighter ring every four bars.
Version 2: around each cross-section the angle is the note's place on the chromatic wheel, C at the top and one semitone every 30 degrees clockwise as you ride forward, so every octave of a note points the same way. Each note is its own peak: its height is 3 times its loudness and its width runs from 0.18 radians for a whisper to 0.5 radians at full loudness; peaks on one spoke add up.
Version 1: around each cross-section the angle is pitch on a 200-degree arc: bass points down, the triplets point toward you, the melody points up, and every bulge has the same width. A note's bulge swells at its attack, holds, and shrinks on release.
Colour comes from pitch (red for low, violet for high); brightness comes from loudness; the rainbow sheen is a thin soap film whose thickness follows the note's overtones.
Each note's attention weight is half focus (how much it changed from the same spot one beat earlier, fading fast and dropping to a fifth when a higher note arrives), three tenths pull (how far it moved from the line's previous note), and one fifth intensity (loudness, brighter for higher notes).
The bass is scored against its own steps: its focus and pull are scaled by its largest step, and a new bass pitch gets a bonus that fades like focus, so a bass step wins its slice. Before this the triplets held the attention in 90 percent of slices and the bass in 6; now the bass holds it in 15.
Version 3 shows every note being played as a ball on its spoke: farther out for a higher note, bigger for a louder one, coloured by its own frequency like the film, with its name beside it (a dot marks the bass). Bass notes struck together (within 20 ms) are one body: two balls joined by a bar, three by a plate. Each triplet note and each melody note is a lone ball chained to its own line's ball before and after, so the note-to-note movement shows. The balls stay along the tube; the soap film starts off there and Bubble brings it back.
Threads shows the gold links (off at first): at each new note, a thin arc to the three notes it relates to most, brighter for a stronger link. Spread stretches time along the tube in every view, from half to four times, so notes sit further apart. Axes switches and sizes each axis's labels on its own.
Version 3 draws one ball per note at its attack; a straight trail runs from it along the time axis, as thick as the note is loud at each moment, down to nothing where the sound ends. Balls are joined by hourglass links, thinnest in the middle, colour flowing from one note to the next: rungs join notes struck together, rails join each line's next attack (the bass voice by voice). Rim hides the dotted outline of the current slice and Line the white line of attention and its dot. Labels, Balls, Chords (the rungs) and Time (the rails) switch independently. A tail always runs at the speed of the song for as long as the note really sounds; tail fade sets how fast it dims along its length (0 keeps it solid, the middle fades it to nothing at its end, the top within its first quarter). Note fade sets over how many of the following notes a ball, its tail and its links fade out (1 to 400, or never; 40 at first). Sync nudges the picture earlier or later against the sound; the picture already follows the audio clock minus the output latency (80 ms unless the browser reports its own).
Notes (Version 2) turns the hits on or off: the letters of the notes struck in the current slice, fading over 0.6 seconds; a dot under a letter marks the bass. Size and fade scale and dim those letters and, in Versions 3 and 5, the note names beside the balls.
Axes turns each axis's labels on or off and sets their size from half to twice: Time is the bar numbers; Pitch is the wheel's twelve letters (Versions 2 and 3), the arc's lowest and highest note (Version 1) or the letter column (Version 5); Width (Version 5) names the two ends of what width measures, dull and bright, narrow and wide, or soft and loud. Version 4's own Axes does the same for the charts' tick labels and for the 3D view's legend and labels. At any size two labels never sit on top of each other: the less important one is left out (the wheel's sharps before its naturals, most bar numbers before every fourth, the chromagram's sharps before C and the other naturals).
The white centre line bends toward the attention-weighted pitch (Version 2: toward the attention-weighted direction on the wheel). The gold threads are the three strongest links at each new note, weight = the two notes' weights multiplied; they fade as the music moves on.
Every note was rendered alone on the MuseScore grand piano to measure its real pitch, its overtones and its loudness over time; the audio you hear is the same piano playing all 1,169 notes of the first movement.
The whole first movement is here: 69 bars, 5 minutes 10 seconds at one steady 54 beats a minute (the last chord's pause is not stretched). Chords after bar 23 are named by the same rule; over the long G♯ in the bass (bars 28 to 40 and 63 to 65) a chord with A above G♯ is G♯7♭9, and bar 34's diminished chord is F𝄪 dim7.
Measured pitch: C#2 68.67 Hz, G#3 207.28 Hz, C#4 276.55 Hz, E4 329.48 Hz, G#4 414.86 Hz. Attack from 10 to 90 percent: bass C#2 43 ms, melody G#4 10 ms.
Ride puts the camera on the white line, a second behind the music by default, looking half a second past now. It stretches time twelve times along the line so the slices have room to be seen.
In Ride, slices ahead fade with distance, the current slice keeps a white outline, and slices behind stay lit by how loud their notes still ring, for about a second and a half past the camera. Gold threads stay while both of their notes still sound.
Each note pushes out its own bulge, with bass hanging below and melody rising on top.
The data is in app/field.json next to this page.
Version 5 draws Version 3's balls, tails, chord bodies and time links on a vertical chromatic scale: time runs along the axis, height is pitch at 0.25 units per semitone from F1 at the bottom to D♯6 at the top, and the scale is as wide as it is tall, with a faint line on every semitone and every bar, a brighter one on each C and every four bars, and the note letters in a gutter at the left of the view. An octave in the bass is a vertical rung.
Version 5 width (toward you) is the note's timbre. Brightness, the default, is the spectral centroid of the note's own sound, rendered alone on the same piano at the velocity and length it has in the piece, divided by its fundamental: from 0.9 times (a dull high note) to 12.6 times (a bright low one), on a log scale. Spread uses the spectral spread the same way, loudness the note's peak level, and off lays every note flat like a piano roll. Stretch widens each ball across by its spread, up to three times. The white line follows the attention-weighted pitch, and Follow tracks time without bobbing. Rim outlines the scale at now; in Ride the lines move to the left wall and the letters stand on the rim's left edge. A time link fades with the earlier of its two notes.
Version 6 shows a faint shell marking how far each note reaches at its attack. The bright live form grows at now with each note's measured loudness, fades while held, drops at release, and leaves a trail.
Version 7 puts notes around the circle of fifths, with C at the top. The rim grows with the beat's distance from lowest to highest pitch. The bright centre follows loudness and focus, with extra weight on the lowest note. Its path follows the changing harmony; lines point to the next note struck, and gold arcs mark turns over 20 degrees. Turn shows the centre's change of angle since the last beat. Move shows how far its position moved on a wheel of radius one. Agreement runs from zero for balanced directions to one for a shared direction. From home measures the angle away from the centre of C♯, E and G♯.
Frequency (Versions 3, 5 and 6, on at first) turns each ball into its frequency ball: harmonic k of the note's own render bulges out at latitude −90° + 45° × log₂(k), so the fundamental sits on the south pole, which points down the pitch axis, and 30 degrees per circle-of-fifths step around from the front; each bump is 25 degrees wide, 60 percent of the radius for the strongest harmonic and nothing at −40 dB. Off gives plain spheres.
Version 4 is measured from the recorded sound, not the note list. Every 23 ms (512 samples at 22,050 per second) it takes the spectral centroid (how bright the sound is), the spread, the flux (how fast the sound changes), the roll-off, the loudness, the share of each of the 12 notes (chroma) and 20 timbre numbers (MFCC).
Version 4 manifold: the 12 chroma values (Harmony) or timbre numbers 1 to 19 (Timbre) of each moment are folded into three dimensions by UMAP, using 10, 30 or 80 nearest neighbours, or by PCA. Moments that sound alike sit close together, and a grey line joins them in time order. Every note averages each triplet slice into one ball, 834 in all.
In Version 4 the white ball is now, the trail of the last seconds is brighter, older balls fade, and balls still to come show dimly until Show ahead is off, when the shape grows as the music plays, as in Lucio Arese's video. Chord feeling colours come from the same chord table as the Version 2 caption; Loudest note colours each moment by its strongest note.
Version 4 charts below: the sound wave around now, the spectrogram (48 steps per octave) with the spectral centroid in red, the chromagram, and the cepstrogram, blue below zero and red above. Now sits two thirds across and what is ahead is dimmed. The 3D spectrogram, chromagram (a tube round the chromatic wheel, C on top) and cepstrogram scroll the same window past a fixed playhead.
Version 4's data is in app/v4/v4-data.js, and its settings are kept apart from Versions 1 to 3.

Music Analysis i

Music Analysis