Spectrograms & the STFT

Watch each short spectrum stand on end and drop into the picture as one column: a spectrum with a clock.

One spectrum, no clock

The spectrum of a whole recording, from The DFT (13.2), says which frequencies were present and how strong they were. It does not say when. The time of an event is stored in the phases of the spectrum, and there it is hard to read. A tone that played for the first second and a tone that played for the last one look alike in a plot of sizes.

The remedy is to look at the recording through a short window, one piece at a time. I cut the recording into short frames, multiply each by a window such as the Hann window of Window functions compared (15.2), take the FFT of each, and stand the spectra side by side. The result is the short-time Fourier transform, the STFT. Drawn with time across, frequency up and level as the strength of the ink, it is a spectrogram.

A frame has $N$ samples. The next frame starts $N_\text{hop}$ samples after this one, and $N_\text{hop}$ is the hop. Frame $m$ is centred at sample $mN_\text{hop}$, and the samples of $x$ outside the record count as 0:

This is the DFT of one windowed frame, with $w[n]$ the window. The index $m$ numbers the frames and $k$ numbers the bins, as in The DFT (13.2). The centre time of frame $m$ is

and the overlap between neighbouring frames is $N-N_\text{hop}$ samples. SciPy’s stft(..., boundary='zeros') makes the same frames, and it divides by $\sum w$.

I draw the level in dB re amplitude 1, using the amplitude scaling of Reading a spectrum: scaling and units (15.4): divide the size by half the window’s sum, so that a steady tone of amplitude 1 reads 0 dB.

A row of short spectra

The picture at the top of this page builds a spectrogram of a recording that plays 1000 Hz for 0.25 s and then 2000 Hz, at $f_s=8$ kHz, with a Hann window of $N=256$ (32 ms) and a hop of 128 (16 ms). Frame 0 shows 1000 Hz at −5.9 dB. Frame 5 holds a full window of the tone and reads 0 dB. Frame 16 straddles the change at 250 ms and holds 2000 Hz at −1.6 dB and 1000 Hz at −15.6 dB. The 33 frames, one column each, are the spectrogram: it shows what one spectrum of the whole record cannot, when each tone played.

Frame 0 is the easiest to check. Half its window lies before the record, where $x$ is 0, so the frame holds only half of the window’s weight. It sees about half the tone, and half an amplitude is $20\log_{10}0.5=-6.02$ dB, close to the $-5.9$ dB the instrument shows. Frame 16 is centred at 256 ms, so its samples are numbers 1920 to 2175. The change happens at sample 2000, so 80 of them belong to the 1000 Hz tone and 176 to the 2000 Hz tone. Both tones are in the column, and the 2000 Hz one is louder because more of the window holds it.

After the clip ends, tap a column to see that frame. Notice frame 16. One column holds both tones, because one window held both. A frame that contains a change has the spectrum of the change, not of either side alone.

Hop and overlap

Frames that do not overlap waste the samples near their ends, where the window is small. Anything that happens at a join between two frames is seen by neither at full weight. The figure shows $N=128$ Hann windows placed every 128, 64 and 32 samples, with their sum as a thick dashed line.

The Hann window is $w[n]=0.5-0.5\cos(2\pi n/N)$, and the sums are exact for a reason you can see in the cosines. At hop $N/2$ the second window is the first one shifted by half a period, so its cosine is the first one with the sign flipped. The two cosines cancel and the two constants add to 1. At hop $N/4$ four windows overlap, and their four cosines are spaced a quarter period apart, so they add to 0 and the four halves make 2.

So a hop of $N/2$ or $N/4$ is a good choice for a Hann window, and no moment falls between frames. The cost is the number of FFTs: a hop of $N/4$ makes twice as many frames as $N/2$, so it takes twice as many FFTs for the same record.

Short window or long window

Watch two clicks and two tones as the window grows: no length shows both pairs sharp.

One window length sets two things at once. A frame of $N$ samples spans $N/f_s$ seconds, which I call the time detail, $\Delta t$: events closer together than that fall in one frame. Its bins are $f_s/N$ apart, the frequency detail, $\Delta f$, as in Zero-padding and resolution (15.3): tones closer than that merge. The product is

Making one finer makes the other coarser. This is the pulse-width rule of Properties of the Fourier transform (8.2), applied to frames, and I use it as a rule of thumb, not as an exact bound.

The picture’s signal needs both kinds of detail at once: two steady tones, 1000 and 1100 Hz, and two clicks 10 ms apart, at 300 and 310 ms, sampled at 8 kHz. With $N=64$ (8 ms) the clicks stand apart, but the tones merge into one band. With $N=256$ (32 ms) the tones split into two lines, but the clicks blur into one stripe. With $N=1024$ (128 ms) the tones are sharp and the clicks are smeared over 128 ms.

The numbers follow from the two formulas. At $N=64$ a frame lasts $64/8000=8$ ms, shorter than the 10 ms gap, and the bins are $8000/64=125$ Hz apart. The tones are 100 Hz apart, which is 0.8 of a bin, so they cannot be told apart. At $N=256$ a frame lasts 32 ms and the bins are 31.25 Hz apart. The tones are 3.2 bins apart and split, but one window now holds both clicks.

After the clip ends, drag across the picture to set $N$ yourself, from 32 to 2048. I count two clicks as apart only when the picture dips by at least 3 dB between them. At 32 the tones merge and the clicks stand apart. At 128 both pairs merge, and at 512 and 2048 the tones split while the clicks merge. The hop stays $N/4$.

Try this: step from $N=64$ to 128 and then to 256. The clicks merge first, at 128, and the tones split only at 256, so at 128 neither pair is resolved. Between 32 and 2048, no length gives both a sharp pair of clicks and a sharp pair of tones.

Reading a chirp

Watch the strongest cell climb as the tone glides, then the probe land on the line.

A tone whose frequency changes draws a sloping line. A siren rising past you is a pitch that climbs, a line and not a dot. The chirp in this instrument glides at a steady rate, so its phase is

and its frequency at time $t$ is the derivative of the phase divided by $2\pi$:

It starts at 200 Hz and gains 1800 Hz every second, so at $t=2$ s it reaches 3800 Hz.

The picture draws that line, with the formula dashed over it. At 1.000 s the brightest cell is at 2000 Hz, on the line. After the clip ends, tap the picture to move the probe and read the brightest frequency at any moment.

The brightest cell is within half a bin of the formula, because the bins are 31.25 Hz apart and the probe picks one of them. At 1.496 s the formula gives $200+1800\cdot1.496=2892.8$ Hz, and the nearest bin is number 93, at $93\cdot31.25=2906.25$ Hz. The thickness of the line is the window’s blur. In one 32 ms window the tone moves $1800\cdot0.032=57.6$ Hz, which is $57.6/31.25\approx1.8$ bins, so one frame cannot say exactly which frequency the tone had.

For speech the usual choice is a 25 ms window every 10 ms. At $f_s=16$ kHz that is $N=400$, so $\Delta f=16000/400=40$ Hz, and a hop of $N_\text{hop}=160$ samples, which gives 100 frames per second. A 100 ms window would give 10 Hz bins but blur syllables. A 32 ms window gives bins 31.25 Hz apart at any sampling rate, because $\Delta f=1/(N/f_s)$. Frames are often zero-padded to the next power of two, as in Zero-padding and resolution (15.3). That makes a finer grid of the same detail.

Worked example

1. Frames. A record of 0.5 s at 8 kHz has 4000 samples. With $N=256$ and $N_\text{hop}=128$ the frames are centred at 0, 128, 256, and so on up to 4096, which makes 33 frames, one every 16 ms.
2. A frame at the change. Frame 16 is centred at 256 ms and holds samples 1920 to 2175. The change is at sample 2000, so 80 samples are the 1000 Hz tone and 176 are the 2000 Hz tone. It reads 2000 Hz at $-1.6$ dB and 1000 Hz at $-15.6$ dB.
3. The trade. At $N=64$ the bins are 125 Hz apart and the window is 8 ms. At $N=256$ they are 31.25 Hz apart and 32 ms. At $N=1024$ they are 7.8125 Hz apart and 128 ms. Each product is 1.
4. A chirp. At 1.000 s the formula gives $200+1800=2000$ Hz and the brightest bin is at 2000.00 Hz. At 1.496 s the formula gives 2892.8 Hz and the bin is at 2906.25 Hz.
5. Speech. A 25 ms window at 16 kHz has 400 samples and 40 Hz bins. A hop of 10 ms is 160 samples, so there are 100 frames per second.
6. Hop. A Hann window with a hop of $N/2$ sums to exactly 1, and with $N/4$ to exactly 2.

Where you’ll meet this

Spectrogram displays in audio software are STFTs, and many speech features start from one. Inverting the STFT (15.6) turns the picture back into sound, using the window sums of the figure on this page, so that you can edit the spectrogram and listen to the result. Filter banks (23.1) and Wavelets (23.2) come back to the trade of this page. The averaged periodograms of Averaged periodograms: Bartlett and Welch (25.2) average the columns of the spectrogram to get a steadier estimate.

Reference card