Cutting Your Teeth on FMOD Part 4: Frequency Analysis, Graphic Equalizer, Beat Detection and BPM Estimation

In this article, we shall look at how to do a frequency analysis of a playing song in FMOD and learn how to use this as the basis for a variety of useful applications.

What is frequency analysis?

Note: The SimpleFMOD library contains all of the source code and pre-compiled executables for the examples in this series.

Frequency analysis is where we take some playing audio and identify the volume level of each range of frequencies being played. In a simple example, this lets us identify the current volume of the bass, mid-range and treble in a song individually, or any other desired range of frequencies.

Stuff you don’t really need to know: The analysis is done using a process called Fast Fourier Transforms (or FFT), which looks back in time at all the recently played frequencies to build up a picture of the volume of each. The FFT covers the whole spectrum up to the sample rate of the song (typically 44.1kHz) – or specifically, the so-called nyquist rate of the song (half of the actual sample rate – the highest frequency which can be measured for the audio), but you can specify how many equal-sized ranges to break this up into. The number of ranges is known as the sample size. So, for example, a sample size of 100 on a song sampled at 44.1kHz will produce a bucket of 100 ranges, each covering 220.5Hz (that’s half the sample rate, divided by the sample size, ie. (44100 / 2) / 100). Therefore, the higher the sample size, the more accurate the measurement, but at the cost of lag since the FFT algorithm must search further back in time from the current playback position the more samples are taken into account.

Luckily, FMOD takes care of all this for you and all you need to do is tell it what sample size you want and it will return a float array containing a breakdown of the volume of each frequency range.

What is frequency analysis for?

Frequency analysis is the lowest level audio processing that must be performed to enable various other functionality. For example, you can use the resulting data to detect when there is a beat in a song (or other specific simple sound types). Here I am mostly concerned with video games and graphics, and it is a quite common effect to sync on-screen effects with the beat of a song. Using beat detection lets us trigger these effects in time with the music.

Once you have beat detection, you can then use the timing information of each beat to estimate the bpm (beats per minute) of the song. While this is generally unimportant for games, it is essential for high-level audio processing applications such as DJ tools which alter the bpm of two songs so they can be mixed (crossfaded) without a break in the music.

In this article, we will mostly look at how to use beat detection to trigger on-screen effects in games.

Retrieving the volume distribution

Down to business then. First, and very importantly, to enable frequency analysis you must use the software mixing flag when creating the sound or stream you want to analyse. This causes FMOD to mix the audio for the channel in software rather than passing it to a hardware-accelerated sound card, but DSP operations such as frequency analysis are only allowed in FMOD when software mixing is used, so we have no choice. Create your sound or stream as follows:

FMOD::Sound *song;
system->createStream("Song.mp3", FMOD_SOFTWARE, 0, &song); // or createSound

You will also need to retrieve the channel of the song once it starts playing as this is used as a handle to the frequency analysis function:

FMOD::Channel *channel;
system->playSound(FMOD_CHANNEL_FREE, song, true, &channel);

All pretty familiar so far. You can now do a frequency analysis at any time. If you are planning to draw a graphic equalizer (VU bars) from the results, or use beat detection, you will want to do this on every frame, so in your per-frame update code, first update FMOD, then perform the analysis:

// Per-frame FMOD update ('system' is a pointer to FMOD::System)
system->update();

// getSpectrum() performs the frequency analysis, see explanation below
int sampleSize = 64;

float *specLeft, *specRight;

specLeft = new float[sampleSize];
specRight = new float[sampleSize];

// Get spectrum for left and right stereo channels
channel->getSpectrum(specLeft, sampleSize, 0, FMOD_DSP_FFT_WINDOW_RECT);
channel->getSpectrum(specRight, sampleSize, 1, FMOD_DSP_FFT_WINDOW_RECT);

The first step is to allocate memory to a float array, then pass a pointer to it to getSpectrum() in which to retrieve the volume distribution. The sample size must be a power of two in the range 64-8192. The third argument specifies which part of the audio to examine; for a stereo track, 0 represents the left channel and 1 the right channel. Here we retrieve the volume distribution for both. The fourth argument specifies a smoothing filter to use to help guard against false readings (false transients). FMOD_DSP_FFT_WINDOW_RECT uses a rectangular filter, which essentially means everything is allowed through.

To get the average volume distribution for a stereo track we need to take the average of the left and right channels:

float *spec;

spec = new float[sampleSize];

for (int i = 0; i < sampleSize; i++)
    spec[i] = (specLeft[i] + specRight[i]) / 2;

Depending on what you want to do with the data, you now have a choice. The floats returned by getSpectrum are in dB (decibels) with a range of 0-1 where 1 is the loudest possible output and 0 is silence. Many of these values may be quite low even when the music is playing at maximum volume, so you can optionally normalise (scale) the data such that the loudest frequency is always represented by 1. To do this, find the maximum of all the volumes returned and then divide this into each volume as follows:

// Find max volume
auto maxIterator = std::max_element(&spec[0], &spec[sampleSize]);
float maxVol = *maxIterator;

// Normalize
if (maxVol != 0)
  std::transform(&spec[0], &spec[sampleSize], &spec[0], [maxVol] (float dB) -> float { return dB / maxVol; });

There are many ways to implement this but the combination of the C++ Standard Library template functions and a lambda function in C++11 is quite neat. The transform function performs an in-place transform of each volume to scale it relative to the maximum volume for the distribution.

Again optionally, you can calculate the range in Hz of frequencies covered by each array entry as follows, if you need it:

float hzRange = (44100 / 2) / static_cast(sampleSize);

Don’t forget to change 44100 to the sample rate (in Hz) of the audio.

You can now do whatever you want with the data. Don’t forget to clean up at the end of your function:

delete [] spec;
delete [] specLeft;
delete [] specRight;

Plotting a graphic equalizer (VU bars)

Figure 1. VU bars. In this case the sample size is 128 on a track sampled at 44.1kHz, therefore there are 128 bars representing a range of 172.266Hz each. The lowest frequency ranges are shown on the left, with the height of each bar representing its average volume.

VU bars are usually represented by a row of rectangles (or lines) with a common bottom Y co-ordinate, with the height of each rectangle representing the volume of the frequency range it represents – see Figure 1.

How you plot this depends on the graphics engine you are using. If you want to ensure that at least one bar (the loudest) is always at the maximum height, you should normalize the frequency data before plotting. Here is some example code for my Simple2D library which uses 80% of the screen width with a 10% border at the left and right, and scales the width and gap between each bar according to the sample size:

// Earlier in code...
freqGradient = MakeBrush(Colour::Green, Colour::Red);

...
// VU bars

int blockGap = 4 / (sampleSize / 64);
int blockWidth = static_cast((static_cast(ResolutionX) * 0.8f) / static_cast(sampleSize) - blockGap);
int blockMaxHeight = 200;

// Parameters: Left-hand X co-ordinate of bar, left-hand Y co-ordinate of bar, width of bar, height of bar (negative to draw upwards), paintbrush to use

for (int b = 0; b < sampleSize - 1; b++)
    FillRectangleWH(static_cast(ResolutionX * 0.1f + (blockWidth + blockGap) * b),
                    ResolutionY - 50,
                    blockWidth,
                    static_cast(-blockMaxHeight * spec[b]),
                    freqGradient);

This produces a display identical to that shown in figure 1 when placed in your application’s DrawScene() function.

Beat detection

The principle of detecting when a beat occurs in the music is to examine a low frequency range (where the percussion occurs – typically 60-120Hz for a bass kick drum and 120-150Hz for a snare drum) to see if its volume exceeds a certain threshold value. In the following example, we simply consider the beat to have occurred when this threshold is exceeded, then ignore the volume of the track for a given period of time afterwards to avoid false positives. We also simply examine the lowest bar in the volume distribution which works for small sample sizes but will be looking at too low a frequency range for larger sample sizes. Ideally you should look at the frequency ranges above, but I leave that as an exercise for you. With a sample size of 128 and a track sampled at 44.1kHz, looking at the first item in the array will cover all frequencies from 0-172Hz, so it’s a reasonable estimate. A more sophisticated approach may aggregate the average of several bars from a larger sample size, and require the threshold to be exceeded for more than a single frame.

Normalization should not be used when coding for beat detection, as it disproportionately distorts the volume distribution during quiet periods of the music.

We need to set some variables to start with. Through trial and error, I found that these work reasonably well for dance music:

float beatThresholdVolume = 0.3f;    // The threshold over which to recognize a beat
int beatThresholdBar = 0;            // The bar in the volume distribution to examine
unsigned int beatPostIgnroe = 250;   // Number of ms to ignore track for after a beat is recognized

int beatLastTick = 0;                // Time when last beat occurred

Here is how we detect a beat:

bool beatDetected = false;

// Test for threshold volume being exceeded (if not currently ignoring track)
if (spec[beatThresholdBar] >= beatThresholdVolume && beatLastTick == 0)
{
  beatLastTick = GetTickCount();
  beatDetected = true;
}

if (beatDetected)
{
  // A beat has occurred, do something here
}

// If the ignore time has expired, allow testing for a beat again
if (GetTickCount() - beatLastTick >= beatPostIgnore)
  beatLastTick = 0;

This code can of course be adapted in a variety of ways, but as it stands, beatLastTick will retain the system tick time of the last detected beat while the audio is being ignored, and 0 at all other times. On the first frame that the beat is detected, the trigger code will execute and you can generate on-screen effects or user interactions here.

BPM Estimation

While not really needed for game development, I thought it would be interesting to include this by way of example. At the start, it should be noted that there are many ways to use beat detection data to calculate bpm – some more accurate than others – and the example below only produces makeshift estimates; it is not the best method. Also, bpm estimation relies on perfect beat detection, which is unlikely to be produced by the unsophisticated code above.

The correct way of determining a song’s bpm: The song should be scanned from start to end in memory without playing it, and a frequency analysis performed on each frame. Beat detection should be performed on each frame from the frequency analysis results, and the relative times of every beat in the song stored in an array. Heuristics should be used to ignore portions of the song with no beat, and the remaining portion of the song should have its playback time divided into the number of beats. This will provide a fairly accurate calculation of the song’s true bpm.

In our example, we will use a moving average bpm estimation over the previous 10 seconds of the playing track. First, we’ll need storage for the times of all the beats for the last 10 seconds:

// List of how many milliseconds ago the last beats were
std::queue beatTimes;

// The number of milliseconds of previous beats to keep in the list
unsigned int beatTrackCutoff = 10000;

We’ll then need to modify the beat detection code above to keep the list of detected beats up to date:

if (spec[beatThresholdBar] >= beatThresholdVolume && beatLastTick == 0)
{
    beatLastTick = GetTickCount();
    beatDetected = true;

    // Store time of detected beat
    beatTimes.push(beatLastTick);

    // Remove oldest beat if it is older than the cut-off time
    while(GetTickCount() - beatTimes.front() > beatTrackCutoff)
    {
        beatTimes.pop();
        if (beatTimes.size() == 0)
            break;
    }
}

The rest of the code remains the same. Now we have our updating list of beat times, we can do a simple calculation to estimate the track’s bpm:

// Predict BPM

float msPerBeat, bpmEstimate;

if (beatTimes.size() >= 2)
{
  msPerBeat = (beatTimes.back() - beatTimes.front()) / static_cast(beatTimes.size() - 1);
  bpmEstimate = 60000 / msPerBeat;
}
else
  bpmEstimate = 0;

What happens here is that we take the times of the oldest and newest beats in the list, and divide them by the size of the list minus one (which is the number of gaps between the beats rather than the number of beats) to get the average number of milliseconds between each beat. We then divide this into one minute (60000 milliseconds) to get the estimate of the bpm, which is stored in bpmEstimate.

Demo application

Here you can download a fully fleshed out demo using all the techniques above plus a few other tweaks. The full source code is presented below – note that Simple2D is used for rendering so the application follows the Simple2D framework. Please forgive the choice of music 🙂

To try the application, first press P to unpause the music after running the EXE file. You can use N to toggle normalization (beat detection and bpm estimation only run when normalization is off) and 1 and 2 to increase and decrease the FFT sample size. When normalization is off, the word ‘BEAT’ flashes on the screen when a beat is detected, and the current bpm estimation is shown at the bottom of the screen.

Full source code:

// FMOD Frequency Analysis demo
// Written by Katy Coe (c) 2013
// No unauthorized copying or distribution
// www.djkaty.com

#include "../SimpleFMOD/SimpleFMOD.h"
#include "Simple2D.h"

#include <queue>

using namespace SFMOD;
using namespace S2D;

class FrequencyAnalysis : public Simple2D
{
private:
	// FMOD
	SimpleFMOD fmod;
	Song song;

	// Graphics
	TextFormat freqTextFormat;
	Gradient freqGradient;

	// Normalization toggle and sample size
	bool enableNormalize;
	int sampleSize;

	// Beat detection parameters
	float beatThresholdVolume;
	int beatThresholdBar;
	unsigned int beatSustain;
	unsigned int beatPostIgnore;

	int beatLastTick;
	int beatIgnoreLastTick;

	// List of how many ms ago the last beats were
	std::queue<int> beatTimes;
	unsigned int beatTrackCutoff;

	// When the music was last unpaused
	int musicStartTick;

public:
	FrequencyAnalysis();
	void DrawScene();

	virtual bool OnKeyCharacter(int, int, bool, bool);
};

// Initialize application
FrequencyAnalysis::FrequencyAnalysis()
{
	// Make paintbrushes
	freqTextFormat = MakeTextFormat(L"Verdana", 10.0f);
	freqGradient = MakeBrush(Colour::Green, Colour::Red);

	song = fmod.LoadSong("Song.mp3", FMOD_SOFTWARE);
	song.Start(true);

	// Load song
	enableNormalize = true;
	sampleSize = 64;

	// Set beat detection parameters
	beatThresholdVolume = 0.3f;
	beatThresholdBar = 0;
	beatSustain = 150;
	beatPostIgnore = 250;
	beatTrackCutoff = 10000;

	beatLastTick = 0;
	beatIgnoreLastTick = 0;

	musicStartTick = 0;
}

// Handle keypresses
bool FrequencyAnalysis::OnKeyCharacter(int key, int rc, bool prev, bool trans)
{
	// Toggle pause
	if (key == 'P' || key == 'p')
	{
		song.TogglePause();

		// Reset bpm estimation if needed
		if (musicStartTick == 0 && !enableNormalize && !song.GetPaused())
		{
			musicStartTick = GetTickCount();
			beatTimes.empty();
		}

		else if (song.GetPaused())
			musicStartTick = 0;
	}

	// Toggle normalization
	if (key == 'N' || key == 'n')
	{
		enableNormalize = !enableNormalize;

		// Reset bpm estimation if needed
		if (!enableNormalize && !song.GetPaused())
		{
			musicStartTick = GetTickCount();
			beatTimes.empty();
		}
	}

	// Decrease FFT sample size
	if (key == '1')
		sampleSize = max(sampleSize / 2, 64);

	// Increase FFT sample size
	if (key == '2')
		sampleSize = min(sampleSize * 2, 8192);

	return true;
}

// Per-frame code
void FrequencyAnalysis::DrawScene()
{
	// Update FMOD
	fmod.Update();

	// Frequency analysis
	float *specLeft, *specRight, *spec;
	spec = new float[sampleSize];
	specLeft = new float[sampleSize];
	specRight = new float[sampleSize];

	// Get average spectrum for left and right stereo channels
	song.GetChannel()->getSpectrum(specLeft, sampleSize, 0, FMOD_DSP_FFT_WINDOW_RECT);
	song.GetChannel()->getSpectrum(specRight, sampleSize, 1, FMOD_DSP_FFT_WINDOW_RECT);

	for (int i = 0; i < sampleSize; i++)
		spec[i] = (specLeft[i] + specRight[i]) / 2;

	// Find max volume
	auto maxIterator = std::max_element(&spec[0], &spec[sampleSize]);
	float maxVol = *maxIterator;

	// Normalize
	if (enableNormalize && maxVol != 0)
		std::transform(&spec[0], &spec[sampleSize], &spec[0], [maxVol] (float dB) -> float { return dB / maxVol; });

	// Find frequency range of each array item
	float hzRange = (44100 / 2) / static_cast<float>(sampleSize);

	// Detect beat if normalization disabled
	if (!enableNormalize)
	{
		if (spec[beatThresholdBar] >= beatThresholdVolume && beatLastTick == 0 && beatIgnoreLastTick == 0)
		{
			beatLastTick = GetTickCount();
			beatTimes.push(beatLastTick);

			while(GetTickCount() - beatTimes.front() > beatTrackCutoff)
			{
				beatTimes.pop();
				if (beatTimes.size() == 0)
					break;
			}
		}

		if (GetTickCount() - beatLastTick < beatSustain)
			Text(100, 220, "BEAT", Colour::White, MakeTextFormat(L"Verdana", 48.0f));

		else if (beatIgnoreLastTick == 0 && beatLastTick != 0)
		{
			beatLastTick = 0;
			beatIgnoreLastTick = GetTickCount();
		}

		if (GetTickCount() - beatIgnoreLastTick >= beatPostIgnore)
			beatIgnoreLastTick = 0;
	}

	// Predict BPM
	float msPerBeat, bpmEstimate;

	if (beatTimes.size() >= 2)
	{
		msPerBeat = (beatTimes.back() - beatTimes.front()) / static_cast<float>(beatTimes.size() - 1);
		bpmEstimate = 60000 / msPerBeat;
	}
	else
		bpmEstimate = 0;

	// Draw display
	Text(10, 10, "Press P to toggle pause, N to toggle normalize, 1 and 2 to adjust FFT size", Colour::White, MakeTextFormat(L"Verdana", 14.0f));

	Text(10, 30, "Sample size: " + StringFactory(sampleSize) + "  -  Range per sample: " + StringFactory(hzRange) + "Hz  -  Max vol this frame: " + StringFactory(maxVol), Colour::White, MakeTextFormat(L"Verdana", 14.0f));

	// BPM estimation
	if (!enableNormalize)
	{
		if (GetTickCount() - musicStartTick >= beatTrackCutoff && musicStartTick != 0)
			Text(10, ResolutionY - 20, "Estimated BPM: " + StringFactory(bpmEstimate) + " (last " + StringFactory(beatTrackCutoff / 1000) + " seconds)", Colour::White, MakeTextFormat(L"Verdana", 14.0f));
		else if (musicStartTick != 0)
			Text(10, ResolutionY - 20, "Estimated BPM: calculating for next " + StringFactory(beatTrackCutoff - (GetTickCount() - musicStartTick)) + " ms", Colour::White, MakeTextFormat(L"Verdana", 14.0f));
		else
			Text(10, ResolutionY - 20, "Paused", Colour::White, MakeTextFormat(L"Verdana", 14.0f));
	}
	else
		Text(10, ResolutionY - 20, "Disable normalization to enable BPM calculation", Colour::White, MakeTextFormat(L"Verdana", 14.0f));

	// Numerical FFT display
	int nPerRow = 16;

	for (int y = 0; y < sampleSize / nPerRow; y++)
		for (int x = 0; x < nPerRow; x++)
			Text(x * 40 + 10, y * 20 + 60, StringFactory(floor(spec[y * nPerRow + x] * 1000)), Colour::White, freqTextFormat);

	// VU bars
	int blockGap = 4 / (sampleSize / 64);
	int blockWidth = static_cast<int>((static_cast<float>(ResolutionX) * 0.8f) / static_cast<float>(sampleSize) - blockGap);
	int blockMaxHeight = 200;

	for (int b = 0; b < sampleSize - 1; b++)
		FillRectangleWH(static_cast<int>(ResolutionX * 0.1f + (blockWidth + blockGap) * b),
						ResolutionY - 50,
						blockWidth,
						static_cast<int>(-blockMaxHeight * spec[b]),
						freqGradient);

	// Clean up
	delete [] spec;
	delete [] specLeft;
	delete [] specRight;
}

void Simple2DStart()
{
	FrequencyAnalysis test;
	test.SetWindowName(L"FMOD Frequency Analysis");
	test.SetBackgroundColour(Colour::Black);
	test.SetResizableWindow(false);
	test.Run();
}

I hope you found this exploration of frequency analysis in FMOD useful! In Part 5 we’ll check out how to generate audio on the fly from user-defined functions. If you want to do frequency analysis of real-time sound card output, check out Part 6 for the details! Enjoy.