Wednesday, July 13, 2011

Installing FFTW3 with xcode and OpenFrameworks

Moving on from loading audio files with libsndfile (here and here), you'd probably like to perform some frequency domain analysis using Fourier transforms (that's a good video if you think of formulas as algorithms). It seems that the best/most popular library for FFT implementations is the FFTW3 library.

Installation of FFTW3 is very similar to the installation of libsndfile, in that you need to use macports (or homebrew) to install a universal build of the library for use with OpenFrameworks. As discussed earlier, a 'normal' install will most likely build an X86 64-bit library, where we need a 32-bit i386 library build.

Assuming you are using macports, type this into a terminal window:

> sudo port install fftw-3 +universal

Aside - A handy trick for seeing which architecture your library is built for is to use the 'lipo' command. Navigate to the folder where your library lives, and type this into a terminal window:


> lipo -info libfftw3.a


Anyways, the port install command will place the fftw3.h header file in your_HD/opt/local/include folder, and the libfftw3.a library in your_HD/opt/local/lib folder. To use the library in your OF project, add these two files to your project via the menu option Project -> Add to project...

Now, you can include the library in your source by adding the following include statement:


#include "fftw3.h"


The FFTW3 documentation is very thorough, but if you'd like some boiler plate code to see how to calculate a simple FFT, it can be hard to come by. As a warning, copy/pasting example code with this library seems pretty dicey as older versions of the library used considerably different syntax. Thankfully, this guy posted some very nice code for testing the operation of the library. I'm reposting it here after some simple explanation:

int SIZE : This is the size or length of the transform. FFTs are designed to work with transform sizes that are a power of 2. In this case, we just transform a simple signal that is 4 samples long that looks like this:

[1.0, 1.0, 1.0, 1.0]

data: This structure holds the data we want an FFT of. The FFTW library is built to allow for transforms of complex data, but audio data is real-valued only (no imaginary component). As such, we will only fill one 'side' of the structure with our real valued samples (check out the first for loop).

When you run this example, you should see the following output in your xcode console (Run->console) if everything is working with your install:


data[0] = { 1.00, 0.00 }
data[1] = { 1.00, 0.00 }
data[2] = { 1.00, 0.00 }
data[3] = { 1.00, 0.00 }

fft_result[0] = { 4.00, 0.00 }
fft_result[1] = { 0.00, 0.00 }
fft_result[2] = { 0.00, 0.00 }
fft_result[3] = { 0.00, 0.00 }

ifft_result[0] = { 1.00, 0.00 }
ifft_result[1] = { 1.00, 0.00 }
ifft_result[2] = { 1.00, 0.00 }
ifft_result[3] = { 1.00, 0.00 }


You can check the correctness by performing the equivalent commands in MATLAB:

>> data = [1.0, 1.0, 1.0, 1.0];
>> fft(data)
ans = 
4    0    0   0
>> ifft(ans)
1    1    1   1


Here's the example C code:

int SIZE = 4;
fftw_complex    *data, *fft_result, *ifft_result;
fftw_plan       plan_forward, plan_backward;
int             i;
data        = (fftw_complex*) fftw_malloc(sizeof(fftw_complex) * SIZE);
fft_result  = (fftw_complex*) fftw_malloc(sizeof(fftw_complex) * SIZE);
ifft_result = (fftw_complex*) fftw_malloc(sizeof(fftw_complex) * SIZE);
plan_forward  = fftw_plan_dft_1d(SIZE, data, fft_result,
                                 FFTW_FORWARD, FFTW_ESTIMATE);
plan_backward = fftw_plan_dft_1d(SIZE, fft_result, ifft_result,
                                 FFTW_BACKWARD, FFTW_ESTIMATE);

for( i = 0 ; i < SIZE ; i++ ) {
    data[i][0] = 1.0; // stick your audio samples in here
    data[i][1] = 0.0; // use this if your data is complex valued
    }

for( i = 0 ; i < SIZE ; i++ ) {
    fprintf( stdout, "data[%d] = { %2.2f, %2.2f }\n",
    i, data[i][0], data[i][1] );
}
fftw_execute( plan_forward );

for( i = 0 ; i < SIZE ; i++ ) {
    fprintf( stdout, "fft_result[%d] = { %2.2f, %2.2f }\n",
i, fft_result[i][0], fft_result[i][1] );
}
fftw_execute( plan_backward );

for( i = 0 ; i < SIZE ; i++ ) {
    fprintf( stdout, "ifft_result[%d] = { %2.2f, %2.2f }\n",
i, ifft_result[i][0] / SIZE, ifft_result[i][1] / SIZE );
}

fftw_destroy_plan( plan_forward );
fftw_destroy_plan( plan_backward );
fftw_free( data );
fftw_free( fft_result );
fftw_free( ifft_result );

Tuesday, July 12, 2011

Using ofSoundStream to play arbitrary audio data

If you're looking for the ability to pipe raw audio through OpenFrameworks, and aren't interested in using the built in ofSoundPlayer objects, start here. Grimus has put up some very nice code for what is essentially a framework for loading, playing, and effecting audio in a manner that'll be familiar and easy for those looking to do DSP.


If you just want a bare-bones explanation, I'll give you a little overview here. In your application's setup() method, you need to setup the ofSoundStream with a reference to your application, and some other information like the sample rate:


void testApp::setup(){
 
...


sampleRate = 44100;
lAudio = new float[256];
rAudio = new float[256];
ofSoundStreamSetup(2, 0, this, sampleRate, 256, 4);


...

}

In the above case, we are setting up the ofSoundStream with 2 output channels, 0 input channels, a reference to our application, the sample rate of 44100 hz, a buffer length of 256 samples, and 4 somethings (I forget what the four is for). Once you've done this, your application will periodically be asked to fill a buffer full of audio for output. To ensure that the application can properly respond to this request, you need to implement the following method in your testApp implementation (testApp.cpp or whatever you've called it). Make sure you also update your header file!



void testApp::audioRequested (float * output, int bufferSize, int nChannels){
    for (int i = 0; i < bufferSize; i++){
lAudio[i] = output[i*nChannels    ] = left_channel_sample_goes_here;
rAudio[i] = output[i*nChannels + 1] = right_channel_sample_goes_here;
    }
}


At this point, you are now responsible for filling up 256 (as defined above for the buffer size) samples of audio output every time the method is called. We save the outputted audio in the lAudio and rAudio buffers in case we'd like to graph the output or do something else nifty with recent output.

That's all there is to it!

Monday, July 11, 2011

Loading a wav File with SndfileHandle

Assuming you have libsndfile installed and properly building with your project (check here), it is time to read some audio from disk into an array. I think it's most convenient to use the C++ wrapper class 'SndfileHandle' defined in the sndfile.hh include as shown below:


#include "testApp.h"
#include <sndfile.hh>
#include <stdio.h>



void testApp::setup(){

.
.
.



const char* fn = "/Users/.../test.wav";
SndfileHandle myf = SndfileHandle(fn);
printf ("Opened file '%s'\n", fn) ;
printf ("    Sample rate : %d\n", myf.samplerate ()) ;
printf ("    Channels    : %d\n", myf.channels ()) ;
printf ("    Error       : %s\n", myf.strError());
printf ("    Frames      : %d\n"int(myf.frames())); 
puts("");



 static float buffer [1024]; 
 myf.read (buffer, 1024);
.
.
.



Aside - The 'frame' terminology initially confused me, as I tend to think of 'frames' as being chunks of multiple samples across the time axis (as in STFT analysis). In the libsndfile world however, a 'frame' is all of the values/samples for each channel at a given time instant. For example, if you have a stereo (2-channel) file, a frame will consist of two values or samples. If you have a mono (1-channel) file, a frame will consist of only one value or sample.

It's fairly non-obvious what is actually stored in our float buffer after the call to the SndfileHandle read() method. if you were dealing with MATLAB, you'd have a 2-dimensional array with each column holding the samples for either the right or left channel after a similar call to wavread(). You might even think it would make more sense for the read() method to return the samples corresponding to each channel separately. However, the read() method implemented in libsndfile fills up a 1-dimensional array of values.

For a mono (1-channel) audio file, the interpretation is simple. The buffer is filled with the audio samples just as you might expect:


buffer = [sample1, sample2, sample3, sample4, etc]


For a stereo (2-channel) audio file, the interpretation is less obvious. The buffer is filled with the audio samples in an interleaved fashion:


buffer = [left_sample1, right_sample1, left_sample2, right_sample2, etc]


This pattern continues for n-channel multichannel files.

If you want to inspect the contents of 'buffer' by hand, place a breakpoint after the read() function (double click in the wide column to the left of the text. A little blue tab/flag should appear). Next, go to Build -> Build and Debug, and wait until your program halts at the breakpoint. Now, go to Run -> Show -> Expressions. Enter the following in the the text field at the bottom of the dialogue:


*buffer @ 1024


Hopefully any newcomers to libsndfile and xcode will find this useful!

Installing LibSndFile with xcode + OpenFrameworks

If you're looking to load audio files (wav, aiff, etc.) and process them on a sample-by-sample basis for basic DSP needs, the best option available appears to be the 'libsndfile' library: http://www.mega-nerd.com/libsndfile/

I am not well versed when it comes to the unix terminal, and the notion of building libraries from source gives me the heebie jeebies. If you're like me, the next few paragraphs might be of help.  

To obtain the library, i used macports, but I have it on good authority that homebrew is a better choice for managing your code libraries. Assuming you've installed macports, you should enter the following commands into a terminal window. 

> sudo port - d selfupdate
> sudo port install libsndfile

Aside - If your mac has a Core 2 Duo processor or higher, it's very likely that your machine's architecture is 64-bit or 'X86_64' in the xcode parlance. If this is the case, the above command will build a X86_64 version of the library. This is all well and good, but the OpenFrameworks examples are configured to build for the i386 32-bit architecture. As such, you'll have to execute the following command to make sure the libsndfile library is built to work with the 32-bit OF libraries, and can play nicely with the OpenFrameworks code. Again, this is probably painfully obvious to any C/C++ programmer, but it wasn't immediately obvious to me. If you're encountering a build error along the lines of "libsndfile.dylib is built blah blah blah and not the target architecture i386", try to reinstall libsndfile with the +universal argument as shown below:

> sudo port install libsndfile +universal


Once this is complete you should open your OpenFrameworks project, or copy/paste/rename one of the existing example projects in the OpenFrameworks download. For instance, I used the 'soundPlayerFFTExample' project. In xcode, do the following: 

Project ->add to Project...

Add the following three files:
/your_mac_HD/opt/local/include/sndfile.h (the C header file)
/your_mac_HD/opt/local/include/sndfile.hh (the C++ wrappers for libsndfile)
/your_mac_HD/opt/local/lib/libsndfile.a (the library)

At the top of your main C/C++ file i.e. in 'testApp.cpp' you should add an include statement for the libsndfile library, and then some testing code in the body of the setup() method (Please make sure to actually specify a valid path to a wav file in the declaration of fn):

#include "testApp.h"
#include <sndfile.hh>
#include <stdio.h>


void testApp::setup(){

.
.
.


const char* fn = "/Users/.../test.wav";
SndfileHandle myf = SndfileHandle(fn);
printf ("Opened file '%s'\n", fn) ;
printf ("    Sample rate : %d\n", myf.samplerate ()) ;
printf ("    Channels    : %d\n", myf.channels ()) ;
printf ("    Error       : %s\n", myf.strError());
printf ("    Frames      : %d\n", int(myf.frames())); // frames is essentially samples
puts("");
.
.
.



Build and run your project, open your xcode console (Run -> console), and you should hopefully see something like this:

run
[Switching to process 11816]
Running…
Opened file '/Users/Jack/Documents/coles.wav'
    Sample rate : 44100
    Channels    : 2
    Error : No Error.
    Frames : 161943

If you get meaningful data for the sample rate, number of channels, and frame number, you're in business! If you get bogus values, ensure that the path you specified to the file is correct. Again, this is probably quite trivial for the C/C++ programmers coming to OpenFrameworks, but for the Processing, Flash, Java, and Matlab users, hopefully you'll find this helpful. 



Alternatives to OpenFrameworks

Before being introduced to OpenFrameworks, I spent some time looking into software packages that would allow me to create nice visuals and UI elements while still allowing for relatively serious mathematical processing. In particular, I needed something with basic DSP routines (FFT, IFFT, etc), and matrix mathematics libraries.  If you happen to be in a similar boat, and haven't heard of these packages...

(update: It was recently pointed out to me that Cinder is probably the most closely related alternative to OF if you are interested in the C++ route)

For the maths:
Matlab
Mathematica
Python + Numpy + Scipy
iOS 4 with the Accelerator Framework
Processing + JAMA library
MAX/MSP with FTM library

Easy GUI creation, plotting and drawing APIs:
iOS4
Processing
Flash/Flex
MAX/MSP
Python + MatPlotLib
Mathematica

Easy access to serious audio output/input libraries:
- MAX/MSP
- Flash/Flex + Tonfall Library
- ChucK

Please comment if you know of any similar tools