Manipulating waveforms live in-game

Gnarlyman 101 Jan 13, 2012 at 03:17


I’m looking for a way to manipulate sound using code in the following manner. Basically, when one uses some sort of audio software, you are able to open an audio file and adjust several hundred different audio properties in the program via sliders, buttons, value inputs, etc. What I would like to do is have the ability to do all of that in-code. I.e., I have a “generic” audio file, and based on user input, the program can intelligently apply various quantities and degrees of all those bajillions of effects to the file.

Also, is there a way to get down ‘n dirty and actually mess with the waveform itself? Basically, the goal would be to apply all the same aforementioned effects but do so by actual root-mathematical manipulation of a very generic audio file. Not really sure how to ask this, as I’m new at digi audio and don’t really know how it all works. However, the main thing is, I want to be able to have code that can
intelligently edit audio on the fly as I myself would in an actual sound software interface.

Thanks for any help!

13 Replies

Please log in or register to post a reply.

Reedbeta 167 Jan 13, 2012 at 03:56

Yes, you absolutely can generate and/or process the waveform directly, in real time. It’s a fascinating area of programming. If you’re interested in getting your feet wet with it in a not-too-demanding programming environment, you might try SuperCollider. If you want to do it more directly in C/C++, and you’re using Windows, you can use the Waveform Audio API (part of the Windows API). This API allows you to generate buffers of waveform data using any algorithm you like and send them to the sound chip to play. You can probably find some articles about it online and/or read the docs to figure out how to make it work.

Stainless 151 Jan 13, 2012 at 11:16

Also look at the mod file format and source code for mod players.

Mod files were created as a way of getting sampled sound into games without a massive memory overhead.

So they work by taking a single sample, say of a piano playing a middle C, then changing the playback rate to generate other notes.

Simple really, and very effective.

The guy who wrote the music for Cannon Fodder (was it Rob?) used his own voice for the lyrics, then upsampled it to sound like a woman.

Worth looking at for some cool techniques.

TheNut 179 Jan 13, 2012 at 13:36

Two other areas you can look into are MIDI and VST. MIDI is a specification for working with digital instruments and VST (look into version 2.X, not 3.X) is a plugin format which is basically its own little software synthesizer, most commonly used as a soundfont, generate effects, or filter audio input. The two are really great specs for getting an inside look to building your own synthesizer. I based my own synthesizer off these two and added a couple extras like ADSR envelope and sound tracks. The possibilities are limitless.

As for performance, on today’s CPUs it’s a walk in the park. If you want to get real creative you could even do this on the GPU at blazing speeds. With my synthesizer, I can mix two 44KHz triple oscillators in real-time on a mobile phone, or about nine 11KHz oscillators.

Gnarlyman 101 Jan 13, 2012 at 16:09

This is absolutely terrific. Thanks you all! I assume that all of these said techniques/tools can be easily put to the task of “3D” soundspace? I’m also assuming that that’s a function of programming (therefore it can be done in-game, in-code).

Stainless 151 Jan 14, 2012 at 10:44

Most sound API’s have two play functions.

One is for simple, standard, mono or stereo.

The other includes a 3d vector for the position of the sound in 3d space.

In principle it is trivial to put a sound source anywhere you like. In practice it can be tricky.

The problem comes in the equation the api programmer has used for attenuation. Often you find that you have to scale the position to fit into the designed space.

However that is a problem we can help you with when you get further down the line.

Gnarlyman 101 Jan 15, 2012 at 04:57

Thanks Stainless, very helpful there. Yes, I’ll certainly give a calling card when I need more help! This is all great stuff to start with. Thanks again.

Gnarlyman 101 Jan 15, 2012 at 22:54

PS–how much sound is generated, these days, in top-end games? In other words, beyond the recorded dialogue, what percentage of audio is recorded then edited in game on the fly, versus generated completely in-game in pure coding? I realize there’s not a standard %; I’m just curious about how intensive the recording side and the in-game coded audio side are in a modern-tech game.

Reedbeta 167 Jan 16, 2012 at 00:46

I believe in most games the sounds are almost entirely pre-recorded, and what happens at runtime is essentially just playing specified sounds when an event fires (e.g. a gunshot) and mixing all the currently-playing sounds together.

There are a few things layered on that, such as having multiple versions of a sound that are randomly chosen (e.g. several different gunshot sounds for each gun, so that not all shots from that gun sound exactly alike). Certain sounds might have pitch-shifting or other adjustable parameters that can be changed by the game engine, e.g. a helicopter rotor sound that changes with speed. Some voice lines might get processed to make them sound like they’re coming through a cell phone or walkie-talkie, although this can also be done as a preprocess.

There’s probably very, very little direct synthesis of sounds from scratch at runtime.

Gnarlyman 101 Jan 16, 2012 at 05:00

Hmm, I see. That’s interesting. I figured that there’d be a little more on-the-spot synthesis. Whatever the case, I’m gonna work on that myself!

Stainless 151 Jan 16, 2012 at 10:32

Most of the time you get the sound effects in memory, and the music on the DVD.

There is some buffering on the music player so that you don’t get drop outs when you access the DVD for data reads.

The XBOX only has 512M of ram, so you have to do something like this.

One of the things that is done a lot (and I hate when they get it wrong) is to try and link the music to the game events.

So if you are wandering through a forest you get Jethro Tull, then when you get attacked by the big bad wolf you hear Metallica.

Which is fine, and can even be useful when you don’t know a wolf is coming up behind you but the music has suddenly gone metal.

But when you kick the wolf of a cliff and it’s stuck 100 meters away and cannot get to you and the music is still blasting your ears because it’s not dead and so you must still be in combat with it, or you are level 129,001 and a wolf only takes a blink of the eye to kill, it’s annoying.

TheNut 179 Jan 16, 2012 at 18:16

Synthesized sounds work great for 8-bit games. Nothing beats a good chiptune :D

You can pull off purely synthesized sounds in a music touchpad game. Something like this. It’s very easy to produce sounds that are pleasing on the ears. You could also synthesize some musical instruments fairly well. DSK Music produces a lot of free, great VST plugins that do just this.

For other gaming though, I don’t think realtime synthesis would be used (not to be confused with realtime DSP, which is widely used). You could produce sounds in advanced using a synth, but it would be difficult and time consuming finding an algorithm to match the type of sound effect you’re aiming for. Libraries like OpenAL and FMOD provide a nice API that allows you to plugin custom effects applied to your sounds at runtime.

Gnarlyman 101 Jan 16, 2012 at 22:37

Thanks for the input peoples, great stuff again. I figured that games do a lot of prerecorded stuff. I myself would like to do what probably amounts to an uncommon percentage of realtime production and synth based on/derived from game logic which I’m developing myself. The good thing is, the type of sound I’m looking for is quite similar to that uber-sweet music touchpad game T.N. linked to above. Basically, I’d like to be able to take a handful of basic tones and have a realtime effects engine run synth on it based on said logic I’m cooking up. I don’t plan on trying to produce the rustle of grass or the flap of wings or something, but the soundscape and style of the world itself is very “tone-ish”, so I think I could pull it off. Yah, doable?

bschmidt 101 Feb 27, 2012 at 20:32

I sent out a brief survey a few months ago to a mailing list with a lot of AAA game audio developers on it..
They said for current AAA titles, around 50% of the non-dialog, non-music ‘sounds’ were something more complicated than “just play a wave file.” Most often, it meant layering of multiple wav files to create one “sound.” Games are also starting to use a lot of run-time DSP (digital signal processing effects, using either built-in effects in things like FMOD or WWise, or licensed from a 3rd party. (Some of the ‘pro audio’ dsp companies such as McDSP, WaveArts and Izotope are now licensing their algorithms to game developers).
Taking one or more wav files and processing them in real-time, tying the DSP parameters to the game physics engine can create some really great results..

And some games are have started basic physical synthesis algorithms for some sound effects– think metallic ‘bangs’, wind wooshes, etc. WWise has a couple of physical modelling packages they make available.

Brian Schmidt
Brian Schmidt Studios
Executive Director, GameSoundCon