Intel Spatial Audio for Java* Package User's Guide

Contents

Overview

The Intel Spatial Audio for Java Package provides a simple interface for audio rendering. The package was designed to make the addition of audio to 3D environments very straightforward. In a typical 3D application, a scene is described, and a view of the scene is selected by specifying a camera. When components in the scene need to change, the application will inform the rendering engine, and a new image from the defined camera's perspective is rendered.

Support for audio rendering follows the same model typically used by graphics libraries. An audio scene is described with objects that emit sounds, called sound sources. A listener, analogous to the graphical camera, defines the audio view of the scene. A new audio view is rendered whenever the position or orientation of the listener or a sound source changes.

The state of the audio environment must be maintained along with the graphical environment. When objects move, both the graphical and audio rendering engines need to be notified. Likewise, when the view of the scene changes, the camera and listener positions must be updated.

A sound source, like a graphical object, can be placed and oriented in 3D space. Additional properties describing the directionality and extent of the sound source also affect the final rendering effect; these properties are defined by the sound model.

SoundModel

The model describes a sound in terms of two elliptical regions. The sound source location is a focus for both the inner and the outer ellipse (Figure 1). Within the inner ellipse, the sound maintains a constant, maximum intensity; this supports a controllable volume in the immediate proximity of the sound source. The sound has no directionality in this region: it seems to come from all around the listener. Between the two ellipses, the sound intensity decreases with distance and seems to originate at the sound location. The outer ellipse represents the limit of audibility of the sound. (Note that this description represents the model in two dimensions; however, audio is rendered using the 3D model obtained from rotating the 2D description about its directional axis.)

Figure 1. Sound Emitter Model

This sound model accurately represents a wide range of audio sources. Sounds are frequently described as "ambient," "directional," or "point" sounds. Each of these types can be described in terms of the elliptical model used by Intel Spatial Audio for Java.

Directional Sounds

According to this model, the directionality of a sound is determined by the shape of the two ellipses that describe it. A strongly directional sound will be very elliptical, with the sound audible much further in front of the sound location than behind it. A less directional sound will be described by ellipses which, in three dimensions, are almost spheres.

This model has the advantage of combining simplicity and realism. A point in space describes a sound's location, and a three-dimensional vector describes its "front" or "forward" direction. Four distance values describe its extent and degree of directionality: the minimum front range and minimum back range determine the inner ellipse, and the maximum front range and maximum back range determine the outer ellipse.

Because the sound is described by two continuous curves, it behaves naturally when approached from any angle. This is a significant advantage over other models such as the Cone Model, shown in Figure 2. Here a listener approaching the sound source from its front will hear a gradual increase in volume, but a listener approaching it from behind will be shocked by the sudden appearance of the sound at full intensity. The addition of a spherical ambient region around the directional sound source alleviates this problem (Figure 3), but leads to complicated and non-intuitive behavior in the area where the ambient and directional regions intersect.

Figure 2. The Cone Model	Figure 3. The Combination Model

Point Sounds

A point sound is a directionless sound which emanates equally in all directions; the orientation of the sound source does not affect the way it sounds. The elliptical model defines a point sound when the minimum front range equals the minimum back range and the maximum front range equals the maximum back range. In effect, the sound is described by a location in space and two spheres, as shown below in Figure 4. Within the minimum range, the sound is of constant intensity and ambient; between the minimum range and the maximum range, the intensity drops to zero and the sound seems to originate at the sound location.

Figure 4. A Point Sound

Ambient Sounds

An ambient sound comes from all around the listener and maintains a constant intensity. Because a sound is effectively ambient within the inner ellipse, an entirely ambient sound is one in which the inner ellipse encompasses the entire range of the sound. In this case, the minimum front range equals the maximum front range and the minimum back range equals the maximum back range (Figure 5). A sound in which these values are nearly equal would be "mostly ambient," with a large ambient region and a small region in which the sound intensity drops rapidly.

Figure 5. An Ambient Sound.

Simple Usage Model

Creating an Environment and a Listener

The SpatialAudio class provides a static method, createEnvironment(), which creates and initializes a new spatialAudio Environment. This method can throw two Exceptions: If the package is not properly installed on the client machine, a spatialAudio.InitializationException is thrown; if the method fails for any other reason, a spatialAudio.SpatialAudioException is thrown.

Set general and default characteristics of the environment using the Environment interface setEnvironment() method. This method takes an instance of the EnvironmentModel class as its parameter. The EnvironmentModel class allows you to specify the orientation of the coordinate system you are using, the speed of sound in the environment, and the default quality of audio spatialization. The speed of sound is used to automatically calculate Doppler effects. To disable Doppler effects, set the speed of sound to zero (this is the default).

Once you have successfully created an Environment, you can use it to create a Listener and one or more Sound Sources.

After creating an environment, you should create a Listener. The Intel Spatial Audio for Java package supports exactly one Listener per environment. A DirectListener sends the audio output directly to the audio device. A StreamingListener receives buffers of data which have been processed by the package. This data can then be stored or further manipulated by the application.

public void init() {

. . .
. . .

/*
* Create the environment, then set the environment model.
* The EnvironmentModel used here shows the package default options.
*/

try {
     Environment env = SpatialAudio.createEnvironment();
     env.setEnvironment(new EnvironmentModel(RenderOptions.USE_LEFT_HAND,
                                             0.0f,
                                             RenderOptions.LOW_SPATIALIZE));    
     } catch (InitializationException e) {
     /*  The Intel Spatial Audio for Java package is not properly installed. */
     /*  Handle the exception */
     } catch (SpatialAudioException e) {
     /*  Failed to create the Environment */
     /*  Handle the exception */
}

/*
* Next we create the listener.
* If we do not specify a preferred format, it will be PCM_22K_16BIT_STEREO.
*/

try {
     DirectListener listener = env.createDirectListener(WaveFormat.PCM_22K_8BIT_STEREO);
     } catch (SpatialAudioException e ) {
     /*  Failed to create the Listener. */
     /*  Handle the exception */

}

. . .
. . .

} // init()

Managing the Listener

When you create a DirectListener, it automatically makes contact with the audio device. You should normally call the DirectListener.connect() and DirectListener.disconnect() methods in an applet's start() and stop() methods, respectively. This ensures that the Listener is connected to the audio device while the applet is running, but that it frees the device for use by other applets and applications when it is not. Disconnecting the Listener from the audio device also stops the play of any SoundSources.

You can control the position and orientation of the listener using the setPosition() and setOrientation() methods. Use the corresponding "get" methods to access the current position and orientation of the Listener.

The listener in an audio environment is analogous to the camera in a graphical environment. If you are using both audio and graphics, you should generally update the state of the listener whenever the position or orientation of the camera changes. The example below demonstrates this with calls to to an imaginary graphics library.

public void start() {

. . .
. . .

/*
// Set the position and orientation of the listener
*/
theListener.setPosition(0.0f, 0.0f, 0.0f);
GraphicsLibrarySetCameraPosition(0.0f, 0.0f, 0.0f);

Vector3D forward = new Vector3D(0.0f, 0.0f, 1.0f);
Vector3D up = new Vector3D(0.0f, 1.0f, 0.0f);
theListener.setOrientation(forward, up);
GraphicsLibrarySetCameraOrientation(forward, up);

. . .
. . .

} // start()

Adding and Controlling Sound

Use the Environment.createSoundSource() method to create a new sound source. When creating a CachedSoundSource, you must specify the source of the data for the sound source. This can be an InputStream, an URL, or a String representing an URL.

The SoundSource.setModel() method allows you to specify the intensity and audio range of the sound source. Control the position and orientation of the sound source by calling setPosition() and setOrientation(). Use the corresponding "get" methods to retrieve the current state of the sound source.

Sound sources can be created in an applet's init() method or any other time after the environment has been created.

/*
* Create a cached sound source
*/

try {
     CachedSoundSource mySound = env.createSoundSource(getDocumentBase() + "mySound.wav");
     } catch (EException e) { 
     System.out.println("Failed to create the sound source");
     /* handle the exception */
}

/*
// Set the model, position, and orientation of the sound source
*/
SoundSourceModel model = new SoundSourceModel(5.0f, 3.0f, 20.0f, 12.0f, 0.8f);
mySound.setModel(model);

mySound.setPosition(0.0f, 0.0f, 0.0f);
mySound.setOrientation(0.0f, 0.0f, 1.0f);

Use the CachedSoundSource method setPlayState() to start, pause, resume, or stop playback of the sound. To get information on the play state of the sound source, use CachedSoundSource.getPlayState() or SoundSource.getMediaState().

/*
* Start the sound looping over its entire length, then stop it
*/

mySound.setPlayState(MediaState.AUDIO_PLAY);

...
...

mySound.setPlayState(MediaState.AUDIO_STOP);

Cleaning Up

Because of the way the audio device is allocated, it is very important that you handle your clean up carefully. This includes calling the DirectListener.connect() and disconnect() methods when the applet becomes active or inactive. In addition, you may announce that you have finished using a SoundSource, Listener, or Environment object, and request that any resources being used by it be permanently freed, by calling release(). Note that the resources are not guaranteed to be released. However, an application must not attempt to use an object once it has been released, and any attempts to do so will fail.

Use of the release() methods is not required, but is provided for cases where the programmer prefers a greater level of control. One common application is in the creation of multiple Listeners. Only one Listener is supported per Environment. However, once the existing Listener has been released, another one can be created.

The typical use audio objects in an applet is shown in the pseudo code below.

import java.applet.*;
import spatialAudio.*;

public class pseudoApplet extends java.applet.Applet {

Environment env;
Listener listener;
CachedSoundSource sounds[];

public void init() {

     /* create the environment */
     /* create the listener */
     /* create the sound source */
}

public void start() {

    /* connect the listener */
    /* set the position and orientation of the listener */
    /* start playing the sounds */
}

public void stop() {

     /* disconnect the listener */
}

public void destroy() {

/* release the sound sources */
/* release the listener */
/* release the environment */

Streaming

Intel Spatial Audio for Java provides streaming support for both Listeners and SoundSources.

The StreamingListener

The StreamingListener supports all the standard functionality described in the Listener interface, and three additional methods. When you create a StreamingListener, you may specify an audio format; this is the format of the audio data that the Listener receives. The default is PCM_22K_16BIT_STEREO. You may also request a buffer size. Use the method StreamingListener.getBufferSize() to get the size, in bytes, of the buffers that will be provided to the listener by the audio rendering engine. Call StreamingListener.requestBuffer() to request the synchronous generation of a buffer of audio data. The application is responsible for timing the requests during a live connection.

/*
* Create a StreamingListener and request a buffer of data
*/

...
...

try {
     StreamingListener myListener = env.createStreamingListener();
     } catch (SpatialAudioException e) {
     System.out.println("Failed  to create the streaming listener");
     /* handle the exception */
    }

byte[] myBuffer = new byte[myListener.getBufferSize()];

while (1) {
     myListener.requestBuffer(myBuffer);
     /* do something with the data */
     }

The StreamingSoundSource

The StreamingSoundSource supports all the standard functionality described in the SoundSource interface, and two additional methods. When you create a StreamingSoundSource, you must specify the format of the audio data that the SoundSource will receive.

The application can use the flush() method to request the immediate return of all buffers in the streaming sound source's playback queue. Use the submitBuffer() method to provide data to the audio renderer. Buffers are submitted through an instance of the BufferHolder class. The application is responsible for timing itself. However, it may specify in the BufferHolder an event which will be signaled when the audio renderer is finished with the buffer and it is available for reuse.

Typical use of a streaming sound source in an applet might follow the model shown in the example below.

public void init() {

...
...

/*
* Create the StreamingSoundSource.  This one will provide data of the format PCM_22K_16BIT_MONO.
*/

try {
     mySound = env.createStreamingSoundSource(WaveFormat.PCM_22K_16BIT_MONO);
     } catch (SpatialAudioException e) {
     /* Failed to create the streaming sound source */
     /* handle the exception */
}

/*
* Create the buffers 
*/

BufferHolder buffer1 = new BufferHolder(new byte[size], (Object)"event1", null, this);
BufferHolder buffer2 = new BufferHolder(new byte[size], (Object)"event2", null, this);
BufferHolder buffer3 = new BufferHolder(new byte[size], (Object)"event3", null, this);

...
...

}

public void start() {

...
...

/* 
* Fill the buffers up with initial data and start feeding them to the sound source
*/

mySound.flush();

/* fill the buffers with data */

mySound.submitBuffer(buffer1);
mySound.submitBuffer(buffer2);
mySound.submitBuffer(buffer3);

...
...

}

public boolean handleEvent(Event evt) {

/*
* Keep feeding buffers back in as soon as they're fee
*/

switch(evt.id) {

case SoundSource.BUFFER_EVENT:

if (evt.arg == "event1") {
     /* fill the buffer with new data */
     mySound.submitBuffer(buffer1);
}

if (evt.arg == "event2") {
     /* fill the buffer with new data */
     mySound.submitBuffer(buffer2);
}

if (evt.arg == "event3") {
     /* fill the buffer with new data */
     mySound.submitBuffer(buffer3);
}

}

...
...

}

Advanced Options

This section describes some of the advanced capabilities provided by Intel Spatial Audio for Java.

Reverberation

Reverberation is a special effect that adds spaciousness to an environment based on the effective delay time for reflected sounds: large open areas have longer delay times than small, enclosed areas. Reverberation is applied to the entire audio environment. You can change the reverberation at any time by calling setReverb() and specifying either the a ReverbModel, specifying the decay time and intensity of the reverberation, or an integer representing one of the preset values. The getReverb() method returns the current reverberation settings.

/*
// Set the reverberation model.  These settings will activate reverb
// with moderate decay time and intensity.
*/

env.setReverb(ReverbModel.CHAMBER_REVERB);

Doppler

The Doppler effect is calculated automatically when rendering a listener. The effect is determined by the relative velocity between the listener and a given sound source. To disable Doppler in the entire environment, set the speed of sound to zero. You can also disable the Doppler effect for a specific sound source by setting the NO_DOPPLER flag when you create it.

Processor Limiting

In order to control the processor load of rendering audio, you may control the processor budget at both the Environment level and at the level of individual sound sources. The Environment sets the default budget for all sound sources in the environment; individual sound sources may override this default to request greater or lesser quality localization. Three budget levels are available to each sound source: MIN_BUDGET, LOW_BUDGET (the default), and MODERATE_BUDGET.

You can specify the default rendering level for sound sources in the EnvironmentModel when you call setEnvironment(). If you do not specify a rendering level, the default rendering level will be LOW_SPATIALIZE. To override the default environment setting for specific sound sources, call the setCpuBudget() method.

You can place additional restrictions on the the rendering of specific sound sources when you create them. The NO_SPATIALIZE and NO_ATTENUATE flags specify that these reduced rendering modes should be applied to the sound source. (NO_ATTENUATE is only valid if NO_SPATIALIZE is also specified.) If NO_SPATIALIZE is set, only distance attenuation will be provided. If NO_ATTENUATE is also set, then only mixing is provided. The NO_DOPPLER and NO_REVERB flags indicate that no Doppler effect or reverberation should be applied when the sound source is rendered.

/*
// Create a cached sound source to which no
// spatialization, Doppler effect, or reverberation
// is applied.  Distance attenutation remains active
// by default
*/

CachedSoundSource mySound = env.createCachedSoundSource(RenderOptions.NO_SPATIALIZE +
                                                           RenderOptions.NO_DOPPLER +
                                                           RenderOptions.NO_REVERB,
                                                           "http://www.mySite.com/mySound.wav");                            

Pitch Control

You can alter the pitch of a sound source using the setPitch() method. The default pitch factor, 1.0, causes no change in pitch. Supported pitch adjustment ranges from 0.25 to 4.0.

/*
// This will dramatically raise the pitch of the sound source.
*/
mySound.setPitch(3.0f);

Synchronizing SoundSources

The control of several cached sound sources can be synchronized in a synchronization group. The synchronization group of a sound source is determined by the groupId parameter in the CachedSoundSourceDesc when the sound source is created. A value of zero indicates that the sound source does not belong to a synchronization group, and any non-zero value can be used to represent a synchronization group. When you call setPlayState() on any member of a synchronization group, the control is applied to all members of the group.

The following code demonstrates the synchronization of three cached sound sources using the group ID 5.

/*
// Create three sound sources, specifying the same synchronization group ID.
*/
CachedSoundSource myFirstSound = env.createSoundSource(0, "http://www.mySite.com/myFirstFile.wav", 5, null, null, null);
CachedSoundSource myFirstSound = env.createSoundSource(0, "http://www.mySite.com/mySecondFile.wav", 5, null, null, null);
CachedSoundSource myFirstSound = env.createSoundSource(0, "http://www.mySite.com/myThirdFile.wav", 5, null, null, null);

/*
// Start playing all three sound sources with continuous looping.
*/
myFirstSound.setPlayState(MediaState.AUDIO_PLAY);

Half-Duplex Support

When you create a listener, the audio driver it uses may become unavailable to other clients because most audio drivers do not support full-duplex and/or shared operation. A common way to work around this limitation is to switch between input and output mode for the driver as needed. The connect / disconnect mechanism supports this switching model.

After creating your Listener, you must call connect() to connect it to the audio output device. Anytime you wish to make the audio output device available to other applications, you must call disconnect(). To resume audio output, call connect() again. In an applet, this normally means that you should call connect() in your start() method and disconnect() in your stop() method.

Support for DirectSound*

This implementation provides support for Microsoft DirectSound* II. This not only provides better performance, but also allows sharing of the audio device within an application. DirectSound requires a window handle from the application. Intel Spatial Audio for Java will extract a native window handle from any AWT Component. To enable DirectSound on machines that support it, pass a Component as the info parameter when creating a DirectListener. Normally, this can be the applet itself ("this").

Support for Multiple Environments

The package supports the use of multiple Environments by an applet or application. Because of the way the audio device is allocated on Windows* operating systems, however, some constraints must be observed. On machines that support DirectSound, multiple DirectListeners from different Environments may share the audio device as long as they originate from the same process. (Support for DirectSound is discussed above.) If DirectSound is not supported or active, only one Listener may have access to the audio device at a time. This means that listeners in different environments must take turns accessing the audio device. Use connect() and disconnect() to ensure that only one DirectListener is trying to use the audio device. (NOTE: When a DirectListener is created, it automatically seeks contact with the audio device. If the audio device is already in use, the creation will fail. Therefore, you must disconnect any existing listeners before creating a new one.)

This page was last updated on Feb 11th, 1997.

Legal Stuff