Learning Beyond Sight
Making scientific visualization more accessible to people with vision impairments.
An accessible, embodied, audio-haptic VR-infographic about the effects of the climate change on the water cycle. Master thesis of Helena Klein (Knowledge Visualization).
Sound Design
(Binaural 3D audio using Steam Audio, use headphones)
Each storage of water (pool) has its own sound. When the player approaches one with. the controller, it becomes louder and can be felt through the vibrations. On the side, you can hear a collage of all the pools, that are in this experience.
Insights into the project (german)
Things about Learning Beyond Sight
I Doesnât Exist is a visual text-adventure inspired game from LUAL Games.
I worked on the implementation and large parts of the Sound Design and Co-Music-Composition. The games sound creation came in two waves. The first was the prototype and Bachelor Project from Lu and Al, the other the final version after the Kickstarter.
Spoilers ahead - beware!
Iâd recommend playing it first :)
Sound Design - Hearty
The first act of the game offered a rich world to fill with life through sound. The safe sound has received the most attention, as it is heard prominently at start. First iterations were either too clichée (heart thump) or abstract (rythmic metal screetches). So I recorded my heart with an ultrasonic doppler device, layered with metal and stone grains, which produced an organic sound.
Other recordings from machines, sad bucket noises or glass shards provided raw material to work on. Soundly was also a great resource for getting raw material audio files to work with.
At the end, there were around 80 sound events, not counting variations.
Another big part was the keyboard sound, which came late into the game's development. By keeping track of how many keys are pressed in Unity, I could implement a separate sound for pressing and releasing that added to the realism and feeling of typing on a completely different-sounding keyboard.
â
This also added a subtle additional means of storytelling.
For instance, at the avatar's complete burnout, the keyboard sounds are more and more distorted. After the maze, they are randomly placed around the player.
Programming - Bitcrush
//assuming some buffers float* input and float* output
//of size n
float factor = powf(2, bitdepth - 1) - 1;
while(n--)
{
*output++ = roundf(*input++ * factor) / factor;
}The idea of using bit crushing for an 8-bit-looking game was appealing. Nevertheless, the sound design had to have an organic feel, which led to bit crushing and sample dividing of the organic material to set accents or contrasts.
â
The first experiments with rendered samples turned out to be very limiting, so programming a Fmod plugin had a big creative impact and provided a lot of freedom and possibilities. Bitcrushing means to quantise the resolution of every sample recorded at each timestep to match a certain bitdepth.
This effect can be achieved by muliplying the input with the maximum amount that is representable by a certain bitdepth, rounding that result and then dividing that again with the maximum amount.
â
The factor is 2 to the power of the bitdepth - 1. The first - 1 is because the sign bit. The other - 1 is for accounting zero.
//assuming some buffers float* input and float* output
//of size n
float storage;
for(size_t i = 0;i < n;i++)
{
if (!(i % divide))
{
*output++ = storage = *input++;
}
else
{
*output++ = storage;
input++;
}
}Dividing the samplerate means to reduce how many samples per second are played back. This can be simulated by prolonging values. At example 2 this is done by either storing or writing the samples to the outbuffer. The easiest way I found, was to use the modulo operation. Every sample that is a multiple of the division variable is stored and output, every other will output the stored sample.
â
A division by two would therefore store every other sample and write the same value into the next.
Music - The Chorale
In the final version, most of the music was composed by Paul Taro Schmidt. Before we worked on the game, he composed the main theme and countless sketches exploring different moods.
One was the chorale, that takes a big part in the game.
â
I took the stems and built them into the game as it felt natural.
It starts when taking the clock's hand but is only heard when arriving back at the safe. The synths are seemingly coming from inside the save. Upon opening it with the clock hand, it gets more present, after taking the key the music shifts from 3d to 2d to the player and the voices start singing from the direction of the door.
When approaching the door, the synths get progressively louder and richer, when trying to open the door, the voices start distorting (more on the bit crushing later) and the pitch of the piece drastically drops.
â
In the green world, the soundtrack is now the choral pitched down without any vocals.
After the minigame intermezzo, some additional synth layers are added to the chorale and at the escalation the voices come back in sweeping with an LFO.
When taking control over the avatar the distorted voice follows the avatar and the synths transfer to an array of 5 emitters around the camera that fly behind over the maze.
The player can walk in the direction of the door. Like an inverse magnet, it becomes progressively harder to get forward while the voices get even more distorted.
When turning the head, pitch and distortion change, and when fully facing away from the door, the voices sound pleasant. Walking to the maze lightens up the music even more.
When the maze has been entered the voices now unheard switch to the door after the maze, where they can be heard when close enough.
â
All the decisions in the maze affect the pitch of the chorale and the additional layer of maze-beeps.
When entering the door after the maze the voices fly to the dome which the player will get to shortly afterwards.
When approaching the dome, the synth part of the chorale fades back to a 2d sound. Words and music are now reflecting the decisions in the maze.
Pressing the button stops the chorale.
It will come back in the gameâs epilogue. This event has spanned a big part of the game and provided a means to connect several scenes by morphing the room and spectra of one piece.
â
One event contained all the sound files and a spatialiser. 6 other events contained nothing but a transceiver to receive audio from the first event. 5 synth with a binaural spatialiser and 1 for the voice with both a fmod and binaural spatialiser.One event contained all the sound files and a spatialiser. 6 other events contained nothing but a tranceiver to receive audio from the first event. 5 synth with a binaural spatialiser and 1 for the voice with both a fmod and binaural spatialiser.
