︎︎︎ indie game
︎︎︎ creative technology
︎︎︎ design for social good
︎︎︎ case study

An immersive gaming experience utilizing audio clues to simulate the underwater environment with noise pollution that marine mammals are facing today

Role
︎︎︎ game designer
︎︎︎ developer
︎︎︎ visual designer
︎︎︎ animator

Time
︎︎︎ April 2018

Tool
︎︎︎ Unity
︎︎︎ Illustrator, Photoshop
︎︎︎ After Effects

Discipline
︎︎︎ game design

Demos
︎︎︎ NYC Media Lab ’18 Demo Expo ︎
︎︎︎ Games for Change Festival '18  ︎ ︎︎︎ Dimensions - MFADT 2018 Major Major Show

︎
The sea is being industrialized...
― Michael Jasny, director of the Marine Mammal Protection Project at the Natural Resources Defense Council

︎
We’ve altered underwater acoustic soundscapes.
― Jason Gedamke who runs NOAA’s ocean acoustics program

︎
A deaf whale is a dead whale.
― Tim Donagh, a senior research specialist with Greenpeace

Whale Heard is an immersive gaming experience using audio clues to raise people's awareness of how underwater noise caused by human activities has been transforming the world that marine life lives.

A player will play as a beaked whale in the game to find its mother with the only provided clue, sounds. While a player can hear the noise from a submarine sonar or a sailing cargo ship, the health bar percentage will decrease. It indicates that artificial noise interrupts whales' communication between each other, which depends mostly on sounds. If a player uses sound clues to find food, including fishes, squids, and shrimps in the game, the health bar percentage will increase.

While navigating in the ocean in the game, a player can only see the other objects when being very close to them, which may cause the feeling of being lost. If a player runs out of the health bar percentage, the whale will be dead; a player wins the game by locating its mother before the health bar percentage gets to zero.

(copyright: DPIPWE)

There were more and more stories about whales found stranded on the shore worldwide. Researchers realized the problem was extremely loud and continuous sounds from navy sonars, commercial vessels, and oil drilling rigs, etc. which resulted in these suicidal-like behaviors.

Whales and other marine species have evolved to use hearing as their primary sense because other senses have limited effects underwater. Sight is unavailable due to the light scattering of the particulates. Smells also do not work well since molecules diffuse slowly in liquid. However, sound in water is able to travel faster than in air and transfer further than it is on the land.

Whales depend on sounds to communicate, feed, and navigate through the ocean. In the natural sea, it is already full of sounds: waves, marine life, rainfall, and natural seismic activities. Although there is no global map of ocean noise, researchers agree that ship traffic approximately has doubled noise intensity every ten years. A burst from a seismic air gun to evaluate the seafloor for oil and gas can be as loud as a rocket launch; A massive container vessel can produce 190 decibels of sound energy in the ocean, which is louder than the sound next to a speaker in a rock concert.

(copyright: nature)

Increasing anthropogenic noises may damage the fragile underwater acoustic ecosystem. Though light cannot traverse far through the ocean, sound can. Researchers experimented and found that they could not hear the whales sound from the Indian Ocean anymore in the middle of the Atlantic Ocean when noise from oil and gas drilling started happening while they could detect it before. Noise disrupted communication between whales. One of the endangered species, right whales, can no longer hear each other as much as 80% of the time.

Humanity's acoustic footprint can influence whales' emotion-related hormone performance, alter their behaviors, and cause sickness to their bodies. Studies have shown that if the ambient noise is loud, dolphins whistle at a lower frequency with less variation. Some humpback whales even stop singing. Moreover, specimens of stranded and dead beaked whales showed evidence of hemorrhaging around their brain, and blood around their ears due to possible panic dives.

Animation created from data gathered from recorders at the bottom of Boston harbour shows the whistles and clicks of whales (small dots) being drowned out by ships (large dots) sailing in and out. (credit: D. Ponirakis, L. Hatch and C. W. Clark )

For the very first prototype, I wanted to see how noise in an environment would affect communication between individuals. I invited two participants to figure out where they wanted to have lunch together later with and without ambient noise from one of the busiest streets in Manhattan. Although the sound was not loud enough due to technical problems, they still felt bothered during the discussion. This experiment also reminded them of their experience at one open studio on campus, where a lot of people had their meetings and thus disturbed each other with people's talking.



This experiment proved that types and volume of noise influenced the level of interference to the people in the current space. Despite visual clues from mouth shapes and hand gestures, the participants still found difficulties while talking with each other when the noise was on, which emphasized ocean life's struggle thriving in a world with humanity-added soundscape dependent solely on their hearing system.

Inspired by the experiment, I decided to make an immersive gaming experience because it allowed players to be in a whale's shoe, and to understand the issue only by the practice of playing the game. When starting playing the game, a player started experiencing a reproduced situation of what whales were facing.

In the first version of the game prototype, players could hear sound from an object and see visual sound waves at the same time so they would know the relative position of the object. However, this setting could not present the real condition underwater in which marine life determines the relative direction of others, mainly by sound. Thus, instead, I changed the rule to that players could only see an object which was making the sound they heard when they were very close to it.


To make the game juicier, I animated sprites of each element, so they blended in the background of the ocean without looking like some random pictures floating on the screen. Besides, I also used the particle system in Unity for bubbles to make the scene more alive.

In this game, a player will play as a beaked whale undersea and aim to find its mother whale and hunt for food with auditory clues under the interference of anthropogenic noise. This immersive experience of the game reveals and simulates the current condition in the ocean with noise pollution where marine animals are living.

Intro Video
In the intro video, players will hear the ambient sound underwater and sound from a beaked whale. They will also see the goal of the game and the prompt of playing with an Xbox controller.

Play with Auditory Clues
As hearing is whales' primary sense, auditory clues are the main resources that players can rely on to achieve the goal of the game.



Outro Video
The purpose of the outro video is to introduce each sound players may have heard earlier while playing with its corresponding object.

︎ Auditory Clues
Each object in the game has its corresponding audio. During the game, players will hear sound from two different directions. Audio from the left ear is from the top of the screen corresponding to the left side of the whale on the screen, and vise versa. The volume of the sound from an object is according to the distance between it and a player. Players can only see objects when they are close enough to an object while they may have been hearing the sound for a while.



︎ Health Bar
Sound from cargo ships and navy sonars disturbs the communication between whales. Therefore, to display this situation, if players can hear the audio from a vessel or a submarine sonar even without seeing it, the health bar percentage will decrease. They can also use sound clues to find food (fishes, squids, and shrimps) and increase the health bar percentage by eating it. If players run out of their health, the beaked whale will die, and they will lose the game.

︎ Audio

︎︎︎ Ocean Background

︎︎︎ Beaked Whale
︎︎︎ Fish/Squid/Shrimp

︎︎︎ Submarine Sonar

︎︎︎ Vessel

︎ Sprites & Other Visuals

︎ Favicon

From some of the testing sessions and demo expos, a lot of players mentioned that they felt lost or hopeless during the game when they could not find the mother whale with auditory clues. This feedback of experience from players entirely reflected ocean life's struggles with artificial noise nowadays. A lot of people also commented that they felt the sound from a submarine sonar peaceful at first and then realized the massive sound energy came with it did harm marine animals.

For the next steps for Whale Heard, I would like to add more kinds of whales based on more information from scientific studies on the influences on different species. Besides, it could include more levels with more complicated game mechanisms and setting to make the game more challenging and fun. Instead of a PC game, I would also like to make Whale Heard into a VR game to utilize the feature of sounds in a 3D place to create a different immersive experience for players.

1. Adele Peters. “Underwater Noise Is Killing Whales, But We Can Make The Oceans Quieter.” Fast Company, May 16, 2016.
   
2. Brendan Borrell. “Why do whales beach themselves.” Scientific American, June 1, 2009.
   
3. Jason Bittel. “Noise Pollution Forces Whales and Dolphins From Their Homes.” EcoWatch, April 5, 2017.
   
4. Joseph Mayton. “'A deaf whale is a dead whale': US navy sonars could be cause of strandings.” The Guardian, June 14, 2015.
5. Molly Solomon. “A Plan To Reduce Ocean Noise Pollution For Marine Mammals.” Hawaii Public Radio, September 14, 2016.
6. Nicola Jones. “Ocean uproar: saving marine life from a barrage of noise.” Nature, April 10, 2019.
   
7. Smita Singla. “Effects of Noise Pollution from Ships on Marine Life.” Marine Insight, December 9, 2019.