9.35 Illusion Laboratory
Spring 2023 Auditory Lab
Our illusion addresses whether visual stimuli can extend the duration of the continuity effect for sound textures
In the McGurk effect, different mouth movements change the viewer’s perception of the sound being produced, despite the audio staying the same. Notably, the McGurk effect focuses on the domain of speech perception, a task which generally relies on both visual and auditory input. Our illusion extends this idea of multimodal perception to sound textures, which are sounds composed by many sound sources such that the overall sound is both random and somewhat uniform.
Our illusion investigates whether paired visual stimuli can extend sound texture illusions. In the paper Illusory sound texture reveals multi-second statistical completion in auditory scene analysis, the authors found that certain sound textures were perceived as continuing across a masking noise. Based on the proportion of participants who experienced this continuity effect, we selected three sound textures:
- High Continuity: Applause
- Medium Continuity: Drumroll
- Low Continuity: Manual Typewriter
In the original experimental design, participants listened to a 2s clip of a sound immediately followed by 2s of noise and finally, 1s of the original sound. In our version, participants hear a 3s sound texture followed by 10s of white noise with three different video conditions:
- No video during the sound texture or the white noise mask
- Video only during the white noise mask
- Video during both the sound texture and the white noise mask
For each condition, we are measuring the length of the continuity illusion on a scale of 0 (no continuity) to 10s. Participants start a timer when they begin playing the sound/video and press stop when they no longer hear the sound texture, subtracting 3s for the original sound texture. We also included a fourth condition where no sound texture is played but the associated video is played over white noise to test if the video alone is enough to induce the continuity illusion. In this case, we will only record a yes/no value of whether participants heard an illusion.
We expect to observe the longest continuity effect when the video is played during both the sound texture and the white noise mask (condition 3) and the second longest continuity effect when the video is only played during the white noise mask (condition 2). We expect the audio-only condition (condition 1) to have the shortest continuity effect, and we do not expect listeners to hear any textural illusion when only the white noise and video are played without the preceding sound texture (condition 4).
First Three Conditions
- Find a timer that can measure in seconds
- After picking an illusion, start the time at the same time as the video. A sound texture (e.g. rain falling, rattlesnake, etc.) will play, followed by white noise.
- Stop the timer as soon as you no longer hear the original texture. Subtract 3 from your time and record it as the duration of the continuity effect.
Fourth Condition
- Play the video.
- Record whether you heard a sound texture (as opposed to white noise only).
Texture #1: Clapping
Texture -> Noise
Texture -> Video + Noise
Video + Texture -> Video + Noise
Video -> Video + Noise
Texture #2: Drumroll
Texture -> Noise
Texture -> Video + Noise
Video + Texture -> Video + Noise
Video -> Video + Noise
Texture #3: Typewriter
Texture -> Noise
Texture -> Video + Noise
Video + Texture -> Video + Noise
Video -> Video + Noise
McGurk, Harry, and John MacDonald. “Hearing Lips and Seeing Voices.” Nature, vol. 264, no. 5588, Dec. 1976, pp. 746–48, doi:https://doi.org/10.1038/264746a0 AND McWalter, Richard, and Josh H. McDermott. “Illusory Sound Texture Reveals Multi-Second Statistical Completion in Auditory Scene Analysis.” Nature Communications, vol. 10, no. 1, Nov. 2019, doi:https://doi.org/10.1038/s41467-019-12893-0.