It’s happened to all of you before, I’m sure. You’re streaming a video, only to find that, at some point, the character’s mouth movements are no longer lining up with the audio you expect to hear. That is, the voice appears to be coming on after the video, or before it. It’s called lag, and it drives us absolutely crazy. Amazingly, we can perceive lag even when the visual stimulus (mouth movement) and the audio stimulus (the voice) are only a few milliseconds apart.
Now, professional musicians have much more experience than us normal people do with trying to synchronize different kinds of stimuli. Does this mean that they are better at perceiving lag when video and audio are asynchronous?
Hweeling Lee and Uta Noppeney from the Max Planck Institute in Germany sought to answer this question, and published their findings in PNAS last year. They wanted to know 1) if musicians were better at perceiving lag in general, or 2) if they were only better at perceiving lag in particular situations that were related to their expertise. In this case, they studied professional pianists, who would presumably be very good at temporally connecting the pressing of a piano key and the sound that emerges from that movement.
In their study, they had two groups (one normal, non-musician group and a group of pianists) watch two sets of videos. In the first set of videos, the subjects watched actors saying words, and they had to respond with whether the audio was off or not (the lag was varied from 360 ms before the video started to 360 ms after the video started). The second set of videos showed hands playing a keyboard, and the task was the same – is the audio off or not when the person presses a key?
It turns out that musicians and non-musicians are equally good at telling when speech is temporally off from face movement (this is the phenomenon that we all experience when streaming videos). But the pianists were much better at knowing when the sound that was played on the keyboard was off from the finger movement. In an MRI scan, the researchers found that areas associated with movement were activated when pianists watched these scenes. This means that – probably – the pianists were playing the notes themselves in their minds, and this made them better at knowing when the sound was supposed to appear.
Musicians probably have stronger connections between motor areas and sensory areas in the brain, which allow them to perfect their timing when playing music. But it’s not true that they’re better at perceiving lag in general. All of these brain and behavioral changes are based on specific experiences. Musicians and non-musicians alike have plenty of experience watching people speak, so we all know how to match sound to the scene of someone else moving his jaw. But only well-trained pianists have the extensive experience of hearing a sound after pressing a piano key, so their senses are much sharper in that particular domain.