My clocks is wrong

PH was staying with his daughter when he remarked she should, ‘Get a decent telly — one where the sound and programme are synchronised.’ He wandered through into the kitchen to make a cup of tea, and looking up at the tv on the wall there, noticed that that one was out of sync too. ‘Hey you’ve got two tvs that need sorting!’ he called to her. She came through.

‘There’s nothing wrong with the tv, Dad,’ she said.

Watching her speak, PH realised her lips were out of sync with her voice. He started to answer, but he was out of sync too. He could hear his words coming out before he’d started to move his mouth …

Quite what is going on with PH isn’t clear. The stay with his daughter was following heart surgery, and a spell in intensive care, and the associated trauma may have upset his neural functioning. Subsequent brain scans have indeed revealed lesions in areas thought to play a role in hearing, timing and movement; though it’s possible these were there before. To confirm the desyncing effect, PH was asked in an experimental setting to watch video clips as the experimenter slid the sound back and forth, and sure enough he identified the clips as being in sync when the sound was delayed by 200 milliseconds. Confoundingly, when presented with the McGurk illusion test, where the subject is played the video for one syllable alongside the audio for another, producing the illusion of a third syllable, PH experienced the illusion when the sound was 200 milliseconds early. Why is his brain late in one experiment, and early in another?

A complete theory of the brain’s clock is a long way off, but the case of PH is richly suggestive. The most immediate idea to draw out is that of multiple neural clocks. For a desyncing to occur in PH’s brain, there must be at least two clocks to be out of time with each other. In a piece of AV media, there are precisely two clocks: the audio and the video. But for a human, in the sense realm alone, there are potentially many.

Imagine you drop a mango on your foot. The impact will be signified by: the sight of it hitting, the noise of the splat, the sensation against the foot (and a personal internal awareness of where the foot is — the proprioceptive aspect), and the smell of ripe mango flesh rising from the split skin. An awareness of the smell may come a little later, but the other packets of sense information all need to land together. And this clearly won’t happen by itself. The light will reach your eye in around 6 nanoseconds (6 millionths of a millisecond), the sound will hit your ear in 5 milliseconds, and the sense signal trawling up the nerve from your foot will take some 20 milliseconds to arrive. Then, once they’ve all made it to the brain, there is nothing to say the modular processing that happens next in separate sense areas will take the same amount of time. Quite the opposite — it seems highly likely that some senses will be processed faster than others, meaning that modally different pieces of information regarding the same incident will be available for sharing with other parts of the brain at different moments — i.e. out of sync with each other.

The reintegration of these sensory inputs — arriving at different times, and coming through modular processing operations at different rates — into a synchronised awareness of the real world, must be a challenge. It is conceivable that there is some form of resyncing algorithm, but this would only work if the delays in the inputs, and the differential processing rates, are always consistent. Yet we have a good sense of syncing even if the mango hits us in the face (when there is a much shorter nerve distance to travel), or if the things to be resynced are very complex (involving lots of processing) or very simple (involving little).[1] But rather than performing complicated resyncing operations, there’s a simpler way the brain could do things: it could deploy a concept of temporal tolerance.

The idea for temporal tolerance goes like this: if events from two different senses arrive into awareness (or into a “reintegration operation”) within the stipulated tolerance time, and look as if they go together — e.g. the sight of a mango hitting your foot, and the sound “splat” — then sync them. In effect, the tolerance time gives the reintegration operation a degree of temporal discretion as to what should sync with what. We know the operation of putting things together to build our overall perception of the world likes to seek out patterns and find sense-making connections,[2] and so in processing terms, it is no more expensive to perform this operation of “hunting out what looks like it might go with what”. But by introducing temporal tolerance, the problem of desynced inputs immediately resolves. Things that need to be in sync — e.g. the sight and sound of an impact, or mouths speaking and the stream of language heard — magnetically find each other; and things that don’t need to be in sync — e.g. the noise of the blender switching on and the aeroplane going by out the window — can float around within the confines of the tolerance time fairly harmlessly.

According to this, we would notice a desyncing only if the two items that the sense-making operation wants to sync — e.g. the mouth and the language — are out by more than the tolerance time. Or another way of putting this would be to say that the clocks of the various modular processing areas of the brain disagree by more than the limits of temporal tolerance. With poor Mr. PH, it would seem something screwey is going on with his clocks, or his tolerance time, or both.

But for those of us living in an in-sync world, there is still an interesting implication. If there is this tolerance time for syncing across the senses, it potentially allows things that aren’t perfectly in sync to be perceived as synced if there is a strong sense-making reason to draw them together. I.e. our sensory mechanisms have the freedom to perform acts of magical syncing if so moved.

Most of the time, most of us aren’t moved to sync the noise of the blender with the sight of an aeroplane going by out the window. Though if we are feeling tingly, we may be. And we may be if we are in the presence of a strong syncing impulse, as when feeling a rhythm, as when listening to music.

Given the freedom afforded by a window of temporal tolerance, the influence of music can be to create a series of “magnetic nows” — points in time that have a strong syncing attraction. The sense-making potential embedded within the rhythm or musical shifts endows these nows with the power to pull toward them nearby occurences from other senses. For example, imagine a downbeat in the music coming at around the time the aeroplane crosses a contrail from a previous aeroplane. If sensory information regarding the two becomes available within the neural tolerance time, a sense-making impulse has the opportunity to unite them, and without violating any rules of perception, make the world a little bit more musical. And the same may happen with another point in the music coming at around the time the cat is jumping off the sideboard; at another moment when the leaves are rippling in the breeze; and then the smell of oranges comes up into your nose just as a rubato slows the phrase, as though the cello were breathing in …. The extraordinary power of music is to subsume the world into its sense of time. Or to take as much of the world as can fit within the tolerance time of any of the multitude of moments it creates, and bend the world’s time to them. Which is enough to be severly beautiful.

I wrote on this blog previously about the “Christmas tree light effect”[3] — the term given by neuroscientists to the observation that subjects exposed to randomly blinking Christmas tree lights while music is playing will naturally find synchronicities. I.e. a cognition illusion takes place by which the lights and music match up. The idea of temporal tolerance in truth doesn’t add all that much — it’s really just a toy model for how such an effect may be operating on the perceptual level.

One thing I do have to add this time however is that in situations where you want your timing to be particularly precise, you may wish to avoid the potentially timebending effects of music, and the magnetic nows it creates. Put more concretely, if you’re playing ping-pong and it’s match point, I’d say switch off the Brahms.

Information in this post about PH comes from an article by Helen Thomson published in the New Scientist. Elliot Freeman at City University London suggests the idea of a brain with many clocks. The idea of temporal tolerance and music performing timebends are mine.

 [1] Clearly you could recalibrate nerves from all over the body according to distance from the brain, and it’s possible that the processing time for e.g. the sight of lots of complicated objects of different sizes moving at different depths, and the sight of a black dot on a white screen, is the same (with the former, the visual processing module may just be more radical in throwing excess information away). However the processing times may not be the same, in which case the problem remains difficult: how does the reintegration algorithm know how long any signal has spent travelling or in processing? Further algorithms could be built in to deal with this, but I think the idea of temporal tolerance is easier (especially if you are using the sense-making equipment anyway), and the brain tends to prefer easy.

[2] See Kolers’ color phi phenomenon for a classic example.

[3] See How Cognitive Illusions Can Make You A Better Dancer