My clocks is wrong

dali_thepersistenceofmemory_325PH was staying with his daughter when he remarked she should, ‘Get a decent telly — one where the sound and programme are synchronised.’ He wandered through into the kitchen to make a cup of tea, and looking up at the tv on the wall there, noticed that that one was out of sync too. ‘Hey you’ve got two tvs that need sorting!’ he called to her. She came through.

‘There’s nothing wrong with the tv, Dad,’ she said.

Watching her speak, PH realised her lips were out of sync with her voice. He started to answer, but he was out of sync too. He could hear his words coming out before he’d started to move his mouth …

Quite what is going on with PH isn’t clear. The stay with his daughter was following heart surgery, and a spell in intensive care, and the associated trauma may have upset his neural functioning. Subsequent brain scans have indeed revealed lesions in areas thought to play a role in hearing, timing and movement; though it’s possible these were there before. To confirm the desyncing effect, PH was asked in an experimental setting to watch video clips as the experimenter slid the sound back and forth, and sure enough he identified the clips as being in sync when the sound was delayed by 200 milliseconds. Confoundingly, when presented with the McGurk illusion test, where the subject is played the video for one syllable alongside the audio for another, producing the illusion of a third syllable, PH experienced the illusion when the sound was 200 milliseconds early. Why is his brain late in one experiment, and early in another?

A complete theory of the brain’s clock is a long way off, but the case of PH is richly suggestive. The most immediate idea to draw out is that of multiple neural clocks. For a desyncing to occur in PH’s brain, there must be at least two clocks to be out of time with each other. In a piece of AV media, there are precisely two clocks: the audio and the video. But for a human, in the sense realm alone, there are potentially many.

Imagine you drop a mango on your foot. The impact will be signified by: the sight of it hitting, the noise of the splat, the sensation against the foot (and a personal internal awareness of where the foot is — the proprioceptive aspect), and the smell of ripe mango flesh rising from the split skin. An awareness of the smell may come a little later, but the other packets of sense information all need to land together. And this clearly won’t happen by itself. The light will reach your eye in around 6 nanoseconds (6 millionths of a millisecond), the sound will hit your ear in 5 milliseconds, and the sense signal trawling up the nerve from your foot will take some 20 milliseconds to arrive. Then, once they’ve all made it to the brain, there is nothing to say the modular processing that happens next in separate sense areas will take the same amount of time. Quite the opposite — it seems highly likely that some senses will be processed faster than others, meaning that modally different pieces of information regarding the same incident will be available for sharing with other parts of the brain at different moments — i.e. out of sync with each other.

The reintegration of these sensory inputs — arriving at different times, and coming through modular processing operations at different rates — into a synchronised awareness of the real world, must be a challenge. It is conceivable that there is some form of resyncing algorithm, but this would only work if the delays in the inputs, and the differential processing rates, are always consistent. Yet we have a good sense of syncing even if the mango hits us in the face (when there is a much shorter nerve distance to travel), or if the things to be resynced are very complex (involving lots of processing) or very simple (involving little).[1] But rather than performing complicated resyncing operations, there’s a simpler way the brain could do things: it could deploy a concept of temporal tolerance.

The idea for temporal tolerance goes like this: if events from two different senses arrive into awareness (or into a “reintegration operation”) within the stipulated tolerance time, and look as if they go together — e.g. the sight of a mango hitting your foot, and the sound “splat” — then sync them. In effect, the tolerance time gives the reintegration operation a degree of temporal discretion as to what should sync with what. We know the operation of putting things together to build our overall perception of the world likes to seek out patterns and find sense-making connections,[2] and so in processing terms, it is no more expensive to perform this operation of “hunting out what looks like it might go with what”. But by introducing temporal tolerance, the problem of desynced inputs immediately resolves. Things that need to be in sync — e.g. the sight and sound of an impact, or mouths speaking and the stream of language heard — magnetically find each other; and things that don’t need to be in sync — e.g. the noise of the blender switching on and the aeroplane going by out the window — can float around within the confines of the tolerance time fairly harmlessly.

According to this, we would notice a desyncing only if the two items that the sense-making operation wants to sync — e.g. the mouth and the language — are out by more than the tolerance time. Or another way of putting this would be to say that the clocks of the various modular processing areas of the brain disagree by more than the limits of temporal tolerance. With poor Mr. PH, it would seem something screwey is going on with his clocks, or his tolerance time, or both.

But for those of us living in an in-sync world, there is still an interesting implication. If there is this tolerance time for syncing across the senses, it potentially allows things that aren’t perfectly in sync to be perceived as synced if there is a strong sense-making reason to draw them together. I.e. our sensory mechanisms have the freedom to perform acts of magical syncing if so moved.

Most of the time, most of us aren’t moved to sync the noise of the blender with the sight of an aeroplane going by out the window. Though if we are feeling tingly, we may be. And we may be if we are in the presence of a strong syncing impulse, as when feeling a rhythm, as when listening to music.

Given the freedom afforded by a window of temporal tolerance, the influence of music can be to create a series of “magnetic nows” — points in time that have a strong syncing attraction. The sense-making potential embedded within the rhythm or musical shifts endows these nows with the power to pull toward them nearby occurences from other senses. For example, imagine a downbeat in the music coming at around the time the aeroplane crosses a contrail from a previous aeroplane. If sensory information regarding the two becomes available within the neural tolerance time, a sense-making impulse has the opportunity to unite them, and without violating any rules of perception, make the world a little bit more musical. And the same may happen with another point in the music coming at around the time the cat is jumping off the sideboard; at another moment when the leaves are rippling in the breeze; and then the smell of oranges comes up into your nose just as a rubato slows the phrase, as though the cello were breathing in …. The extraordinary power of music is to subsume the world into its sense of time. Or to take as much of the world as can fit within the tolerance time of any of the multitude of moments it creates, and bend the world’s time to them. Which is enough to be severly beautiful.

I wrote on this blog previously about the “Christmas tree light effect”[3] — the term given by neuroscientists to the observation that subjects exposed to randomly blinking Christmas tree lights while music is playing will naturally find synchronicities. I.e. a cognition illusion takes place by which the lights and music match up. The idea of temporal tolerance in truth doesn’t add all that much — it’s really just a toy model for how such an effect may be operating on the perceptual level.

One thing I do have to add this time however is that in situations where you want your timing to be particularly precise, you may wish to avoid the potentially timebending effects of music, and the magnetic nows it creates. Put more concretely, if you’re playing ping-pong and it’s match point, I’d say switch off the Brahms.

 

Information in this post about PH comes from an article by Helen Thomson published in the New Scientist. Elliot Freeman at City University London suggests the idea of a brain with many clocks. The idea of temporal tolerance and music performing timebends are mine.
 [1] Clearly you could recalibrate nerves from all over the body according to distance from the brain, and it’s possible that the processing time for e.g. the sight of lots of complicated objects of different sizes moving at different depths, and the sight of a black dot on a white screen, is the same (with the former, the visual processing module may just be more radical in throwing excess information away). However the processing times may not be the same, in which case the problem remains difficult: how does the reintegration algorithm know how long any signal has spent travelling or in processing? Further algorithms could be built in to deal with this, but I think the idea of temporal tolerance is easier (especially if you are using the sense-making equipment anyway), and the brain tends to prefer easy.
[2] See Kolers’ color phi phenomenon for a classic example.
[3] See How Cognitive Illusions Can Make You A Better Dancer
This entry was posted in Knowledge, Thoughts and ideas. Bookmark the permalink.

2 Responses to My clocks is wrong

  1. Leon says:

    Excellent article – thanks Adrian. I enjoyed following this train of thought with you and I am very interested to explore the idea of temporal tolerance further.

    Playing improvised soundtracks to silent films with my band Sink has led us to interesting experiences with the ‘magnetic nows’ you describe. During our preparation for the performance we approached the films with some idea in mind about how we would musically ‘respond’ to specific scenes in each film.

    We spent hours struggling to sync up our playing with specific scenes and moments. This never really worked because without a score we could not be precise enough to accurately match the visual cues. In our frustration we decided to try playing without watching the films, whilst recording our music and, as if by magic, suddenly the music and moving image were synced up quite gracefully – as if we had studied the films carefully for months, planning exact timings and practising thoroughly.

    We came to a similar conclusion – that somehow the viewer makes associations between the sound and the moving image which bring them together in a cohesive, subjective whole.

    We have experimented with Tom Demajo in this area. He set us the task of responding to a piece of video which he made in collaboration with two dancers. Tom tracked the dancers’ limbs and assigned to their movements different coloured moving shapes. We then individually (our band is made up of 3 players – a sax, accordion and violin) improvised a sonic response to the video as we watched it, recorded the results and then layered the tracks on to one another. The result can be viewed here: http://vimeo.com/41116680

    Incredibly the three lines successfully work together somehow as a cohesive piece – at least we think so – I wonder if you would agree?

    Finally, I’d like to share with you an experience I had the other day which I feel is related to the phenomenon of temporal tolerance somehow. Pottering around my house in this barmy summer I wandered into the toilet for a pee. As I walked past the radiator on the wall I felt the illusion that my leg was being heated up on the radiator’s side. Of course our central heating has been turned off for the whole summer but I was so convinced that I needed test it out just to check. It was cold.

    It is as though my syncing capacities are going beyond the call, bringing sensory phenomena together out of sheer habit, and filling in these sensory gaps.

    This whole area would be very exciting to explore further – more experiments in sound, more collaborations – very interested to see what you have in store for us as we work together on some spoken word pieces and for Moko and Loupe…

    Much to ponder,

    Leon

  2. Hey Leon – yesyes re. music and film and magical self-syncing. With the Tom Demajo video, there are so many things moving at the same time that it’s almost impossible for inflections in the music not to catch on something, and then that’s the something you see. Works very well. As for your incident with the radiator, it’s interesting, but I wonder if the bladderful of hot piss you had as you went into the toilet to pee contributed to the effect. Would it have been the same I wonder if you’d gone in for a shit?

Leave a Reply to Leon Cancel reply

Your email address will not be published. Required fields are marked *