Inferotemporal cells represent both conscious and unconscious percepts in binocular rivalry

 

Hesse & Tsao just published this great study. It’s a binocular rivalry study (BR). In BR, incompatible stimuli are presented to each eye –– for instance: Obama’s face, and a Taco. Conscious perception alternates between the two every few seconds. So, researchers can compare conscious perception of a stimulus (e.g. Obama’s face) with unconscious processing of the same stimulus (e.g. Obama’s face when it is suppressed and the subject is conscious of the taco instead).

What’s new about Hesse & Tsao’s study? They developed a new no-report paradigm. It’s probably better than other no-report paradigms. But I’m not so sure the no-report aspect of the study is so important. They focus mostly on neuronal activity in the inferotemporal (IT) cortex, which shouldn’t exhibit so much report-related activity –– compared to PFC, for instance. So I won’t talk about the “no-report” aspect of this study (despite the hype, no-report is not new, and quite often unnecessary, in my humble opinion).

From a methodological perspective, the main new thing with this study – compared for instance to previous work by Logothetis et al. addressing similar questions – is more advanced recording and decoding. Hesse & Tsao recorded from multiple (mostly face-selective) IT neurons simultaneously during BR, and could decode the representational contents carried by those populations on a trial-by-trial basis.

Now the result. As expected IT neurons modulate their activity with BR alternations. But the main new and exciting finding is that these neurons encode both the conscious percept and the suppressed percept at the same time. That’s a case of mixed selectivity. IT neurons do track the conscious percept, but they also track the unconscious percept.

Since we can decode unconscious contents from IT neurons, it means that a special kind of population code in IT neurons correlates with the conscious content, and not any neuronal activity. That’s a really cool finding.

On Twitter Doris Tsao opposed these findings to a common – but incorrect – interpretation of earlier single-neuron recording findings, by Sheinberg & Logothetis. 

Here’s, for instance, what Koch et al. (2016) write about this: “Single-neuron recordings in monkeys, carried out during paradigms such as binocular rivalry, suggest that activity in most V1 neurons is linked to the identity of the physical stimulus rather than the percept. This contrasts with the activity of neurons higher up in the visual hierarchy, which correlates with the percept rather than the stimulus.”

To the best of my knowledge, Logothetis et al. never claimed that IT neurons track only the conscious percept. In any case, Hesse & Tsao have now proved this story wrong. They also convincingly argue that there’s no subset of IT neurons dedicated only to the representation of the conscious percept.

So that’s it? We’ve found that the neural correlate of consciousness (of faces) is a special kind of population coding in IT neurons?

Probably not. Binocular rivalry can happen for unconscious percepts. There’s also evidence that binocular rivalry – at least for vertical vs. horizontal gratings – operates at an earlier level than other forms of suppression like dichoptic metacontrast masking. (And metacontrast masking likely operates earlier than crowding, which itself probably operates earlier than object substitution masking)

So, the mechanism responsible for rivalry is upstream from the mechanism responsible for consciousness of contents. Which means that whatever contrast you get from comparing conscious vs. suppressed contents encompasses neural correlates of consciousness and a set of pre-requisites of consciousness.

Now, there’s a big difference between finding the neural correlate of a content of consciousness, and finding the neural correlate of consciousness of that content. 

Hesse & Tsao have convincingly shown that the conscious perceptual content correlates with a specific population code in IT. They have also shown that the same neuronal population can carry both conscious contents and unconscious contents at the same time. That’s a nice discovery.

Does it mean that it is in virtue of being coded in a specific manner by IT neurons that the content is conscious rather than non-conscious? We don’t really know. We still don’t know whether this kind of population coding in IT would be sufficient on its own for consciousness of the representational content carried by this neuronal population. But with more studies like Hesse & Tsao’s we’ll surely get there.

I thank Janis Hesse & Doris Tsao for discussing their study with me and for reading a draft of this blog post.

The unconscious lag between reality and conscious perception

It seems that we consciously perceive events exactly as they unfold in time. But that’s probably wrong. Here’s an example.

On the left, a green disk and a red disk are successively presented at the same location for 20 milliseconds each. On the right, the red and green disks are successively presented in different locations. The disks on the right are clearly visible. On the left, however, if the effect works as planned, you don’t perceive the green and red disks, but only a single yellow-ish disk. In this case the effect partly depends on the properties of your screen – it’s better if the disks are equiluminant – so it’s possible that the demonstration above doesn’t work so well for you. In any case, this phenomenon is called “color fusion”, and in proper lab settings, subjects can’t discriminate between the fused green / red disks and a yellow disk.

If we perceived events exactly as they unfold in time, we’d perceive a green disk quickly followed by a red disk. That’s not what we see.

It’s possible to show that the green and red disks are processed in your visual cortex. That’s evidence that you do see them. But it doesn’t feel like it. Why? A reasonable hypothesis is that your brain/mind unconsciously integrates the green and red disks during 40 milliseconds, and then you become conscious of the result of this unconscious processing as a single, integrated yellow disk.

This is an example of a postdictive effect. The presentation of the red disk causes a change in the way you see the green disk, even if the red disk is presented after the green disk. A wide variety of postdictive effects have been documented in the 100 to 150 milliseconds range. The way you consciously perceive something at time t depends on what’s going to happen at time t + 150 milliseconds – approximately the time of an eye blink.

I find these effects fascinating, but they’re not revolutionary. Unless you’re lucky enough to have an immaterial soul doing that job for you, signal processing performed by a material object like the brain has to take some time. As you’re reading this, it already takes approximately 50 or 60 milliseconds for visual signals to go from your retina to your visual cortex. So, I don’t find it so hard to believe that conscious perception can be preceded by 150 milliseconds of unconscious processing.

Some postdictive effects, however, can operate over a longer time range. In a recent article, Herzog, Drissi-Daoudi, and Doerig, review long-lasting postdictive effects in the 350-450 milliseconds range. I’ll talk about some of their main findings here, but I really encourage you to read the article to get the full picture. With Adrien Doerig, we also talked about some of the consequences of this research for theories of consciousness, so if you're interested you can check this article.

 

 

Here’s an example with the “Sequential Metacontrast Masking Paradigm”. This paradigm involves Vernier stimuli (the vertical bars). These stimuli integrate very well when presented one after the other, a bit like the colored disks above. If a Vernier with an offset to the right is followed by a Vernier with an offset to the left (call that an anti-Vernier), what observers report seeing is just a single Vernier with no offset at all (a “neutral Vernier”). This is a postdictive effect: the anti-Vernier changes the appearance of the Vernier even if the former is presented after the latter.

This integration also works when the Verniers appear to move. As illustrated in the figure above (V-PV3), the offset of the first Vernier is “transported” in the stream of Verniers, even if the subsequent Verniers are neutral. Participants just report seeing a Vernier moving to the right, and no neutral Verniers.

Here’s the main finding now. As depicted in the figure above (V-AV3), if a Vernier is followed by neutral Verniers, themselves followed by an anti-Vernier, observers report perceiving a stream of neutral Verniers. It’s a bit as if the anti-Vernier reached back in time to change the way in which the entire stream of Verniers is perceived.

How far back in time? Well, that’s where things get a little crazy… An anti-Vernier can change the way in which the entire stream of Verniers is perceived even if it appears 450 milliseconds after the first Vernier has been presented. This means that the reported appearance of the stream of Verniers depends on what will be presented 450 milliseconds after the stream has started. Yes, you read that correctly. 450 milliseconds. This finding suggests that, at least in this case, there’s a window of 450 milliseconds of unconscious processing before the entire stream is perceived.

This experiment also provides evidence that conscious perception is constituted of discrete episodes, instead of being continuous. But I won’t focus on this aspect here. I won’t focus either on all the other cases of long-lasting postdiction reported by Herzog et al. Instead, let me finish with a short reflection on what this research teaches us about consciousness, and baseball.

Baseball is a game of milliseconds. A fastball thrown at 100mph (161 Km/h) reaches the batter in about 375 milliseconds. If Herzog et al. are right, it takes 400ms for the batter to construct a conscious percept of the ball. Which means that the batter’s conscious visual percept of the ball lags far behind its actual location. In fact, it lags so far behind that the batter hits the ball before she consciously perceives the pitch leaving the pitcher’s hand.

 

To illustrate a similar – though less extreme – scenario, Nijhawan & Wu (2009) made this great picture:

 

 

The white circle represents the hypothetical perceived location of a tennis ball travelling at 60mph, compared to its actual location, assuming a processing latency of just 100 milliseconds. I let you imagine the perceived location of the ball if the latency is actually 400 milliseconds, as Herzog et al. convincingly argue.

That's just unbelievable. I don’t know much about the phenomenology of hitting a 100mph fastball. But my guess is that baseball players don’t feel like they swing the bat before consciously seeing the ball leaving the pitcher’s hand. Instead, they probably feel like they swing the bat at the moment they consciously see the ball approaching towards home plate.

If Herzog et al. are correct, and if that’s indeed how baseball players feel, baseball players are deeply wrong. It’s impossible for them to actually hit the ball at the moment they consciously see it approaching towards home plate. Instead, they unconsciously process the type of pitch the pitcher has thrown, whether it’s going to end up in the strike zone or not, whether they should swing or not, and how. All that, long before they consciously perceive the actual trajectory of the ball.

This leaves us with a puzzle. How can we, baseball players, and Roger Federer of all people, be so wrong? How is it possible that we don’t realize that our conscious experiences can lag 400ms behind reality? I think that two phenomena can independently contribute to the fact that we don’t realize it. Just a caveat before I continue: I’m not an expert on this, so that’s entirely speculative and I’m probably wrong.

First, most of the time the visual world isn’t like an unpredictable stream of Verniers stimuli, and most of us don’t spend our lives guessing what the next pitch is going to be. In ordinary situations, conscious percepts could be constituted before the entire bottom-up visual processing is over, based on predictions of what’s likely to come next. So, your stream of consciousness could “catch-up” with reality if predicting what’s coming requires less processing time than processing incoming sensory information from scratch.

Second, consciousness could be constituted of discrete episodes, and the intuition that the stream of consciousness is a continuous succession of feelings is an illusion. That’s the view held by Herzog et al. I still have a hard time understanding it, even if I’m starting to see the appeal, so I apologize if what comes next is a bit confusing.

 

Let's start with an analogy. A picture can represent 3-dimensional spatial relations between depicted objects, even if the picture itself only has two dimensions. Just as a picture doesn’t need to be 3-dimensional to represent spatial relations in a 3D space, your experience doesn’t need to be continuous to represent a succession of events. According to Herzog et al.'s view, your experience represents a succession of events, which gives rise to a feeling of succession, even if the experience itself is not constituted by a succession of feelings.

Here’s what actually happens if this view is correct. In the case of the stream of Verniers, for instance, we do not have a succession of experiences – each experience representing a single Vernier in the stream. There’s no succession of feelings. Instead, there’s just a feeling of succession. The entire stream of Verniers is experienced in a single conscious experience with temporal properties assigned to each element of the stream. The fact that elements of the experience are “tagged” with these temporal properties give us a feeling of succession in a single, discrete conscious episode, even if there’s no actual succession of feelings.

According to this view, in between those discrete conscious episodes – which give you the impression that consciousness is continuous, you’re not conscious of anything. We live most of our lives as zombies, unconsciously accumulating information, and reconstructing conscious experiences of what just happened after the facts.

It’s starting to be a bit too vertiginous for my taste, so I’ll stop here. There’s a lot to think about in Herzog et al.’s article. I thank Adrien Doerig for discussing these issues with me, and for reading a previous draft of this blog post.