What would it mean to decode a non-trivial memory from a map of the brain?
Some reflections on the Aspirational Neuroscience Prize challenge
Ultimately, function in neurobiology must derive from structure. But, to date, nobody has shown that it is actually possible to reconstruct meaningful electrophysiological patterns corresponding to memories from static structural information.
While a lot of people think this reconstruction should be possible eventually, there are major uncertainties about what types of structural information in the brain will be necessary for reconstructing different types of memories. If it weren’t for these uncertainties, evaluating the quality of brain preservation would be much easier because we’d know what to measure the preservation of.
The Aspirational Neuroscience Prize aims to help resolve some of these uncertainties. In addition to four annual (or close to annual!) prizes that highlight recent research into how learning and memory are physically encoded in the brain, they also have a main challenge prize for a group that can “go the distance”. Specifically, the main challenge prize is awarded to the group that can “decode a non-trivial memory from a static map of synaptic connectivity”.
The main question I had when I read this criterion was “what exactly does a non-trivial memory mean”? It seems wishy-washy. But that’s actually part of the point — it’s supposed to be a little bit unclear what exactly the goal is because our conceptual understanding of the problem is still primitive.
But first, let’s talk about synaptic connectivity
Before we talk about what a non-trivial memory refers to, we need to briefly touch on another part of the challenge criterion, which is “a static map of synaptic connectivity”.
I like the word “static” — yes, structure alone without recorded functional data is the whole point. I also like the word “map”. Maps are cool. But “synaptic connectivity” is problematic.
Almost certainly, reconstructing information represented in the brain will require mapping synaptic connectivity. But in my view, it will also almost certainly require mapping cellular morphology and some amount of biomolecular annotation. It seems to me that there will be too much variation mediated by neurite morphometry, ion-interacting proteins, myelination, and other structural features for engrams to be captured by synaptic connectivity alone.
Thus, I prefer the more agnostic term "brain map", which could potentially include those other structural features, rather than just "synaptic connectivity", which makes other structural features seem not required or even not included. I would amend the criterion to “decode a non-trivial memory from a static brain map”.
Back to what a non-trivial memory means
A friend of mine recently posed me a related challenge: to reconstruct a bit of information stored in preserved brain tissue. Since — arguably — nobody has ever done this, one might say that this would entail going from zero to one bits of information reconstructed. Always a major milestone.
I slept on it a couple of nights and then I thought to myself: has the research group I’m a part of already done this? In our recent paper, we used deep learning to classify fixed brain tissue as coming from brain donors with or without antemortem clinical evidence of cognitive impairment. Our model performed above chance, seeming to focus on the white matter of the brain.
Our lowest p-values for the classification task were around p = 0.002, which, under various assumptions that I won't go into here, can be converted to more than one bit of information.
Did we just go from zero to one bits of information extracted from preserved brain tissue? Sadly, the answer is most certainly not. First, this type of study has been done before, so we weren't at zero bits prior to our study.
Second, this isn’t the type of information that we care about when we talk about information in the brain. It’s information contained in the brain tissue, but it’s not information represented by the brain like a memory would be.
Another example of something that wouldn’t count as a non-trivial memory
Let's say that you are studying banked brain tissue from a group of brain donors. A hypothetical question is whether the person who donated their brain ever had the experience of being near an area with very high radioactivity. This might lead to the deposition of high amounts of carbon-14 in the preserved brain.
If you stretch the definition, you might think of the presence of carbon-14 as being a molecular “memory” of having been in that location. And measuring carbon-14 might allow you to tell whether the brain donor had been in a highly radioactive area to a high degree of accuracy. But it would also be trivial from the perspective of extracting memories because it’s not information that is represented by the brain.
Here’s something that would count as reconstructing a non-trivial memory
A couple of weeks ago, an amazing article was submitted to ICLR 2023. It’s currently anonymous while under review.
In this study, the authors used StyleGAN3 to generate faces from random latent vectors and then showed those faces to a macaque with cortical implants in a passive fixation task.
This allowed them to record electrophysiological data while the macaque was viewing the images.
They then trained a multiple linear regression decoder model to predict how latent vectors were dependent on brain activity.
This allowed them to reconstruct the faces that the macaque was looking at in a test set of held-out faces that was not part of the model training.
The results are incredible and speak for themselves. Below, the stimulus is the face that was presented to the macaque and the reconstruction is what the model predicted the macaque was looking at:
They also used different types of metrics to evaluate the performance of the reconstruction. This type of metric could be used by the Aspirational Neuroscience Prize to evaluate the accuracy to which a memory has been decoded since decoding is very unlikely to be perfect. Perhaps they could require a certain threshold to be passed.
The problem with this study from the perspective of the Aspirational Neuroscience Prize challenge, i.e. why it wouldn’t win the main challenge prize, is because it records electrophysiological data.
We can imagine an alternative scenario that would win the prize. This would require for the macaque’s brain structure to be mapped, virtual electrophysiological data to be predicted based on a model of how the brain works, and then that emulated electrophysiological data to be decoded using a similar approach as in this study.
Obviously, predicting electrophysiological responses to a stimulus from a static brain map is a much, much harder problem than recording electrophysiology. We can’t even do it on C. elegans yet.
Quite plausibly, you would need a full whole brain emulation or close to one, alongside activation experiments of particular neural pathways, to be able to identify this type of memory from the preserved brains. I think we’re pretty far away from this.
Could we theoretically skip the electrophysiology emulation step and go straight from brain structure to stimuli response prediction? Eventually, I think probably yes, although this would be even harder and might require some kind of “code” to be discovered in neural structure, which perhaps only statistical models will be able to identify.
I don’t know if semantic memories are the only type of memory that is non-trivial
In this study, the decoded face representations are a type of semantic memory. I originally thought about suggesting that the Aspirational Neuroscience Prize challenge should be switched to “semantic memory” rather than “non-trivial memory” based on this study, as well as reading a review about neural representation that touches on the importance of the distinction between semantic information and Shannon information.
But then I decided that I don’t know enough about memory to say that. As far as I can know, non-semantic types of memory could also be non-trivial.
Who cares?
When Richard Semon introduced the term engram in the early 1900s, he declined to speculate upon the neural substrates of engrams, instead writing “to follow this into the molecular field seems to me… a hopeless undertaking at the present stage of our knowledge; and for my part I renounce the task”.
Some might think it is not possible at all, with any level of structural resolution detail, to reconstruct a memory from static brain tissue. Although, I would like to hear their reasoning.
If we can extract a bit of non-trivial memory information from a preserved brain, we can get a much more detailed sense of what the problem to be solved is. For example, it might suggest that some types of engrams would require brain mapping at different levels of resolution than others.
That’s why I care about this topic. Others probably have other reasons for caring.
It’s definitely worth pointing out one of my main concerns about this research topic. Which is: activation experiments based on data from preserved brain tissue, even in silico, might lead to conscious experience. In my view, it’s essential to not perform these experiments until they can be done in a way that either prevents a phenomenological experience or promises a high-quality experience. Otherwise, they should not be done.
Interesting post.
I think about neurophysiological memory in the same way I think about memory in general - memory is what specifies a particular state out of a set of possible states. Given an adult human brain has ~85 billion neurons with 10,000 synapses for each one, that's obviously an awfully large space of possible connections or 'states' (without even considering their activity levels).
I think thinking about a C Elegans gives a better way of thinking about triviality vs. non-triviality. We know each worm has 302 neurons, and we know all the patterns of connections between them. In a sense these neural connections are a kind of memory - they encode the evolutionary memory of which connection setups produce successful worm behaviour, out of the set of other possible ways of wiring up those neurons. This kind of memory is 'trivial' though when considering within-lifetime memory formation (as opposed to genetic memory) as it's preset, the same way neural circuitry controlling basic regulatory functions like heart rate in a human would be a 'trivial' genetic memory.
What is non-trivial is any kind of memory that indicates what happened to that particular worm, as opposed to any other c. elegans. A human equivalent would be some synaptic or morphological information that tells you something specific about that particular human's experiences, as opposed to a generic human. I think anything in this vein counts as 'non-trivial', and the better you can specify the particular experience the animal/human had out of a set of possible experiences they may have had, the better a job you've done of decoding a non-trivial memory.
Brilliant! Btw, an alternative candidate as example of using electrophysiology (rather than static structure) would be the work by the Berger team at USC, which has already involved human volunteers who were implanted and tested while recovering from surgery for epilepsy. The spatio-temporal patterns recorded there were clearly memory-related and were even used to generate a model with which to stimulate and support memory encoding in the same volunteers.