AKA Decoding the neural code.

I'd like to share a project I've been involved in for over two years. I was employed at a university for a while and worked on data analysis associated with cybernetic eye implant development. The aim was to allow people with a group of neurodegenerative diseases to see again.

If you're into sci-fi or Elon Musk, you probably heard claims that this work can result in us in the future being able to communicate via similar implants. Or allow us to connect our brains directly to the Internet. While it is still under debate, this technology has a potential to be a start of an interesting branch of solutions for interfacing with our computing devices.

Could we use it to transfer images to our brains as well? When could we provide quality of life improvements to the people with fractured spinal cord? I don't necessarily have all the answers but I can tell you what I know about neurobiology and neural implants, where the technology currently is and where it could be in the near future.

The project was led by Pawel Hottowy and we were cooperating with the Chichilnisky Lab at Stanford. Follow the links if you want to get to know the hard scientific background behind what I'm simplifying here. Images in the article come from their work as well as mine and my colleague Tezeusz Woronko.

Introduction

Before I tell you more, I need to throw your way a few concepts from the domain. In a friendly way, I promise.

Neurons and neural signals

Here's a neuron:

And here's what an electrical signal it emits sometimes looks like when recorded:

A banger feature of a neuron is that it doesn't only emit electrical signals when reacting to stimuli (chemical, visual, pain, etc.). It also reacts to electrical stimulation by responding to it by propagating information further.

That's why we can influence how our neural circuits work by basically zapping the right neurons in the right way.

A single neuron is a fairly well understood element. The neural tissues, however, their structure and communication pathways, are very difficult to reverse engineer and decode or stimulate. In my work I was focused on understanding how we can replicate a signal sent from an eye to the brain, so we can replace faulty visual werware (photoreceptors of the retina) with some shiny electronics.

Why bother

This all sounds fairly abstract but there are strong reasons to be interested in this.

1. To understand and treat neurological disorders and injuries
   1. Spinal cord injury
   2. Alzheimer’s
   3. Limb prosthetics
   4. Sight loss
   5. Parkinson’s
2. Human enhancement

Current problems

Okay, it seems that we know what to do pretty well. What makes it interesting and difficult? My last hands-on experience is from two years ago but I know some of the problems still prevail.

1. We can only read and stimulate a small number of neurons - implants are pretty basic.
2. The stimulation and reading is non-selective (when stimulating one neuron we are usually stimulating multiple by accident).
3. Implants are chonky and cannot last decades.
4. Operations are risky.
5. Data analysis applied in neurobiology is extremely complex.

Why now?

What makes current time perfect for advancing this field? When the team was discussing it, two things came to mind.

1. Miniaturization of electronics (smartphone tech FTW) - it just wasn't possible before to create mobile supporting electronics small enough to be usable.
2. Advancements in computing - improvements in data analysis hardware and methodology make problems previously unfeasible to solve more approachable.

Retina

As I mentioned, the complexity is in the structures the neurons form and the way they send signals. The structure we were focusing on in our research was the eye retina. A lot of sight loss (e.g. retinitis pigmentosa) happens because photoreceptors (rods and cons) marked on the image below degenerate and lose ability to detect light. The curious thing is that in many of those cases, the rest of the neural tissue stays intact.

The important thing to know is that the retina is not only receiving light and passing "pixel" information to the brain. We know that some of the neurons (bipolar cells on the image above) are responsible for preprocessing images, including basic shape detection.

We were working on so-called epiretinal implants, ones applied at the top of the retina (see the image above). They stimulate neurons (RGCs) that connect directly to the brain, bypassing preprocessing "hardware". Because of that we wanted to learn how our retina converts light signals to neural impulses. This would allow us to send images recorded by a camera to the brain by stimulating the right cells.

Reverse engineering human retina

To build this understanding we had to come up with a set of the right questions.

What did we want to do?

We had several goals for our project, here they are in the order of immediate usefulness.

1. Make further research easier.
2. Help patients suffering from retinal neurodegenerative diseases.
3. Better understand how our nervous system works.

Key questions

These goals were reflected in our research questions.

1. How do we clean up the data from recording neuron activity?
2. Which neurons can we stimulate and record with a given electrode?
3. How to stimulate a particular neuron?
4. What neurons (and how) to stimulate to show the image?

We were primarily focused on contributing to answers to the first three while some of our colleagues from Stanford worked also on the fourth.

Hardware

The key thing to know is that when we lay an implant on top of a retina, it covers particular neurons like in the image below. Dots are electrodes in our array.

A convenient thing was that our electrode array was denser than any other solution out there. This allowed us to be so precise to even stimulate particular parts of the neuron at times.

Coding it

Getting information from the electrical signals recorded in the neural tissue was the actual challenge.

Data cleanup

The problem is that when we electrically stimulate a neural cell and try to record its behavior, the residual of a charge we used to excite it leaves a trace in the signal. In the image above, on the left you can see what we see when recording. On the right is actual neuron activity. The small bumps in the lines are different neurons in the area passing information. The group of big spikes that are visible on the left is called an "artifact" in the neurobiology field.

One of our biggest challenges was cleaning up the data so we could later work with what's on the right. We created an artifact detection and clean up algorithm. It was based on years of research by Pawel Hottowy and his coworkers.

Targeting the right neurons

After we had the right data, we needed to know which neurons' activity we were actually seeing. By using other methods, we built a library of neuron activity templates for each neuron’s activity detectable at a given electrode. Using them we could differentiate their signals. The image above shows how our template matching algorithm worked. It was a pretty simple approach that did wonders. On the bottom most row you can see what happened when we added detected neurons' templates. It resembled actually recorded activity with the only difference being a small amount of noise.

This algorithm allowed us to recognize which neurons are affected by a particular stimulating electrode and target exactly what we wanted.

Tools

This blog is mostly about programming. I may talk in detail about technical solutions sometime else, but here are the main tools we used for each part of the project. Skip it if what is below looks like dark magic to you.

Artifact detection was done with: Python, CUDA (pyCUDA), OpenCL (pyOpenCL). It was a statistical algorithm using a lot of domain knowledge. It was implemented to work on graphics cards (GPUs) which made it faster than anything else available.

Template matching was implemented in Python and C++17 (pybind11). The simple brute force algorithm was written in the most efficient, parallelized way, making it feasible to run even for large datasets.

What were our results

Okay, so we were able to clean up the data and recognize which neurons were activated. This was a groundwork for making anything else work. Thanks to it, we could stimulate retinas in the lab with different visual signals and then try to replicate it in the living organisms. That's what the team is still working on.

Shortly about our achievements:

• Our algorithm most likely had better efficacy compared to other solutions at the time.
• Our implementation was so efficient, it could feasibly be run on consumer grade electronics. This would make implants actually possible to develop.

When do I get my implant?

If you're a person with a vision impairment and you're willing to take a risk, you'll probably be able to get a decent device in the coming decade. While currently available implants are more of an experiment, the ones in development look really promising.

If you're looking to become a cyborg, then the answer is not very soon. Especially if you're not up for a surgery. However, with each patient treated, we are getting closer to developing a consumer-grade technology. There is a growing amount of ideas that aim to provide some benefits with decreased invasiveness. If we actually come up with a worthy use case when the appropriate devices become available - why not? Sure there are ethical and safety concerns but they're purely theoretical at this point.

I am really proud to be able to say that I did my small part in the creation of the described technologies. I was really close to actually publishing a scientific paper describing mine and the team's accomplishments in the field. Although I now think I could have spent some more time to see this through, I am happy with my contributions.

Hope this text inspires you. If you're a software engineer, you really can work in every field you can imagine. Go, do it.

Scientific references

[1] G.D. Field and E.J. Chichilnisky, Information Processing in the Primate Retina: Circuitry and Coding, Annual Review of Neuroscience 2007 30:1, 1-30

[2] Hottowy P, Skoczeń A, Gunning DE, et al. Properties and application of a multichannel integrated circuit for low-artifact, patterned electrical stimulation of neural tissue. Journal of neural engineering. 2012;9(6):066005. doi:10.1088/1741-2560/9/6/066005.

[3] Sekirnjak C, Hulse C, Jepson LH, et al. Loss of Responses to Visual But Not Electrical Stimulation in Ganglion Cells of Rats With Severe Photoreceptor Degeneration. Journal of Neurophysiology. 2009;102(6):3260-3269. doi:10.1152/jn.00663.2009.

[4] Langenkämper D, Jakobi T, Feld D, Jelonek L, Goesmann A, Nattkemper TW. Comparison of Acceleration Techniques for Selected Low-Level Bioinformatics Operations. Frontiers in Genetics. 2016;7:5. doi:10.3389/fgene.2016.00005.