Scientists used advanced genetic engineering techniques to transform a bacterial protein into a new research tool that could detect subtle, real-time changes in brain serotonin levels during sleep, according to National Institutes of Health-funded research published in Cell.

Preclinical experiments, primarily in mice, also showed that the sensor could detect changes in brain serotonin levels during social interactions, as well as test the effectiveness of new psychoactive drugs. The study was funded, in part, by the NIH’s Brain Research through Advancing Innovative Neurotechnologies (BRAIN) Initiative which aims to advance the medical world’s understanding of the brain under healthy and disease conditions.

The study was led by researchers in the lab of Lin Tian, PhD, principal investigator at the University of California Davis School of Medicine.

In the study, the researchers transformed a nutrient-grabbing, Venus flytrap-shaped bacterial protein into a sensitive sensor that fluorescently lights up when it captures serotonin. Previously, scientists in the lab of Loren Looger, PhD, Howard Hughes Medical Institute Janelia Research Campus, Ashburn, Virginia, used traditional genetic engineering techniques to convert the bacterial protein into a sensor of the neurotransmitter acetylcholine.

The protein, called OpuBC, normally snags the nutrient choline, which has a similar shape to acetylcholine. For this study, the Tian lab worked with Looger’s team and the lab of Viviana Gradinaru, PhD, Caltech, Pasadena, California, to show that they needed the added help of artificial intelligence to completely redesign OpuBC as a serotonin catcher.

The researchers used machine learning algorithms to help a computer “think up” 250,000 new designs. After three rounds of testing, the scientists settled on one. Initial experiments suggested that the new sensor reliably detected serotonin at different levels in the brain while having little or no reaction to other neurotransmitters or similarly shaped drugs.

Experiments in mouse brain slices showed that the sensor responded to serotonin signals sent between neurons at synaptic communications points. Meanwhile, experiments on cells in petri dishes suggested that the sensor could effectively monitor changes in these signals caused by drugs, including cocaine, MDMA (also known as ecstasy) and several commonly used antidepressants.

Finally, experiments in mice showed that the sensor could help scientists study serotonin neurotransmission under more natural conditions. For instance, the researchers witnessed an expected rise in serotonin levels when mice were awake and a fall as mice fell asleep. They also spotted a greater drop when the mice eventually entered the deeper, REM sleep states.

In addition, the scientists saw serotonin levels rise differently in two separate brain fear circuits when mice were warned of a foot shock by a ringing bell. In one circuit, the medial prefrontal cortex, the bell triggered serotonin levels to rise fast and high whereas in the other, the basolateral amygdala, the transmitter crept up to slightly lower levels.

In the spirit of the BRAIN Initiative, the researchers plan to make the sensor readily available to other scientists. They hope that it will help researchers gain a better understanding of the critical role serotonin plays in our daily lives and in many psychiatric conditions.