A team of scientists who were trying to identify neurons that drive instinctive behaviors in mice uncovered something surprising on the way: brain cell types that exist only in female or male animals.

Other researchers have previously found genes that are switched on, or expressed, exclusively in male or female brains, but this is the first demonstration of sex-specific neuron types in mammals, the authors said.

In the study, which was published Thursday in the journal Cell, researchers at the California Institute of Technology and the Allen Institute binned brain cells into different groups based on the suite of genes each cell switches on. The resulting categories, commonly referred to as brain cell types, are defined by hundreds or even thousands of subtle differences in these sets of defining genes.

“To me, there is something very profound about the idea that there are cell types in the female brain that are not found in the male brain, and vice versa,” said David Anderson, Ph.D., a Professor of Biology at Caltech who led the study along with Caltech doctoral student Dong-Wook Kim. “Evolution went to all the trouble to select for a particular constellation of genes that are switched on to make a cell type in just one sex. That seems like a true structural difference.”

The researchers don’t yet know what these unique brain cells are doing in male or female mice, but they suspect the female-specific cells could play a role in mating — these cells switch on a gene known to be important in mating, the estrogen receptor, which allows the neurons to respond to the estrogen hormone in the brain.

Can we link brain cells to behavior? It’s complicated.

The research team was studying one section of the hypothalamus, an evolutionarily ancient part of our brains. This tiny part of the hypothalamus, known as VMHvl, plays an important role in instinctive behaviors such as aggression, social fear responses, and mating— in both male and female animals.

Anderson and his colleagues wanted to understand more about the specific cells that underlie these behaviors. Sifting through the 4,000 neurons in this small part of the brain, they uncovered 17 different types of neurons. They found three of those types primarily in the brains of male mice, and one of those types only in female brains.

They then looked at the connections those neurons make in the brain and which cell types are activated when animals were engaged in different social behaviors like fighting or freezing in fear in response to an aggressive mouse. They were expecting to find a one-to-one correspondence between cell types, where they connect in the brain, and specific behaviors. Many researchers thought that since the hypothalamus is so ancient, the rules that govern its organization might be relatively straightforward. But their findings proved otherwise.

“There doesn’t seem to be a fighting cell type or a mating cell type,” said Hongkui Zeng, Ph.D., Executive Director of Structured Science at the Allen Institute for Brain Science, a division of the Allen Institute, who is also a co-author on the study. “Even the most basic, innate behavior is based on complex interactions of individual cells across different types.”

Of the 17 cell types in this region known to drive these innate behaviors, the researchers only found one cell type with a clear link to both a specific behavior — social fear, or fear of aggressive mice — and connections to a single other region of the brain. None of the other 16 cell types showed such a clear relationship.

Anderson and Zeng are now working on a collaborative project to study cell types across the entire mouse hypothalamus. And the Caltech team is also delving further into the sex-specific neurons they identified to try to pin down what those cells are doing in the brain.

“This shows that even the parts of the brain that we assumed were going to be simple are very complicated,” Anderson said. “If we think the hypothalamus is an easy problem, we’re in for a long haul with the rest of the brain.”

The research described in this article was supported by the National Institutes of Health, including awards U19MH114830 from the National Institute Of Mental Health and U01MH105982 from the NIMH and Eunice Kennedy Shriver National Institute Of Child Health & Human Development. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.