Right, so artificial networks also need sleep!

Quick Hits
Daily brief research updates from the cognitive sciences

Ishould say that it is artificial neural networks that seem to need sleep, or a rest. But isn’t one point of an artificial network that that of it not needing sleep, or rest?

For those of you who don’t know, artificial neural networks are networks built by engineers in the computing space to mimic the brain’s cells and therefore hope to get better computing, or different, computing outcomes. Therefore, it comes as a real surprise that, apparently, rest improves their performance. It comes as a real surprise because these are not biological entities, there are numerous reason we need rest as human beings, as biological beings. One is that there is a slow build-up of toxic material as we function. Another is that there is also constant genetic damage that needs to be repaired.

So why do these networks need rest? First let’s understand what happens. These networks are designed to mimic neuronal functions – so far so good – and they have become really good at some things such as computational speed. But there is something called catastrophic forgetting – no, not like when we stand in the supermarket and can’t for the life remember what we wanted. This is when these networks learn sequentially, one task after another, new information can then overwrite old information and it is gone, “forgotten”.

Golden et al. at the University of California have now reported that when these artificial networks are trained on new tasks but with periods off-line mimicking sleep, they could replay old memories but without using old training data. This is due to the patterns that are replicated during our biological sleep. In our sleep the synapses, connections that is, between neurons are strengthened. You brain essentially replays your day’s input and strengthens memories during sleep (that’s one of many reasons sleep is so important). When this process was replicated it mitigated this catastrophic forgetting.

So, fascinating it is that we are building artificial neural networks that replicate the brain’s processes for better computational processing power but also fascinating that these artificial networks also improve performance with sleep.

This goes to show that good old biology, and evolution, seems to have got it right. And for us also another reminder of the importance of getting a good night’s sleep.

Andy Habermacher

Andy is author of leading brains Review, Neuroleadership, and multiple other books. He has been intensively involved in writing and research into neuroleadership and is considered one of Europe’s leading experts. He is also a well-known public speaker, speaking on the brain and human behaviour.

Andy is also a masters athlete (middle distance running) and competes regularly at international competitions (and holds a few national records in his age category).

twitter / LinkedIn

References

Ryan Golden, Jean Erik Delanois, Pavel Sanda, Maxim Bazhenov. 
Sleep prevents catastrophic forgetting in spiking neural networks by forming a joint synaptic weight representation.
 PLOS Computational Biology, 2022; 18 (11): e1010628
DOI: 10.1371/journal.pcbi.1010628

More Quick Hits