The rapid progress of Machine Learning (ML) fuelled by Deep Neural Networks (DNN) in the last several years has created an impact in a wide variety of fields ranging from computer vision, speech analysis, natural language processing etc. With the progress in software, there has been a concomitant push to develop better hardware architectures to support the deployment as well as training of these algorithms [1], [2]. This has rekindled an interest in “Neuromorphic Engineering”–a term coined in 1990 by Carver Mead in his seminal paper [3] where he claimed that hardware implementations of algorithms like pattern recognition (where relative values are of more importance than absolute ones e.g. is this image more likely to be a cat or a dog?) would be more energy and area efficient if it adopts biological strategies of analog processing.S
While the above idea of brain-inspired analog processing is very appealing and showed initial promise with several interesting sensory prototypes, it failed to gain increased traction over time possibly due to the potential difficulties of creating robust, programmable, large-scale analog designs that can benefit from technology scaling in an easy manner.
Fig. 1. (a) Google search trends over the last 15 years for the topic “Neuromorphic Engineering” shows a decline around 2010 followed by a renewed interest in the last 5 years. (b) Number of neuromorphic papers published in journals from the Nature series have shown a steady increase in the last 10 years. Data for 2019 is till the month of October.
However, in the last 5 years, there has been renewed interest in this topic, albeit with a slightly expanded connotation of the term “neuromorphic”. Figure 1(a) shows a history of google searches of the term “neuromorphic engineering” over the past 15 years (obtainable from Google Trends). Data points are plotted for every month with the maximum search number normalized to 100. It can be seen that there was a decline in interest about neuromorphic research around 2010. However, it has again gained momentum in the last five years with a slightly broadened scope which we refer to as version 2 (while referring to the Meadian definition as version 1). A similar trend (plotted in Figure 1(b)) is obtained also by analyzing the number of papers published in relevant journals (Nature, Nature Communications, Nature Electronics, Nature Machine Intelligence, Nature Materials, Nature Nanotechnology) from the Nature journal series over the last years that are on the topic of neuromorphic research. It can be seen that there is a rapid increase in the number of such papers over the last 5 years.
The rest of the paper is organized as follows: the next section introduces the new connotation of the term “neuromorphic” followed by an analysis of some recent research trends in this field. Section IV describes the need for neuromorphic benchmarks and some desired criteria of such benchmarks while Section V proposes brain-machine interfaces as a potentially good benchmark.
As discussed in the last section, the renaissance in Neu- romorphic research over the last 5 years has seen the term being used in a wider sense than the original definition [3]. This is partially due to the fact that scientists from different communities (not only circuits or neuroscientists) ranging from material science to computer architects have now become involved. Based on the recent work, we describe next the key characteristic features of this new version of neuromorphic systems as:
• Use of analog or physics based processing as opposed to conventional digital circuits–this is same as the original version of neuromorphic system from a circuits perspective.
• From the viewpoint of computer architecture, usage of non von-Neumann architecture (independent of analog or digital compute) and low-precision digital datapath are hallmarks of neuromorphic systems. In other words, conventional computers using von-Neumann architectures read from memory, compute and write back the result– this is very different from brain-inspired systems where memory and computing are interspersed [4].
• Computer scientists and algorithm developers on the other hand consider a system neuromorphic if it uses a spiking neural network (SNN) as opposed to a traditional artificial neural network (ANN). Neurons in an SNN inherently encode time and output a 1-bit digital pulse called a spike or action potential. We next illustrate how frequently each type of viewpoint is expressed in neuromorphic research. Figure 2 categorizes the neuromorphic research papers published between 2017-2019 in the Nature series of journals surveyed in Figure 1(b) along with the journals Science and Science Advances. The papers are categorized according to the neuromorphic aspect they primarily focus on–(1) Analog processing, (2) non vonNeumann architecture or (3) SNN. It can be seen that a large majority of the work focussed on the SNN aspect (details of papers used in the survey are available at [5]). Most of these work focus on new materials or device fabrication and then present SNN simulations using the novel device properties
Fig. 2. Survey of neuromorphic systems reported over 2017-2019 in Nature, Science, Science Advances, Nature Nanotechnology, Nature Electronics, Nature Materials, Nature Communications . A large majority use SNN in their work. Details of all papers used in the survey are in [5].
Fig. 3. Survey of IC implementations of non von-Neumann architecture over the same period in ISSCC, SOVC, JSSC however shows very few work uses the term “neuromorphic”. Details of all papers used in the survey are in [5].
bypassing the circuit level. Hence, we also decided to create a survey of ML accelerator integrated circuits (IC) published in IEEE ISSCC and IEEE SOVC conferences inspired by the ADC survey [6]. In addition, we also considered papers published in the IEEE Journal of Solid State Circuits (JSSC). Figure 3 plots the result of categorizing all ML accelerators adopting non von-Neumann architecture published between 2017-2019 (details in [5]). Surprisingly, it can be seen that only 5 papers have used the term “neuromorphic” to describe their work! This clearly shows a stark difference in terminology used across different research communities. This leads us
Fig. 4. A new taxonomy that has non von-Neumann architecture as the overarching topic with neuromorphic v2.0 and ML accelerators as two subtopics under it.
to propose a new taxonomy for neuromorphic systems as shown in Figure 4. It is possibly better to use the term non von-Neumann architecture as the overarching topic. Under its ambit, neuromorphic v2.0 can refer to systems using analog or mixed-signal circuits, implementing SNN algorithms or the extremely quantized version (1-bit) of ANNs. On the other hand, ML accelerators can refer to digital circuits with non von-Neumann architecture implementing multi-bit ANN. With this in mind, we look at some recent performance trends in ML accelerators using non von-Neumann architectures that were reviewed in Figure 3.
There are several important metrics to quantify the performance of ML accelerators such as energy efficiency, throughput and area efficiency. To identify some trends, we plot several combinations of these quantities in Figure 5.
First, we expect bigger chips to have lower energy efficiency in general due to cost of moving data around large areas that dissipates more energy charging and discharging interconnects. Since the area of these ICs are dominated by the static random access memory (SRAM) required to store weights and activations, we use the SRAM size as a proxy for chip area. The energy efficiency in Tera operations (TOPS) per Watt are plotted against SRAM size for these designs in Fig. 5(a) and (b) and indeed show an inverse relation between energy efficiency and SRAM size or chip size. Figure 5(a) further uses different colours to categorize the data points according to bit width of datapath. As expected, it can be seen that the extremely quantized 1-bit designs [7]–[9] show best energy efficiency and are located significantly () above the trend line. The same data is plotted in Fig. 5(b) but colour coded according to the design approach of digital versus analog mixed-signal. It is interesting to note that the mixed signal designs indeed exhibit higher energy efficiencies, but they are in general much smaller than the digital ones.
Thus, in general we can see that the neuromorphic v2.0 principles of non von-Neumann architecture coupled with low data precision and analog computing (described earlier in Section II) do indeed provide great energy efficiencies. However, it can be seen that the energy efficiencies of pure digital approaches using only the principles of non vonNeumann architecture and low bit-width are at least much higher () than the energy efficiency wall of
GMACs/W for traditional von-Neumann processors [10], [11]. Hence, this raises an interesting question–given the scalability, testability and ease of porting across nodes offered by digital designs, is it reasonable to expect large scale industrial adoption of analog neuromorphic designs for an extra 10X in energy efficiency?
Next, we analyze the trade-offs in throughput at peak energy efficiency by plotting it against peak energy efficiency in Fig. 5(c). Interestingly, these two quantities are positively correlated with a majority of designs exhibiting throughput GOPS. Higher throughput would generally mean the static power is better amortized across the operations leading to higher energy efficiency. Also, in general reduced bit precision designs that increases energy efficiency would also reduce critical path delays increasing throughput. Lastly, we analyze area efficiency of the designs (measured in GOPS/mm
) by plotting it against energy efficiency in Fig. 5(d). Again, these two quantities show a positive correlation implying again that good design practices of reduced bit precision and analog design positively impact both the quantities. This is also clarified in Figure 5 by demarcating the designs according to bit precision and design styles. These plots show that apart from energy efficiency, analog mixed signal design styles also provide
improvement in throughput and area efficiency. Coupled with energy efficiency advantages, these points might be sufficient to suggest that in the longer term, there is reason for large scale interest in neuromorphic designs following the principles outlined earlier. However, all of these comparisons are not very relevant unless they can all run a common set of benchmark problems. This is discussed next in the following section.
The comparison between all the hardware designs in the earlier section are not fair unless they can all at least report performance on a minimum set of benchmark algorithms. While there are at least some common benchmarks for the ANN community such as MNIST [12], CIFAR [13] and Imagenet [14] for image recognition, there is not much consensus about good benchmarks for neuromorphic SNN algorithms. Hence, while advocating usage of benchmarks from the ML community for neuromorphic hardware ANNs, we discuss more details about what might constitute desired criteria for SNN benchmarks. While this topic is deemed important, there has been very few dedicated efforts in this area [15]. Given the role the Imagenet benchmark played in catalysing progress in ANN research, we believe it is of utmost importance that the neuromorphic community spend more effort immediately on devising good benchmarks for SNN.
Some recent work on SNN has focussed on converting images from ANN benchmarks to spike trains and then classifying them [16], [17]. While being great pieces of research, we feel that this is fundamentally not a good application for SNN since the original input signal is static and does not change with time. Instead, it might be more natural to use SNN as dynamical systems to track moving objects in video streams [18], [19] or classify signals that vary over time such as speech [20]. With this in mind, we propose the following desired characteristics for neuromorphic benchmarks:
1) The signal being processed should be encoded in time naturally so that the continuous time dynamics of SNN can be more effective than ANN to process it. Signals such as speech, video, etc are good examples. From the biomedical domain, EEG signals are another good example.
2) There should be a need for real-time response of the system such as in closed-loop systems such as sensorimotor loops in robotics. The rapid response time of
Fig. 5. Machine learning hardware trends: Peak energy efficiency in TOPS/W plotted against memory size and categorized by (a) bit-width of datapath and (b) digital vs mixed-signal analog approaches. (c) Throughput at peak energy efficiency and (d) Area efficiency plotted against peak energy efficiency for recent ASIC implementations reported over 2017-2019 in ISSCC, SOVC, and JSSC. The larger dots in (d) indicate ASIC area without pad.
neuromorphic sensors and SNN processing should be useful in such cases.
3) There should be need for the system to adapt or learn frequently. This necessitates learning from few samples, a common complaint with current deep learning based ANNs that require many thousands of examples to train.
4) Ideally, the example applications should be ones that require low-power operation so that the energy efficiency of neuromorphic hardware meets an important design requirement.
5) There would potentially be different benchmarks for different scales of the problem–edge deployment (sensory information processing) or cloud based analytics (large scale search, creativity etc).
We argue in the next section that brain-machine interfaces provide a benchmark application that meets all of the above criteria.
The aim of intra-cortical Brain Machine Interfaces (iBMIs) is to substantially improve the lives of patients afflicted by
Fig. 6. Example of a BMI experimental setup where the NHP is using his thoughts to move a wheelchair (adopted from [21] under CC-BY license). The decoder to convert brain signals to a command provides ideal opportunity for low-power, real-time neuromorphic machine learners.
spinal cord injury or debilitating neurodegenerative disorders such as tetraplegia, amyotrophic lateral sclerosis. These systems take neural activity as an input and drive effectors such as a computer cursor [22], wheelchair [21] and prosthetic [23], paralysed [24] limbs for the purposes of communication, locomotion and artificial hand control respectively. While early work focussed on non-invasive EEG based systems, invasive neural interfaces are needed for fine grained motor control as well as for advancing fundamental knowledge about the brain due to higher signal quality obtainable. Figure 6 demonstrates a typical experimental setup involving a non-human primate (NHP) where an implanted micro-electrode array is interfaced with amplifiers to readout neural activity at the level of single cells [21]. This neural data is collected while the primate is doing different types of tasks according to a given cue (typically visual). Based on the recorded data, a machine learner or decoder is trained to convert the neural recording to an action that affects the physical world and provides feedback to the NHP (again typically visual feedback is used). We argue that a decoder in iBMI satisfies all the conditions required for a neuromorphic system described in Section IV as explained below:
1) Neural data recorded from the brain are indeed a streaming signal arriving continuously over time. Further, the data are naturally in the form of spikes avoiding the question for the need of spike conversion and how to do it.
2) Due to the visual feedback provided to the NHP, decoding has to be done in real-time. In this case, typical update frequencies of 10 Hz are used [25].
3) There is a need to frequently adapt the weights of the decoder since the neural data is non-stationary [26]. The statistics can change due to micro-motion of the electrode or scar tissue formation.
4) The decoder must consume very little energy to prolong the battery life of the system [27]. If included within the implant, its area must be very small as well.
There has been some initial work on neuromorphic decoders [25], [28]–[31]. While [28], [29] performed software simulations, [25], [30], [31] have shown results from custom low-power neuromorphic ICs. Further, closed-loop decoding results from NHP have so far been demonstrated only in [25]. One of the issues behind lack of results in this domain is the difficulty and cost of creating a NHP based experiment. Opensource datasets are just beginning to be available in this field [32], [33]. While these definitely will provide a good starting point, they cannot be used to simulate closed-loop settings. We envision that setting up AI based models to mimic closed-loop BMI experimental settings could be a good research direction for this area.
In this paper, we reviewed the recent trend in papers published on the topic of neuromorphic engineering or computing and showed that the connotation of the term has broadened beyond its original definition of brain-inspired analog computing. Neuromorphic v2.0, as we call it in this paper, includes the concept of non von-Neumann and low precision digital computing from computer architecture and spiking neural networks from the computer science and algorithm community. However, there are differences in the way different scientific communities have used the term and a potential better taxonomy is to consider non von-Neumann computing as an umbrella under which a sub-concept is neuromorphic computing. Trends in recently published ML accelerator ICs indeed show that using the above neuromorphic concepts lead to benefit in energy efficiency, area efficiency and throughput over digital non von-Neumann architectures. We also pointed out the need for benchmarks in SNN research and suggested some potential characteristics of such benchmarks. Finally, we pointed out that brain-machine interfaces (BMI) have all these desired characteristics of real-time response, processing time varying signals, need for quick re-training as well as strict requirement for low-power dissipation. We envision generation of BMI based benchmarks in the future for testing and standardization of different neuromorphic systems.
[1] A. Basu, J. Acharya, and et. al., “Low-power, adaptive neuromorphic systems: Recent progress and future directions,” IEEE Journal of Emerging Topics in Circuits and Systems, vol. 8, no. 1, pp. 6–27, 2018.
[2] C.-Y. Chen, B. Murmann, J.-S. Seo, and H.-J. Yoo, “Custom sub-systems and circuits for deep learning: Guest editorial overview,” IEEE Journal of Emerging Topics in Circuits and Systems, vol. 9, no. 2, pp. 247–252, 2019.
[3] C. Mead, “Neuromorphic electronic systems,” Proc. of IEEE, vol. 78, no. 10, pp. 1629–36, 1990.
[4] G. Indiveri and S. C. Liu, “Memory and information processing in neuromorphic systems,” Proc. of IEEE, vol. 103, no. 8, pp. 1379–97, 2015.
[5] S. K. Bose, J. Acharya, and A. Basu, “Survey of neuromorphic and machine learning accelerators in SOVC, ISSCC and Nature/Science series of journals from 2017 onwards,” https://sites.google.com/view/ arindam-basu/neuromorphic-survey-asilomar, 2019.
[6] B. Murmann, “ADC Performance Survey 1997-2019,” http://web. stanford.edu/murmann/adcsurvey.html, 2019.
[7] D. Bankman, L. Yang, B. Moons, M. Verhelst, and B. Murmann, “An always-on CIFAR-10 mixed-signal binary CNN processor with all memory on chip in 28nm CMOS,” in 2018 IEEE International Solid - State Circuits Conference - (ISSCC), Feb 2018, pp. 222–224.
[8] H. Valavi, P. J. Ramadge, E. Nestler, and N. Verma, “A 64-Tile 2.4-Mb In-Memory-Computing CNN Accelerator Employing Charge-Domain Compute,” IEEE Journal of Solid-State Circuits, vol. 54, no. 6, pp. 1789–1799, June 2019.
[9] S. Yin, P. Ouyang, J. Yang, T. Lu, X. Li, L. Liu, and S. Wei, “An Ultra-High Energy-Efficient Reconfigurable Processor for Deep Neural Networks with Binary/Ternary Weights in 28NM CMOS,” in 2018 IEEE Symposium on VLSI Circuits, June 2018, pp. 37–38.
[10] B. Marr and et. al., “Scaling energy per operation via an asynchronous pipeline,” IEEE Trans. on VLSI, vol. 99, pp. 1–5, 2012.
[11] J. Hasler and B. Marr, “Finding a roadmap to achieve large neuromor- phic hardware systems,” Frontiers in Neuroscience, vol. 7, no. 118, pp. 1–29, 2013.
[12] Y. Lecun, C. Cortes, and C. J. C. Burges, “THE MNIST DATABASE of handwritten digits,” http://yann.lecun.com/exdb/mnist/, 1998.
[13] A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Im- ages,” https://www.cs.toronto.edu/kriz/cifar.html, 2009.
[14] O. Russakovsky, J. Deng, and et. al., “Imagenet large scale visual recognition challenge,” Intl. Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015.
[15] M. Pfeiffer, R. B. Benosman, and J. Tapson, “Benchmarks and Challenges for Neuromorphic Engineering,” https://www.frontiersin.org/research-topics/3448/ benchmarks-and-challenges-for-neuromorphic-engineering#articles, 2016.
[16] A. Sengupta and et. al., “Going deeper in spiking neural networks: Vgg and residual architectures,” Frontiers in Neuroscience, vol. 13, no. 95, pp. 1–10, 2019.
[17] B. Rueckauer and et. al., “Conversion of continuous-valued deep networks to efficient event-driven networks for image classification,” Frontiers in Neuroscience, vol. 11, no. 682, pp. 1–12, 2017.
[18] J. Acharya, A. U. Caycedo, and et.al., “EBBIOT: A Low-complexity Tracking Algorithm for Surveillance in IoVT Using Stationary Neuromorphic Vision Sensors,” in IEEE System on Chip Conference (SOCC), 2019.
[19] J. Acharya, V. Padala, and A. Basu, “Spiking Neural Network Based Region Proposal Networks for Neuromorphic Vision Sensors,” in Intl. Symp. on Circuits and Systems (ISCAS), 2019.
[20] J. Acharya and et. al., “A comparison of low-complexity real-time feature extraction for neuromorphic speech recognition,” Frontiers in Neuroscience, vol. 12, no. 160, 2018.
[21] “Independent Mobility Achieved through a Wireless Brain-Machine Interface,” PLoS ONE, vol. 11, no. 11, pp. 1–13, 2016.
[22] C. Pandarinath, P. Nuyujukian, C. H. Blabe, B. L. Sorice, J. Saab, F. R. Willett, L. R. Hochberg, K. V. Shenoy, and J. M. Henderson, “High performance communication by people with paralysis using an intracortical brain-computer interface,” eLife, vol. 6, p. e18554, 2017 2017.
[23] J. L. Collinger, B. Wodlinger, J. E. Downey, W. Wang, E. C. Tyler- Kabara, D. J. Weber, A. J. C. McMorland, M. Velliste, M. L. Boninger, and A. B. Schwartz, “High-performance neuroprosthetic control by an individual with tetraplegia.” The Lancet, vol. 381, no. 9866, pp. 557–64, 2013 2013.
[24] A. B. Ajiboye, F. R. Willett, D. R. Young, W. D. Memberg, B. A. Murphy, J. P. Miller, B. L. Walter, J. A. Sweet, H. A. Hoyen, M. W. Keith, P. H. Peckham, J. D. Simeral, J. P. Donoghue, L. R. Hochberg, and R. F. Kirsch, “Restoration of reaching and grasping movements through brain-controlled muscle stimulation in a person with tetraplegia: a proof-of-concept demonstration,” The Lancet, vol. 389, no. 10081, pp. 1821–1830, 2017 2017.
[25] S. Shaikh, R. So, and et. al., “Real-time closed loop neural decoding on a neuromorphic chip,” in 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER), 2019, pp. 670–3.
[26] S. Shaikh, Y. Chen, R. So, and A. Basu., “Cortical Motor Intention Decoding on an Analog Co-Processor with Fast Training for NonStationary Data,” in IEEE Biomedical Circuits and Systems conference (BioCAS), 2017.
[27] S. Shaikh, R. So, and et. al, “Towards intelligent intra-cortical bmi (i2bmi): Low-power neuromorphic decoders that outperform kalman filters,” IEEE Trans. on Biomedical Circuits and Systems (Early Access), 2019.
[28] B. Rapoport, L. Turicchia, and et. al, “Efficient universal computing architectures for decoding neural activity,” PLOS One, vol. 7, no. 9, pp. 1–13, 2012.
[29] J. Dethier and et. al, “Design and validation of a real-time spiking- neural-network decoder for brain machine interfaces,” Journal of Neural Engineering, vol. 10, no. 036008, pp. 1–12, 2012.
[30] F. Boi and et. al, “A bidirectional brain-machine interface featuring a neuromorphic hardware decoder,” Frontiers in Neuroscience, vol. 10, no. 563, pp. 1–15, 2016.
[31] Y. Chen, Y. Enyi, and A. Basu, “A 128 channel extreme learning machine based neural decoder for brain machine interfaces,” IEEE Trans. on Biomedical Circuits and Systems, vol. 10, no. 3, pp. 679–692, 2016.
[32] J. I. Glaser, M. G. Perich, P. Ramkumar, L. E. Miller, and K. P. Kording, “Population coding of conditional probability distributions in dorsal premotor cortex,” in Nature Communications, 2018.
[33] J. E. O’Doherty, M. M. B. Cardoso, J. G. Makin, and P. N. Sabes, “Nonhuman primate reaching with multichannel sensorimotor cortex electrophysiology,” 2017. [Online]. Available: https://zenodo.org/record/ 583331