Virtual Reality

Abstract

Recent development in artificial neural networks has led to an increase in performance, but also in complexity and size. This poses a significant challenge for the exploration and analysis of the spatial structure and temporal behaviour of such networks. Several projects for the 3D visualisation of neural networks exist, but they focus largely on the exploration of the spatial structure alone, and are using standard 2D screens as output and mouse and keyboard as input devices. NeuVis is a framework for an intuitive and immersive 3D visualisation of spiking neural networks in virtual reality, allowing for a larger variety of input and output devices. It is a core component of NeuCube, a 3-dimensional spiking neural network learning framework, significantly improving the user’s abilities to explore, analyse, and also debug the network.

FIGURE. The NeuVis 3D framework.

Related Papers and Benchmarking

The proposed methods and systems offers the the following advantages:

  1. NeuVis is a flexible and scalable framework;
  2. It allows for a range of input and output device combinations;
  3. It integrates a large variety of analysis tools that are helpful in gaining insights to NeuCube that were not possible with the previous visualisation tool;
  4. Combined with a motion capture setup with stereoscopic rendering, NeuVis offers an improved spatial perception  that makes navigation as intuitive as moving the user’s own head and body.

See also some of the related papers:

Marks, S. (2016). Immersive visualisation of 3-dimensional spiking neural networks. Evolving Systems, 1-9. doi:10.1007/s12530-016-9170-8

Kasabov, N., Scott, N. M., Tu, E., Marks, S., Sengupta, N., Capecci, E., Othman, M., Gholoami Doborjeh, M., Murli, N., Hartono, R., Espinosa-Ramos, J. I., Zhou, L., Alvi, F., Wang, G., Taylor, D., Feigin, V., Gulyaev, S., Mahmoud, M., Hou, Z. G., Yang, J. (2016). Evolving spatio-temporal data machines based on the NeuCube neuromorphic framework: design methodology and selected applicationsNeural Networks78, 1-14.

Marks, S., Estevez, J., & Scott, N. (2015). Immersive Visualisation of 3-Dimensional Neural Network Structures. In 13th International Conference on Neuro-Computing and Evolving Intelligence (NCEI) 2015. Auckland.
Retrieved from http://www.kedri.aut.ac.nz/conferences/ncei15

Marks, S., Estevez, J. E., & Connor, A. M. (2014). Towards the Holodeck: Fully Immersive Virtual Reality Visualisation of Scientific and Engineering Data. In 29th International Conference on Image and Vision Computing New Zealand (IVCNZ) 2014 (pp. 42-47). Hamilton, New Zealand: ACM. doi:10.1145/2683405.2683424


R&D System

For this project, an R&D system has been developed based on NeuCube. The system can be obtained subject to licensing agreement.


Developer

The developers of this project are:

Dr Stefan Marks

Dr Israel Espinosa Ramos