Signal Processing & Pattern Recognition Group

This is an R&D system that allows deep learning of video data and for accurate classification of moving objects captured in the data using eSNN.

Moving object recognition is a challenging problem in computational intelligence. Fast moving object is considered as the one which could not easily be captured by conventional cameras in real time. The typical examples encompass fast moving cars, flying rockets, bouncing ping-pong balls, tennis balls, balancing pencils etc. It is impossible to recognise such moving objects without using a suitable algorithm and effective software system which are capable to learn and recognize patterns from complex Spatio- and Spectro-Temporal Data (SSTD). Deep learning has improved machine learning in computer vision from end to end. In this paper, we propose a new methodology for deep learning of video data and for accurate classification of moving objects captured in the data using eSNN (evolving Spike Neural Network). Taking video footage encapsulating moving objects as input data, we conduct convolution operations for each video frame by using a Gaussian filter as the first step of deep learning, then adaptive down sampling is used to shrink the video frame both in width and height, after that spike encoding of these features is applied over time to identify the changes of each image block. Finally, we use the spikes of each 10 ×10 block of video frames as features and import them into NeuCube for training and testing using dynamic evolving spiking neural network as a classifier to classify movement of the objects. Compared to other deep neural networks and other machine learning techniques, our NeuCube model has outperformed in various scenes for fast moving object recognition using spatio- and spectro- temporal video data, achieving accuracy of about 90%. It can be used for both high and low-resolution videos. Moreover, it allows to be further trained on new data in an online and incremental mode.

Proposed research structure in stage 1, capture videos with moving objects and then generate spike event based frames into CSV files. These CSV files will then be used as data sets for NeuCube.

Video footage with and without motions (a) No motion was detected; (b) Spike changes based on motions and generate ‘on’ and ‘off’ events.

Video frames with moving cars (a) Original video frame with the size of 1280×720. (b) Frame from DVS simulator.

The spike event based frames are divided into 100 blocks, each block consists of 128×72 pixels, the mean filter has been applied to evaluate the pixel output.

Related papers and benchmarking

The proposed methods and systems, when compared with traditional statistical and machine learning methods, showed superior results in the following aspects:

1. Better data analysis and classification/regression accuracy (by 10 to 40%);
2. Better visualisation of the created models, with a possible use of VR;
3. Better understanding of the data and the processes that are measured;
4. Enabling new information and knowledge discovery through meaningful interpretation of the models.

See also some of the related papers:

• Kasabov, N. K. (2014). NeuCube: A spiking neural network architecture for mapping, learning and understanding of spatio-temporal brain data. Neural Networks, 52, 62-76.
• Kasabov, N., Scott, N. M., Tu, E., Marks, S., Sengupta, N., Capecci, E., Othman, M., Gholoami Doborjeh, M., Murli, N., Hartono, R., Espinosa-Ramos, J. I., Zhou, L., Alvi, F., Wang, G., Taylor, D., Feigin, V., Gulyaev, S., Mahmoud, M., Hou, Z. G., Yang, J. (2016). Evolving spatio-temporal data machines based on the NeuCube neuromorphic framework: design methodology and selected applications. Neural Networks, 78, 1-14.

R&D system

For this project, an R&D system has been developed based on NeuCube. The system can be obtained for R&D subject to licensing agreement.

Developer

• Associate Professor Wei Qi Yan
• Wei Cui

This aging model is based on evolving spiking neural network NeuCube. The utility of aging model is tested for age group classification and gender recognition.

The Brain-Like Artificial Intelligence (BLAI) is pioneered by Professor Nikola Kasabov and here it is applied to a specific application.

In this project we develop methods and systems for two inter-related problems in face recognition using the Neucube neuromorphic computational platform. The two systems developed are for: (1) age classification; (2) and gender recognition.

The well-known FG-NET and MORPH Album 2 image gallery were used and anthropometric features were extracted from landmark points on the face. The landmarks were first pre-processed with the procrustes algorithm before feature extraction was performed. The Weka machine learning workbench was used to compare the performance of traditional techniques such as the K nearest neighbor (Knn) and Multi-LayerPerceptron (MLP) with NeuCube. Our empirical results show that NeuCube performed consistently better across both problem types that we investigated

_{Schematic representation of the NeuCube-based methodology for mapping, learning, visualisation and classification.}

Related papers and benchmarking

The proposed methods and systems, when compared with traditional statistical and machine learning methods, showed superior results in the following aspects:

1. Better data analysis and classification/regression accuracy (by 10 to 40%);
2. Better visualisation of the created models, with a possible use of VR;
3. Better understanding of the data and the processes that are measured;
4. Enabling new information and knowledge discovery through meaningful interpretation of the models.

See also some of the related papers:

• Alvi, F. B., & Pears, R. (2017). A composite spatio-temporal modeling approach for age invariant face recognition. Expert Systems with Applications, 72, 383-394.
• Alvi, F. B., Pears, R., & Kasabov, N. (2017). An evolving spatio-temporal approach for gender and age group classification with Spiking Neural Networks. Evolving Systems, 1-12.

R&D system

For this project, an R&D system has been developed based on NeuCube. The system can be obtained subject to licensing agreement.

Developer

• Fahad Alvi

The proposed research is part of a decision support system based on brain-inspired, facial, social and emotional analysis approach that measures human emotions and behaviours. The automated affective analysis will lead to the development of a spiking neural network (SNN) model of cognitive affective-quotient (SNN-CAQ).

Understanding and measuring human emotions and behaviours have long been an interest of academic and commercial researchers. The proposed research methodology relies on the assumptions that there is a valid reflection of the human state of mind by automatically sensing facial and body expression which will provide a future prediction of a certain behaviour. The proposed research is a new AI, brain-inspired, facial, social and emotional analysis approach as part of a decision support system to measure human emotions and behaviours. The automated affective analysis will lead to the development of a spiking neural network (SNN) model of cognitive affective-quotient (SNN-CAQ) that will provide a better way to understand and recognise individual human behavioural propensity in the decision making. A SNN-CAQ is a temporal and multimodal pattern that evolves dynamically to measure the change of facial expression, body gesture and social cue that recognises the sentiment intensity of a human cognitive function. The significance of this research project will allow an affective-state to associate with a quotient that closely correlates to a behavioural tendency. Consumer behavioural preference towards a product and services is identified as one of the research case studies because the measurement of consumer’s perception, reaction and satisfaction continue to be an open problem in the product design, service provider and retail industries. Importantly, the automated analysis of using Affective Quotient can apply to a wide variety of domain-specific applications like analysing unsafe driving behaviour or aggressive crowd management.

Schematic process flow of input from the emotional expression data into NeuCube.

Five Expressions labels – 01- Contempt, 02- Disgust, 03 – Joy, 04 – Surprise and 05 - Sadness.

Spikes connections after training the Cube and Classifier.

Related papers and benchmarking

The proposed methods and systems, when compared with traditional statistical and machine learning methods, showed superior results in the following aspects:

1. Better data analysis and classification/regression accuracy (by 10 to 40%);
2. Better visualisation of the created models, with a possible use of VR;
3. Better understanding of the data and the processes that are measured;
4. Enabling new information and knowledge discovery through meaningful interpretation of the models.

See also some of the related papers:

• Kasabov, N. K. (2014). NeuCube: A spiking neural network architecture for mapping, learning and understanding of spatio-temporal brain data. Neural Networks, 52, 62-76.
• Kasabov, N., Scott, N. M., Tu, E., Marks, S., Sengupta, N., Capecci, E., Othman, M., Gholoami Doborjeh, M., Murli, N., Hartono, R., Espinosa-Ramos, J. I., Zhou, L., Alvi, F., Wang, G., Taylor, D., Feigin, V., Gulyaev, S., Mahmoud, M., Hou, Z. G., Yang, J. (2016). Evolving spatio-temporal data machines based on the NeuCube neuromorphic framework: design methodology and selected applications. Neural Networks, 78, 1-14.

R&D system

For this project, an R&D system has been developed based on NeuCube. The system can be obtained for R&D subject to licensing agreement.

Developer

• ClarenceTan