EQUATION RECOGNITION

This is a demo of my undergraduate final year project 'Handwritten Equation Recognition. The project was done with two approaches: Using K-means clustering and using Convolutional Neural Networks. This demo shows two applications that we made for the project. First one is the data collection application and second one is the final recognition app which was made for the K-means clustering approach.

The bitbucket repository contains the android project for the final recognition app. The 'app/src/main/res/raw' folder contains the trained models. ie. files for the cluster centroids.

In 'equation-recognition-caffe/app/src/main/java/edu/vit/majorproject/equationrecognizer/' you'll find all the services, their implementations and the main activity which acts as the controller.

Bitbucket Link

Description

‘Handwritten Equation Recognition’ is a project whose end product is an android app that can recognize and calculate a very basic mathematical expression written by hand on the screen. The main objective of this project was identification of handwritten characters. This was achieved by using K-Means Clustering.

Approach:

Process:

Data Collection:

The process began with collection of data. We built an andriod app with a simple interface that would take hand written characters as input and save them in the form of CSV files. The information that we collected form the characters was x & y coordinates, timestamp, stroke duration and the interstroke duration. Although we collected the temporal data, we later decided to go just with the spatial information since we were able to get results with just the spatial data. Each stroke of a character was stored individually for that character. Data was collected for characters 0-9, +, -, * and /. We collected about 100 samples for each. 70 for training 30 for testing.

Data Preprocessing:

We passed the collected data through a smoothing function to take care of outliers. For this we implemented the Rolling Mean algorithm. Then this data was used to calculate fourier descriptors using Discrete Fourier Transform. We initially used the Fast Fourier Transform algorithm provided by Numpy while training on the computer but since we couldn’t port the library into our android app we implemented our own DFT function and used it for both training and testing. The calculation was done for each character individually. From the series that we got for each character we chose the first 6 points (fourier descriptors) after the 0th element. This number was decided by testing for various number of fourier points and calculating the accuracy. Thus each character was represented by a set of 6 complex numbers. This was done for all the samples of a character.

Training:

Once we got the fourier descriptors for all samples of a character we passed the list to the K-means clustering function from the sklearn library. This gave us the cluster centroids for each character. Again to determine how many clusters to use we ran tests for various number of clusters and checked the accuracy. So for every character we got a set of centroids. The centroids of all characters were stored in a file hierarchically called the ‘codebook’ which was later used for testing.

Testing:

For recognizing a character, the test sample went through the pre-processing till we got a fourier descriptor that represented the character. We calculated the euclidean distance between the Fourier descriptors of the test characters and the centroids of a trained character. From this the distance to the closest centroid was considered as the final distance of the test sample to a trained character. The final distances to each trained character were compared and the one with the minimum was declared as the prediction. Using this method, we calculated the accuracy for different number of clusters per character. With all the parameters decided we made an android app that would take an input expression, segment it into individual characters and evaluate it. The segmentation was done on the basis of overlapping of strokes in a character.

Note: A similar approach was tried where we calculated the descriptors for each stroke, created clusters for the character as a whole and calculated the final distance to a trained character by adding up the distances of each stroke to its closest centroid.