Introduction
This project proposes methods for enhancement of MindDesktop, a general purpose Brain Computer Interface for controlling the computers running Windows. The enhancements can be mainly defined as:
- Improvement of inputs hierarchy tree
- Improvement of screen division algorithm for mouse emulation
- Reduction of number of mental commands
- Reduction of device cost
User of the BCI system can control the computer using mental commands through the GUI.
The GUI consists of a hirarchy tree of buttons and a "pointing device" for emulating mouse clicks by recurrent screen divisions and selections.
Training is required to match users' unique EEG signal pattern to the corresponding mental commands. Currently, Deep Learning Neural Networking
is being used for testing the feasibility.
This project introduce a structure of the hierarchy tree that is more user-friendly. The GUI will go back to appropriate layer of the hierarchy tree so as to reduce excessive selections of
buttons to finsih a same particular task.
The MindDesktop equally divided the screen into 4 sub-areas recurrently for point selection and mouse click. This project proposed an algorithm to
divide the screen into four uneqaul areas that contains similar number of GUI elements on the Windows Desktop. In this way, less selections will be
required to reach a point that has sparse GUI elements in the surrounding, while keeping a similar selection speed for a point that has dense GUI
elements in the surrrounding.
Three mental commands(Next, Confirm, Back) have been using by MindDesktop for control, while our BCI use just two commands(Next, Confirm) for full
control, by converting the Back command to an option in the inputs hierarchy tree. So that a confirm command on the "Back" input will have a same
function as a back command in MindDesktop. Through reduction of number of command, mental command training time can be greatly reduced and lowers the
difficulty fo BCI control.
This project also focuses on the cost reduction. A lower end BCI device is used to test the feasibility of controlling the BCI with a cheaper brainwave
monitoring device. At the beginning, Emotiv Insight is adopted but it was mulfunctioned at late middel of the project. OpenBCI Ganglion, costing $199USD
and having 4 EEG channels, is the replacement of the Emotiv headset and currently used in the project. It is relatively much cheaper than the 14-channles
Emotiv EPOC being used by the MindDesktop, which costs $799USD. Though, the performance of OpenBCI Ganglion is comparatively poor due to less number of
EEG channels. As a result, a command prediction algorithm is proposed to increase the accuracy of mental command control.
The architecture of the BCI, screen division algorithm, mental command prediciton algorithm and input hierarchy tree will be further discussed in the
subsequent sessions.
Architecture
A modular approach is used in the design of the architecture in this system rather than packing all items into a single bulky application.
By using this approach, the BCI application can be divided into numbers of smaller components, each responsible for a core function or collection
of related functions. It allows the BCI system to have a cleaner design, effective division of tasks for system development and easier management
of programming code. Most importantly, it greatly increases the flexibility of the application in the case that one of the components can be replaced
with a new one or being modified without affecting other components. As a result, modification to the application can be done with less work.
The application is separated into three main components, namely Connector Component, Main Component and Training Component.
Connector Component is a server application that connects the OpenBCI GUI application, receives the raw or processed data and broadcast the result
of prediction algorithm using the recieved data.
Main Component is responsible for GUI rendering and trigger the mouse clicks and keyboard input events on the Windows OS.
Training Component is responsible for the mental command training. Three mental commands (Neutral, Next, Confirm) are trained,Data from each channels
are trained independently. Four Neural Network models are trained simultaneously and produced four files containing the weights of the trained models.
These weights files will be used in the prediction of mental command during the control of the BCI.
Inputs Hierarchy Tree
The hierarchy tree is designed in a user-friendly manner and related inputs such as characters and punctuation marks are grouped together
for faster and easier searching.
As mentioned in the introduction part, the input hierarchy tree affects the number of selections required to finish a particular task.
For example, enter a sequence of characters. During the BCI control, user can only stay at one layer of the hierarchy at anytime and go back
and forth to select and execute the inputs. Under our design, the GUI will go back to an appropriate layer of the hierarchy when the user finish
an input. For instance, if user just enter a character, then it will move back to the topmost layer of character rather than the root to assume
that user may want to enter another character in the next action. By doing this, user are not required to go the character layer again from the root
and thus reduce the steps for entering another character.
Selection Division Algorithm
Windows UI Automation is being used for implementation of this algorithm. The algorithm tries to divided the selected area into four sub-areas which contains similar number of UI elements. The brief steps are as follows:
- Use Windows UI Automation to inspect the coordinates and size of GUI elements on Windows Desktop
- Eliminates all the elements covered by the foreground windows or those inside the minimized windows
- Scoring each visible element according to the distance and angle from top-left corner of the selected area
- Sort the elements by scores and choose the first quarter of the elements will be elements covered by subdivided area 1
- Divide the remainging area into 3 rectangular areas so that the remaining elements are distributed as even as possible
Command Prediction Algorithm
Current approach predicts the mental commands per each second. The prediction results are congregated as a single predicted commands by following steps. After, consecutive identical predicted commands are required to execute the command.
- Get the FFT features from the EEG signals in a sampling rate of 30/s
- After collecting each second of FFT features, feed into the NN model and conclude a final predicted command in these 30 predictions by voting
- Four results get from step 2 since there are 4 EEG channels with distinct signals
- Conclude the four results by voting and also compare the prediction probabilities to get a single final predicted command
- Three consecutive "Next" or "Confirm" predictions from step 4 are required to execute the command
