Evaluation of Ten Open-Source Eye-Movement Classification Algorithms in Simulated Surgical Scenarios

Abstract

Despite providing several insights into visual attention and evidence regarding certain brain states and psychological functions, classifying eye movements is a highly demanding process. Currently, there are several algorithms to classify eye movement events which use different approaches. However, to date, only a limited number of studies have assessed these algorithms under specific conditions, such as those required for surgical training programmes. This study presents an investigation of ten open-source eye-movement classification algorithms using the Eye Tribe eye-tracker. The algorithms were tested on the eye-movement records obtained from 23 surgical residents, who performed computer-based surgical simulation tasks under different hand conditions. The aim was to offer data for the improvement of surgical training programmes. According to the results, due to the different classification methods and default threshold values, the ten algorithms produced different results. Considering the fixation duration, the only common event for all of the investigated algorithms, the binocular-individual threshold (BIT) algorithm resulted in a different clustering compared to the other algorithms. Based on the other set of common events, three clusters were determined by eight algorithms (except BIT and event detection (ED)), distinguishing dispersion-based, velocity-based and modified versions of velocity-based algorithms. Accordingly, it was concluded that dispersion-based and velocity-based algorithms provided different results. Additionally, as it individually specifies the threshold values for the eye-movement data, when there is no consensus about the threshold values to be set, the BIT algorithm can be selected. Especially for such cases like simulation-based surgical skill-training, the use of individualised threshold values in the BIT algorithm can be more beneficial in classifying the raw eye data and thus evaluating the individual progress levels of trainees based on their eye movement behaviours. In conclusion, the threshold values had a critical effect on the algorithm results. Since default values may not always be suitable for the unique features of different data sets, guidelines should be developed to indicate how the threshold values are set for each algorithm.