Svoboda | Graniru | BBC Russia | Golosameriki | Facebook

research-article

Classification of Alzheimer's using Deep-learning Methods on Webcam-based Gaze Data

Authors:

Anuj Harisinghani,

Harshinee Sriram,

Cristina Conati,

Giuseppe Carenini,

Gabriel MurrayAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 7, Issue ETRA

Article No.: 157, Pages 1 - 17

https://doi.org/10.1145/3591126

Published: 18 May 2023 Publication History

Abstract

There has been increasing interest in non-invasive predictors of Alzheimer's disease (AD) as an initial screen for this condition. Previously, successful attempts leveraged eye-tracking and language data generated during picture narration and reading tasks. These results were obtained with high-end, expensive eye-trackers. Instead, we explore classification using eye-tracking data collected with a webcam, where our classifiers are built using a deep-learning approach. Our results show that the webcam gaze classifier is not as good as the classifier based on high-end eye-tracking data. However, the webcam-based classifier still beats the majority-class baseline classifier in terms of AU-ROC, indicating that predictive signals can be extracted from webcam gaze tracking. Hence, although our results indicate that there is still a long way to go before webcam gaze tracking can reach practical relevance, they still provide an encouraging proof of concept that this technology should be further explored as an affordable alternative to high-end eye-trackers for the detection of AD.

Supplemental Material

MP4 File

Long paper presentation video

Download
201.43 MB

References

[1]

Baltrusaitis, Tadas, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018. "OpenFace 2.0: Facial Behavior Analysis Toolkit." Pp. 59--66 in 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[2]

Barnes, Deborah E., and Kristine Yaffe. 2011. "The Projected Effect of Risk Factor Reduction on Alzheimer's Disease Prevalence." The Lancet. Neurology 10(9):819--28.

[3]

Barral, Oswald, Hyeju Jang, Sally Newton-Mason, Sheetal Shajan, Thomas Soroski, Giuseppe Carenini, Cristina Conati, and Thalia Field. 2020. "Non-Invasive Classification of Alzheimer's Disease Using Eye Tracking and Language." Pp. 813--41 in Proceedings of the 5th Machine Learning for Healthcare Conference. PMLR.

[4]

Biondi, Juan, Gerardo Fernandez, Silvia Castro, and Osvaldo Agamennoni. 2018. "Eye-Movement Behavior Identification for AD Diagnosis."

[5]

Bixler, Robert, and Sidney D'Mello. 2016. "Automatic Gaze-Based User-Independent Detection of Mind Wandering during Computerized Reading." User Modeling and User-Adapted Interaction 26(1):33--68.

Digital Library

[6]

Boote, Bikram, Mansi Agarwal, and Jack Mostow. 2021. "Early Prediction of Children's Disengagement in a Tablet Tutor Using Visual Features." Pp. 98--103 in Artificial Intelligence in Education, Lecture Notes in Computer Science, edited by I. Roll, D. McNamara, S. Sosnovsky, R. Luckin, and V. Dimitrova. Cham: Springer International Publishing.

[7]

Cho, Kyunghyun, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. "On the Properties of Neural Machine Translation: Encoder-Decoder Approaches."

[8]

Chung, Junyoung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. "Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling."

[9]

Cordell, Cyndy B., Soo Borson, Malaz Boustani, Joshua Chodosh, David Reuben, Joe Verghese, William Thies, and Leslie B. Fried. 2013. "Alzheimer's Association Recommendations for Operationalizing the Detection of Cognitive Impairment during the Medicare Annual Wellness Visit in a Primary Care Setting." Alzheimer's & Dementia 9(2):141--50.

[10]

Cummings, Louise. 2019. "Describing the Cookie Theft Picture: Sources of Breakdown in Alzheimer's Dementia." Pragmatics and Society 10(2):153--76.

[11]

D'Mello, Sidney K., Caitlin Mills, Robert Bixler, and Nigel Bosch. 2017. Zone out No More: Mitigating Mind Wandering during Computerized Reading. International Educational Data Mining Society.

[12]

Fernández, Gerardo, Jochen Laubrock, Pablo Mandolesi, Oscar Colombo, and Osvaldo Agamennoni. 2014. "Registering Eye Movements during Reading in Alzheimer's Disease: Difficulties in Predicting Upcoming Words." Journal of Clinical and Experimental Neuropsychology 36(3):302--16.

[13]

Fraser, Kathleen C., Jed A. Meltzer, and Frank Rudzicz. 2016. "Linguistic Features Identify Alzheimer's Disease in Narrative Speech." Journal of Alzheimer's Disease 49(2):407--22.

[14]

Futoma, Joseph, Sanjay Hariharan, Katherine Heller, Mark Sendak, Nathan Brajer, Meredith Clement, Armando Bedoya, and Cara O'Brien. 2017. "An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection." Pp. 243--54 in Proceedings of the 2nd Machine Learning for Healthcare Conference. PMLR.

[15]

Garbutt, Siobhan, Alisa Matlin, Joanna Hellmuth, Ana K. Schenk, Julene K. Johnson, Howard Rosen, David Dean, Joel Kramer, John Neuhaus, Bruce L. Miller, Stephen G. Lisberger, and Adam L. Boxer. 2008. "Oculomotor Function in Frontotemporal Lobar Degeneration, Related Disorders and Alzheimer's Disease." Brain 131(5):1268--81.

[16]

Goodglass, Harold, and Edith Kaplan. 1972. The Assessment of Aphasia and Related Disorders. Lea & Febiger.

[17]

Grnarova, Paulina, Florian Schmidt, Stephanie L. Hyland, and Carsten Eickhoff. 2016. "Neural Document Embeddings for Intensive Care Patient Mortality Prediction."

[18]

Hutt, Stephen, and Sidney K. D'Mello. 2022. "Evaluating Calibration-Free Webcam-Based Eye Tracking for Gaze-Based User Modeling." Pp. 224--35 in Proceedings of the 2022 International Conference on Multimodal Interaction, ICMI '22. New York, NY, USA: Association for Computing Machinery.

[19]

Jang, Hyeju, Thomas Soroski, Matteo Rizzo, Oswald Barral, Anuj Harisinghani, Sally Newton-Mason, Saffrin Granby, Thiago Monnerat Stutz da Cunha Vasco, Caitlin Lewis, and Pavan Tutt. 2021. "Classification of Alzheimer's Disease Leveraging Multi-Task Machine Learning Analysis of Speech and Eye-Movement Data." Frontiers in Human Neuroscience 512.

[20]

Kar, Anuradha, and Peter Corcoran. 2017. "A Review and Analysis of Eye-Gaze Estimation Systems, Algorithms and Performance Evaluation Methods in Consumer Platforms." IEEE Access 5:16495--519.

[21]

Karlekar, Sweta, Tong Niu, and Mohit Bansal. 2018. "Detecting Linguistic Characteristics of Alzheimer's Dementia by Interpreting Neural Models."

[22]

Kong, Weirui, Hyeju Jang, Giuseppe Carenini, and Thalia Field. 2019. "A Neural Model for Predicting Dementia from Language." Pp. 270--86 in Proceedings of the 4th Machine Learning for Healthcare Conference. PMLR.

[23]

Krafka, Kyle, Aditya Khosla, Petr Kellnhofer, Harini Kannan, Suchendra Bhandarkar, Wojciech Matusik, and Antonio Torralba. 2016. "Eye Tracking for Everyone." Pp. 2176--84 in.

[24]

Lallé, Sébastien, Cristina Conati, and Giuseppe Carenini. 2016. "Predicting Confusion in Information Visualization from Eye Tracking and Interaction Data." Pp. 2529--35 in IJCAI.

[25]

Li, Shuai, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. 2018. "Independently Recurrent Neural Network (Indrnn): Building a Longer and Deeper Rnn." Pp. 5457--66 in Proceedings of the IEEE conference on computer vision and pattern recognition.

[26]

Lipton, Zachary C., David C. Kale, Charles Elkan, and Randall Wetzel. 2015. "Learning to Diagnose with LSTM Recurrent Neural Networks."

[27]

MacAskill, Michael R., and Tim J. Anderson. 2016. "Eye Movements in Neurodegenerative Diseases." Current Opinion in Neurology 29(1):61--68.

[28]

Molitor, Robert J., Philip C. Ko, and Brandon A. Ally. 2015. "Eye Movements in Alzheimer's Disease." Journal of Alzheimer's Disease": JAD 44(1):1--12.

[29]

Müller, Philipp, Michael Xuelin Huang, Xucong Zhang, and Andreas Bulling. 2018. "Robust Eye Contact Detection in Natural Multi-Person Interactions Using Gaze and Speaking Behaviour." Pp. 1--10 in Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, ETRA '18. New York, NY, USA: Association for Computing Machinery.

[30]

Nasreddine, Ziad S., Natalie A. Phillips, Valérie Bédirian, Simon Charbonneau, Victor Whitehead, Isabelle Collin, Jeffrey L. Cummings, and Howard Chertkow. 2005. "The Montreal Cognitive Assessment, MoCA: A Brief Screening Tool For Mild Cognitive Impairment." Journal of the American Geriatrics Society 53(4):695--99.

[31]

Pavisic, Ivanna M., Nicholas C. Firth, Samuel Parsons, David Martinez Rego, Timothy J. Shakespeare, Keir X. X. Yong, Catherine F. Slattery, Ross W. Paterson, Alexander J. M. Foulkes, Kirsty Macpherson, Amelia M. Carton, Daniel C. Alexander, John Shawe-Taylor, Nick C. Fox, Jonathan M. Schott, Sebastian J. Crutch, and Silvia Primativo. 2017. "Eyetracking Metrics in Young Onset Alzheimer's Disease: A Window into Cognitive Visual Functions." Frontiers in Neurology 8.

[32]

Prince, Martin, Dr Renata Bryce, and Dr Cleusa Ferri. n.d. "World Alzheimer Report 2011: The Benefits of Early Diagnosis and Intervention."

[33]

Pusiol, Guido, Andre Esteva, Scott S. Hall, Michael Frank, Arnold Milstein, and Li Fei-Fei. 2016. "Vision-Based Classification of Developmental Disorders Using Eye-Movements." Pp. 317--25 in Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2016, Lecture Notes in Computer Science, edited by S. Ourselin, L. Joskowicz, M. R. Sabuncu, G. Unal, and W. Wells. Cham: Springer International Publishing.

[34]

Sims, Shane D., and Cristina Conati. 2020. "A Neural Architecture for Detecting User Confusion in Eye-Tracking Data." Pp. 15--23 in Proceedings of the 2020 International Conference on Multimodal Interaction, ICMI '20. New York, NY, USA: Association for Computing Machinery.

[35]

Tay, Yi, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2022. "Efficient Transformers: A Survey." ACM Computing Surveys.

Digital Library

[36]

Tran, Minh, Taylan Sen, Kurtis Haut, Mohammad Rafayet Ali, and Ehsan Hoque. 2022. "Are You Really Looking at Me" A Feature-Extraction Framework for Estimating Interpersonal Eye Gaze From Conventional Video." IEEE Transactions on Affective Computing 13(2):912--25.

[37]

Trauzettel-Klosinski, Susanne, Klaus Dietz, and the IReST Study Group. 2012. "Standardized Assessment of Reading Performance: The New International Reading Speed Texts IReST." Investigative Ophthalmology & Visual Science 53(9):5452--61.

[38]

Tripathi, Subarna, Zachary C. Lipton, Serge Belongie, and Truong Nguyen. 2016. "Context Matters: Refining Object Detection in Video with Recurrent Neural Networks."

[39]

Valliappan, Nachiappan, Na Dai, Ethan Steinberg, Junfeng He, Kantwon Rogers, Venky Ramachandran, Pingmei Xu, Mina Shojaeizadeh, Li Guo, Kai Kohlhoff, and Vidhya Navalpakkam. 2020. "Accelerating Eye Movement Research via Accurate and Affordable Smartphone Eye Tracking." Nature Communications 11(1):4553.

[40]

Yue-Hei Ng, Joe, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. 2015. "Beyond Short Snippets: Deep Networks for Video Classification." Pp. 4694--4702 in.

[41]

Zhang, Xucong, Yusuke Sugano, and Andreas Bulling. 2017. "Everyday Eye Contact Detection Using Unsupervised Gaze Target Discovery." Pp. 193--203 in Proceedings of the 30th annual ACM symposium on user interface software and technology.

Cited By

Chudzik AŚledzianowski APrzybyszewski A(2024)Machine Learning and Digital Biomarkers Can Detect Early Stages of Neurodegenerative DiseasesSensors10.3390/s2405157224:5(1572)Online publication date: 29-Feb-2024
https://doi.org/10.3390/s24051572
Nicora GCatalano MBortolotto CAchilli MMessana GLo Tito AConsonni ACutti SComotto FStella GCorsico APerlini SBellazzi RBruno RPreda L(2024)Bayesian Networks in the Management of Hospital Admissions: A Comparison between Explainable AI and Black Box AI during the PandemicJournal of Imaging10.3390/jimaging1005011710:5(117)Online publication date: 10-May-2024
https://doi.org/10.3390/jimaging10050117
El Hmimdi APalpanas TKapoula Z(2024)ORASIS-MAE Harnesses the Potential of Self-Learning from Partially Annotated Clinical Eye Movement RecordsBioMedInformatics10.3390/biomedinformatics40301054:3(1902-1933)Online publication date: 26-Aug-2024
https://doi.org/10.3390/biomedinformatics4030105
Show More Cited By

Index Terms

Classification of Alzheimer's using Deep-learning Methods on Webcam-based Gaze Data
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. User studies

Recommendations

Real-time eye gaze tracking with an unmodified commodity webcam employing a neural network
CHI EA '10: CHI '10 Extended Abstracts on Human Factors in Computing Systems

An eye-gaze-guided computer interface could enable computer use by the seriously disabled but existing systems cost tens of thousands of dollars or have cumbersome setups. This paper presents a methodology for real-time eye gaze tracking using a ...
Evaluating Calibration-free Webcam-based Eye Tracking for Gaze-based User Modeling
ICMI '22: Proceedings of the 2022 International Conference on Multimodal Interaction

Eye tracking has been a research tool for decades, providing insights into interactions, usability, and, more recently, gaze-enabled interfaces. Recent work has utilized consumer-grade and webcam-based eye tracking, but is limited by the need to ...
Scalable Webcam Eye Tracking by Learning from User Interactions
CHI EA '15: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems

Eye tracking systems are commonly used in a variety of research domains, but cost thousands of dollars. In my thesis I investigate a new approach to enable eye tracking for common webcams. The aim is to provide a natural experience to everyday users ...

Comments

Information & Contributors

Information

Published In

Proceedings of the ACM on Human-Computer Interaction Volume 7, Issue ETRA

ETRA

May 2023

234 pages

EISSN:2573-0142

DOI:10.1145/3597645

Editor:
Jeff Nichols
Apple Inc., United States

Issue’s Table of Contents

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 May 2023

Published in PACMHCI Volume 7, Issue ETRA

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Data Availability

Long paper presentation video https://dl.acm.org/doi/10.1145/3591126#1061_Sriram_Presentation (2).mp4

Funding Sources

Health Research BC
The Canadian Consortium of Neurodegeneration and Aging
The Alzheimer's Society of Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
375
Total Downloads

Downloads (Last 12 months)226
Downloads (Last 6 weeks)26

Reflects downloads up to 29 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chudzik AŚledzianowski APrzybyszewski A(2024)Machine Learning and Digital Biomarkers Can Detect Early Stages of Neurodegenerative DiseasesSensors10.3390/s2405157224:5(1572)Online publication date: 29-Feb-2024
https://doi.org/10.3390/s24051572
Nicora GCatalano MBortolotto CAchilli MMessana GLo Tito AConsonni ACutti SComotto FStella GCorsico APerlini SBellazzi RBruno RPreda L(2024)Bayesian Networks in the Management of Hospital Admissions: A Comparison between Explainable AI and Black Box AI during the PandemicJournal of Imaging10.3390/jimaging1005011710:5(117)Online publication date: 10-May-2024
https://doi.org/10.3390/jimaging10050117
El Hmimdi APalpanas TKapoula Z(2024)ORASIS-MAE Harnesses the Potential of Self-Learning from Partially Annotated Clinical Eye Movement RecordsBioMedInformatics10.3390/biomedinformatics40301054:3(1902-1933)Online publication date: 26-Aug-2024
https://doi.org/10.3390/biomedinformatics4030105
El Hmimdi AKapoula ZSainte Fare Garnot V(2024)Deep Learning-Based Detection of Learning Disorders on a Large Scale Dataset of Eye Movement RecordsBioMedInformatics10.3390/biomedinformatics40100294:1(519-541)Online publication date: 14-Feb-2024
https://doi.org/10.3390/biomedinformatics4010029
Wagner TColley MBreckel DKösel MRukzio E(2024)UnitEye: Introducing a User-Friendly Plugin to Democratize Eye Tracking Technology in Unity EnvironmentsProceedings of Mensch und Computer 202410.1145/3670653.3670655(1-10)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1145/3670653.3670655
Kennel KRuzika S(2024)Real-time Prediction of Students' Math Difficulties using Raw Data from Eye Tracking and Neural NetworksProceedings of the ACM on Human-Computer Interaction10.1145/36556078:ETRA(1-19)Online publication date: 28-May-2024
https://dl.acm.org/doi/10.1145/3655607
Liu BLye SZakaria Z(2024)An integrated framework for eye tracking-assisted task capability recognition of air traffic controllers with machine learningAdvanced Engineering Informatics10.1016/j.aei.2024.10278462(102784)Online publication date: Oct-2024
https://doi.org/10.1016/j.aei.2024.102784

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents