Project Description

A Cognitive Assistive System for Coaching the Use of Home Medical Devices

This project is developing principles, methods and systems to assist people with the procedures required to operate their home medical devices without errors. The goal is for the systems to be trained on sample recordings of the person operating the device correctly, and be able to detect deviations from the correct operation sequence in the currently performed procedure, and provide corrective feedback to the user, including segments of the video portions that show the appropriate steps.

The project consists of four components:

The project aims to understand the common types of steps required in the operations of home medical devices, map how the critical indicators of these steps can be detected through appropriate sensors, train a system to recognize these steps in the context of a specific operator, establish a range of required training repetitions for different operational step types and corresponding sensors, and provide a set of suitable interventions to the end user when errors occur. The target population benefiting from this work are elderly people living at home, but requiring support from home medical devices such as glucose monitors or dialysis machines. These medical devices can allow a patient to live independently with minimal assistance, as long as the home medical devices provide the required health support.

The intellectual merit is in validating the feasibility of automatically detecting errors in the operation of medical devices. The work provides a new paradigm for learning by observation that does not require complete understanding of detailed activities in arbitrary visual and sensor sequences, but merely aligns a given new sequence in known context with previously established training data to detect significant deviations. The research will create the beginnings of a taxonomy of typical operational steps from an observational perspective for a set of devices, and establish the most effective sensors or sensor combinations to detect the successful completion of each type of step. In addition, the proposed work will help find suitable passages in the video portion of the training observations to use as corrective feedback, together with other interactive dialog interventions that may be appropriate for the device coach.

The broader impact of this work is not only that it will reduce errors in the operation of medical devices, but it will allow people to live independently at home for a longer period than before, thus saving the health care system billions of dollars resulting from earlier admissions to nursing homes or morbidity from incorrect operation of in-home medical devices. Ultimately this paradigm can also extend to the repair and maintenance of numerous other devices by non-specialized personnel.


Work shown on this web site is supported in part by NSF Cooperative Agreement No. 0812465

Contact Alex Hauptmann for more information.