Research: Projects
The mhealth Research Group explores how mobile and ubiquitous sensing might be extended and then used to support health and wellness through personal health informatics. As such, some projects have more of a technical, or computer-science flavor, and other projects have more of a public health or behavioral-science flavor. Most projects, though, combine both areas of interest. This short "What if..." video summarizes some of our goals.
Listed here are our active projects, projects that are current dormant but that we would like to restart given student interest, and projects that are completed that are unlikely to be restarted (although all of our past projects are influencing what we do in the future). Nearly all of our projects involve an interdisciplinary, team-science approach.
Projects generally touch upon one or more of these overlapping areas of interest:
Listed here are our active projects, projects that are current dormant but that we would like to restart given student interest, and projects that are completed that are unlikely to be restarted (although all of our past projects are influencing what we do in the future). Nearly all of our projects involve an interdisciplinary, team-science approach.
Projects generally touch upon one or more of these overlapping areas of interest:
- Activity and context recognition using data from smartphones, smartwatches, other wearable sensors, and/or ubiquitous computing devices
- Human-in-the-loop behavior and habit recognition with limited training data
- Computational modeling of behaviors and behavioral theory, especially to support behavior change and behavior maintenance
- Extensions to context-sensitive ecological momentary assessment (a.k.a. experience sampling), such as microinteraction ecological momentary assessment (uEMA)
- Just-in-time adaptive interventions (JITAIs) that use real-time sensing
- Measuring physical activity, sedentary behavior, and/or sleep (at population scale), as well as other health-related behaviors and contexts
- Using mobile and ubiquitous computing to support active transportation (via bicycling)
- Accelerating research on the topics above through generation of shared datasets, software tools, and living laboratories
- Persuasive technology
Active or planned projects
The PAAWS Study (Physical Activity Assessment using Wearable Sensors)
|
The PAAWS Study is an NIH-funded effort run by the mHealth Group in collaboration with researchers at the Harvard Medical School/Brigham and Women's Hospital.
Project website: https://paawsstudy.org Goals Accurate measurement of human behavior using devices could significantly advance current knowledge on the dose-response relationships between chronic diseases and behaviors such as physical activity, sedentary behavior, and sleep. The primary objective of this proposal is to develop valid approaches to measure 24-hour physical behavior, as well as to demonstrate a procedure via which those approaches can be compared to others. We aim to help the research community to converge on methods that use devices to accurately measure physical activity type and intensity, sedentary behavior and posture, and sleep in adults. Many promising methods have been proposed to measure behavior from activity monitors. Unfortunately, these methods – which are now being proposed in large numbers – are typically validated on small amounts of data. Thus, they may perform well on lab data, but fail when used in the field on large-scale epidemiological or intervention studies. Moreover, the performance of different methods is rarely compared head-to-head, creating uncertainty for public health researchers about which are the best to use. Quantifying the relative performance of methods that produce similar outcome measures but use different devices or on-body device locations is even more unusual. We will make it easy for researchers interested in physical activity measurement to meaningfully compare performance between new methods and confidently apply those methods to both large-scale surveillance studies and longitudinal interventions. The project has four specific aims:
Collaborator(s): Harvard School of Medicine/Brigham and Women's Hospital |
Data are collected from wearable sensors worn by 18+ adults for a full seven days and being provided to the research community to advance algorithm development.
|
Development of Algorithms for Detecting the Activities of Adults and Children from Wearable Sensors
|
We are working on a variety of projects studying how to use mobile sensor data, especially from accelerometers, to detect physical activity (type, duration, and intensity), sedentary behavior, and sleep in adults and children. We are interested in extending this work to detect habits, and ultimately to develop a comprehensive probabilistic model of someone's behaviors.
Collaborator(s): EveryFit, Inc., Stanford Medical School, Case Western Reserve University |
Accelerometer data from the wrist and ankle, and manually-labeled activities. We are developing algorithms to detect such everyday behaviors.
|
Developing a Computational Pathway and Algorithms to Summarize Large-Scale Accelerometer Data
|
We are developing easy-to-use methods and algorithms for summarizing large amounts of accelerometer data collected in important health surveillance studies such as the NHANES and the UK Biobank. Our goal is to advance research by creating non-proprietary and open-source tools and methods that allow large-scale data data-driven discovery. Our group processed the 18k+ dataset of raw accelerometer data from NHANES, computing motion summaries and sleep, wear, and non-wear.
Collaborator(s): Dinesh John (Northeastern University) |
We are creating a simple way for researchers to run algorithms on 18,000+ raw accelerometer datasets from the NHANES study.
|
Developing End-User-Driven Interactive and Real-Time Activity Recognition Algorithms
|
This is an ongoing effort to develop practical activity recognition systems for smartwatches and smartphones using a combination of passive sensing of data (accelerometer, location) and human-computer interaction with the end user. The scientific question is how to create algorithms that integrate human-in-the-loop activity training into machine learning algorithms, using common sense and explanatory systems. Our goal is to create practical systems that can be used for real-time measurement of behavior to support novel just-in-time mobile health systems.
|
We are currently testing a user interface to support zero-shot learning of unseen physical activities by using end-user-derived semantic attributes. These may allow recognizers to be built based representation of activities for which there is no sensor training data.
|
Temporal influences on Movement and Exercise (TIME Study)
|
The TIME study uses mobile and wearable technologies to measure moment-to-moment experiences that shape our health behaviors and decision making. The goal is to study how different contextual and situational factors explain the adoption and maintenance of health behaviors. We are using a combination of phone-based experience sampling, wearable-based micro-EMA, and passive sensing to build predictive models of health behavior at the individual and group levels. We are conducting a prospective within-subject case-crossover observational study across a 12-month period among ethnically-diverse, emerging adults (ages 18-29).
Collaborators: Genevieve Dunton and the REACH Lab team (USC) |
|
Signaligner Pro: Supporting Multi-day Raw Sensor Data Annotation
|
We have repurposed our prior work on human computation games to crowdsource raw sensor data annotations into an interactive tool for experts. We developed this tool to enable activity recognition, physical activity, and sleep behavior researchers interested in using raw sensor data to explore and annotate high-sampling rate raw data collected on individuals for a week of data or more. We have found such a tool to be vital in our own work, and so we have made it available to others and continue to improve it. Currently, the tool is being used to label data for prolonged physical activities and everyday activities, including sleep, but also short lived activities such as hand washing and face touching.
The tool is available on GitHub: https://crowdgames.github.io/signaligner/ Collaborators: Seth Cooper (Northeastern) |
A tool for visualization and annotation of high-sampling-rate sensor data.
|
Using Temporally-Dense Microinteractions to Measure and Promote Behavior Change
|
The purpose of this project is to study how “microinteractions” – quick interactions with smartwatches, or even devices like Google Glass – can be used to create new ways to continuously measure health behavior and provide opportunities for just-in-time health intervention delivery to support health behavior change and behavior maintenance.
Collaborator(s): Justin Manjourides (Northeastern) |
In micro-EMA on a smartwatch, only single questions with a simple answer are asked, but at a surprisingly high and yet sustainable temporal density.
|
Changing Commuting Culture: Virtual Bicycle Highways
|
This project explores how mobile technology can be used to support active commuting via bicycling, where individuals who would normally be commuting by car may instead commute via bicycle. They may do this because of a novel technological system that would create a car-free riding experience, dramatically enhancing comfort and safety. We are looking for students interested in bicycling who want to help with this project, which is in the early stages. We are building a state-of-the-art bicycle simulator, to help support future research in this area.
|
Concept image showing one aspect of the Virtual Bicycle Highways concept.
|
SCC: Smart and Connected Churches for Promoting Health in Disadvantaged Populations
|
The Smart and Connected Communities (SCC) project aims to leverage the power of social networks that exist within churches to promote healthy behaviour change. As a step toward this goal, we are developing a prototype system that will enable activity-initiated messaging via mobile phones. The activity-initiated messaging (AIM) app is a messaging application developed for Android that enhances the capabilities of existing messaging applications by allowing users to choose the right moment to automatically deliver an activity-dependent message. Messages can either be exchanged in real time or based on the occurrence of a specific event. Sensors available in Android phones such as the accelerometer and location sensors are used to detect a user’s context and decide when a message should be delivered. This application will eventually be integrated with a virtual agent system and will enable our research team to explore how end-user participation/contribution can enhance behavior or habit identification for an individual (i.e., via human-in-the-loop activity detection). The aim is to allow models to adapt to individual participants, and then to support health behavior change using activities/habits that can be detected for given individuals.
Collaborators: Tim Bickmore (Northeastern), Boston Medical Center |
The Activity-Initiated Messaging (AIM) app will be tested for assisting with health-related behavior change.
|
Exploring the use of Real-Time Activity Recognition to Support Hand Hygiene
|
In response to the global COVID-19 pandemic, our group is developing machine learning algorithms and UI systems that can detect personal handwashing and face touching behaviors from smartwatches. The algorithms are designed to run in real-time and work in a free-living environment, using data from accelerometers, gyroscopes, and magnetometers. We are creating models and testing them using a dataset acquired from multiple participants in free-living, which is then carefully annotated to support algorithm validation. We are moving towards the goal of creating a real-time prototype and validating it in a real-world, free-living environment.
See more if you are interested in participating in the ongoing study. |
Overview of our prototype hand hygiene behavior detection system
|
Interactive Visualization Tools for Type 1 Diabetes Treatment Decision Management
Type 1 diabetes (T1D), also called insulin dependent diabetes and juvenile diabetes, is an autoimmune disease that afflicts 1.25 million Americans. There are 40 thousand new diagnoses each year, and almost half of those are children and adolescents under age 20 years. The costs for our health care system are enormous, estimated at $14 billion annually. T1D is incurable, and people with T1D are estimated to lose over 10 years from their life expectancy. We are designing, building, and evaluating interactive visualization tools to help T1D patients and their caregivers make treatment decisions. In these visualizations we are showing data we collect using multiple devices and have data on the patient blood glucose levels, insulin administered, food eaten, exercise, and stress to name a few. Our tools are used by patients to understand the trends between scheduled events such as mealtimes, bedtimes, and overnight as well as irregular events like periods of exercise, stress, and illness. With this information, patients will be able to make more informed changes to their treatment protocols.
Collaborators: Cody Dunne (Northeastern), Boston Children’s Hospital
Lifestyle Modification and Social Support via a Text-Messaging Digital Companion
The goal of this project is to develop a completely new way of supporting lifestyle modification using sensor-enabled, just-in-time and context-sensitive text messaging, with a combination of automated feedback and expert feedback, but in a scalable way.
Collaborators: Dinesh John (Northeastern University)
Crowd-Sourced Annotation of Longitudinal Sensor Data to Enhance Data-Driven Precision Medicine for Behavioral Health
|
Longitudinal sensor data collected passively from wearable activity monitors and mobile phones will transform behavioral science by allowing researchers to use “big data,” but at the person-level, to understand how behavior and related environmental exposures impact health outcomes and personalize health intervention and research. We have been exploring whether typical mobile application game players might help scientists improve this type of data, by adding additional annotations that enrich the data, making it more useful for behavioral science and more amenable to automatic processing. This could help researchers to better understand how individual-level behaviors relate to health outcomes in current research studies that collect personal-level sensor data such as the NHANES, and future big data ventures such as the All of Us study.
Collaborators: Dinesh John (Northeastern University), Seth Cooper (Northeastern University) |
Prototype of a game designed to help researchers, by having game players label activities using accelerometer data.
|
Novel Statistical Models for EMA Studies and Longitudinal Analysis of Cancer Control Behaviors
|
This study is focused on the development and testing of novel multilevel statistical methods to examine the effects of subject-level parameters (variance and slope) of time-varying variables in ecological momentary assessment (EMA) studies of physical activity. The results will have the potential to make notable methodological and substantive contributions for analysis of EMA data and in understanding the complex effects of contexts and psychological states on physical and sedentary activity. Given the detrimental health effects and enormous costs of physical inactivity, such information could be of significant value to efforts to improve public health.
Collaborators: University of Chicago, University of Southern California |
Part of the interface of the software developed for researchers interested in new multi-level statistical methods.
|
Just-in-time Adaptive Feedback Systems to Assist Individuals with Spinal Cord Injuries
|
We have developed and validated a smartphone-based Physical Activity Monitoring System (PAMS) that uses machine learning algorithms to detect various wheelchair-based physical activities (PAs) in community settings, estimate energy expenditure, and provide real-time feedback about PA levels to individuals with spinal cord injury. The objective of this work is to develop algorithms that incorporate each individual’s automatically detected PA level and a clinician’s PA level recommendation to provide a smartphone application that helps a person set safe and highly personalized physical activity goals. By adapting the goals in real-time based on the person's actual behavior, the system aims to keep the individual feeling positive and motivated towards a change in the PA behavior.
Collaborators: Temple University |
In the PHIRE project, a smartphone, smartwatch, and wheel sensor are used to estimate energy expenditure.
|
Context-Sensitive Experience Sampling
|
We are studying how real-time activity and context recognition on mobile phones can be used for context-sensitive experience sampling (or context-sensitive ecological momentary assessment). Such systems could improve the quality of self-report data while minimizing burden on scientific participants, but at the same time leading to higher-quality data gathering. In addition, the systems may enable new forms of computer-assisted end-of-day recall, also valuable for certain types of research studies.
Collaborators: Genevieve Dunton (USC) |
Partial screenshot of the MobileTeens application, that used sensor data to help teens label their everyday activities using context-sensitive experience sampling.
|
Contextual Factors Predicting Compliance in Microinteraction Ecological Momentary Assessment (Micro-EMA)
|
The purpose of this project is to investigate the impacts of contextual factors on smartwatch micro-EMA responses. By comparing to previous findings on compliance bias of smartphone EMA, the study intends to advance understanding of the effects of context on participants’ willingness to respond to micro-EMA questionnaires. The findings will inform researchers of the key factors to enhance the quantity of answers in micro-EMA studies.
|
uEMA survey completion can remain well above 90% on days when participants get them prompted on a smartwatch even after a participant has been in a study for nearly 100 days and answered thousands of questions.
|
Projects on hiatus
These projects are currently not being developed, but we would like to restart them given appropriate student interest.
Measuring the Impact of a Community Bikeway
|
As part of our effort to explore the Virtual Bicycle Highway concept, we are gathering data on the Minuteman Commuter Bikeway, which runs from Bedford, Massachusetts to Arlington, Massachusetts. This is a great example of a bikeway that runs from the suburbs to the city. We are exploring how many people have changed their active commuting patterns because of the bikeway, and how much the bikeway has impacted the surrounding community.
|
The Minuteman Bikeway runs through four towns and is heavily used by commuters. But there is more to learn about how and why it is used.
|
Operant Conditioning for Motivating Behavior Change
How can mobile devices motivate behavior change? One way might be to wait patiently, automatically detecting when someone does something just a bit better than what that person normally does using sensors such as accelerometers... and then providing context-sensitive, tailored encouragement. In other words, software on a mobile device might use computational sensing to detect behavior, and operant conditioning to motivate gradual behavior change.
Automatic Detection of Smoking Behavior
|
The goal of this pilot project was to develop and evaluate real-time pattern recognition algorithms that run on mobile phones and process data from miniature, wearable motion sensors (accelerometers) to detect patterns of smoking behaviors in free-living individuals.
Collaborator(s): M.D. Anderson Cancer Research Center |
|
Generating a Free, High-Quality Food Product Database Using Games with a Purpose
|
The specific aim of this pilot project was to explore the use of “games with a purpose” approach as a proof-of-concept to construct a UPC and nutrient database.
Supported by: NIH/NCI |
|
Telemetric Assessment of Movement Stereotypy in Children with ASD
A study with the Groden Center, a school for autistic children, and Prof. Matthew Goodwin at Northeastern, that has eplored the use of wireless accelerometers for automatic detection of autistic stereotypies. We would like to extend our past work to build real-time, interactive interventions for autistic children and their caregivers and teachers.
Collaborator(s): Groden Center, University of Rhode Island, Prof. Matthew Goodwin (Northeastern)
Supported by: National Alliance for Autism
Completed Projects
These projects are currently not being further developed.
Optimizing Hospital Workflow and Quality Through Patient Engagement
|
This project extended the Hospital Buddy, a computer agent that provides continual health counseling and companionship during a patient’s hospital stay via a bedside touch screen computer equipped with sensors (RFID tags on staff, accelerometer on patient, acoustics) that can detect certain events occurring in the hospital room.
Collaborator(s): Timothy Bickmore (Northeastern), Harriet Fell (Northeastern), Boston Medical Center |
A patient using the Hospital Buddy system in a hospital.
|
Enabling Population Scale Physical Activity Measurement on Mobile Phones
|
A study with Stanford School of Medicine to create novel health monitoring tools for mobile phones. This work led to the Wocket wearable accelerometer sensors, which were used for a time in several projects but have gone out of the date. The project included a supplement to develop mobile context-sensitive ecological momentary assessment software for mobile phones (“Extensible Platform for Implementing Experience Sampling on Mobile Phones”). Work continues today on this topic, although through different sources of funding and we are no longer making hardware.
Here is a flash video of the concept. There is still a need for this type of system to be produced commercially. Collaborator(s): Stanford Medical School |
Wocket sensors were thin, lightweight custom sensors we designed for capturing multi-limb accelerometer data in research studies, using common mobile phones.
|
Brain-Computer Interface for Signaling Changes in Physiological States
The major goal of this pilot project was to establish the feasibility of using fNIRS to detect major changes in emotional state and to use a mobile system for context-sensitive self-report on that state change. The prototype suffered from substantial motion artifacts that we were not able to adequately filter out.
Collaborator(s): Dana Brooks (Northeastern), Lisa Feldman Barrett (Northeastern), Matthew Goodwin (Northeastern), Deniz Erdogmus (Northeastern)
Cell Phone Intervention Trial for You (CITY)
|
This was a five-year study to develop and evaluate (in a randomized clinical trial) sensor-enabled mobile phone technology to assist young adults aged 18-35 with long-term weight loss and weight management for two years. We designed and created the mobile app and deployed it with over 200 people.
Collaborator(s): Duke Medical School |
Screenshots from the CITY Android application.
|
Development of Longitudinal Home Activity Datasets as a Shared Resource
|
This project developed portable sensor tools designed to be used in typical homes to collect data for computer science and health research (the BoxLab), as well as to generate shared datasets on home activity from actual homes to be used as a community resource to accelerate research.
Collaborator(s): MIT |
The Boxlab allowed data collection from any home, using portable sensors. The data were aggregated and then could be used for research using this visualization and annotation software.
|
The PlaceLab: A Living Laboratory for Home-Based Technology Research
|
How can ubiquitous computing technologies be created that work for extended periods of time? How can persuasive interfaces designed to motivate behavior change in the home be studied in context? Dr. Intille's research group, in collaboration with partner TIAX, developed the PlaceLab - an apartment-scale shared research facility in a residential condominium building in Cambridge where new technologies and design concepts could be tested and evaluated in the context of everyday living. For a sense of the capabilities it had, watch the PlaceLab overview video, or the more light-hearted video that demonstrates all the sensing. The PlaceLab had to be closed prematurely and was replaced by the BoxLab.
To learn more:
Collaborator(s): MIT, TIAX LLC |
The PlaceLab was a real apartment that had hundreds of sensors embedded within the architecture, including cameras, object usage sensors, current flow sensors, water flow sensors, and more.
Data, including video, audio, and other sensor activation data, were collected and used for experiments, such as automatic recognition of activity.
|
MITes (Portable Object-Usage Sensors)
We developed a kit of wireless portable sensors that could be easily deployed in homes and offices for studying behavior and creating and evaluating novel context-sensitive human-computer interface systems. The sensors were extensively tested (some in the PlaceLab), and they were adopted by a few other research groups for a time. They included sensors for measuring object motion, acceleration, current flow, heart rate, ultraviolet light, indoor position, and more. The mobile version of the sensors evolved into the Wockets.
Computational Perception and Persuasive User Interface Design (multiple projects)
|
Can computers with computational sensing motivate behavior change? Over the years, the group has worked on several projects using mobile and ubiquitous computers to both measure and motivate behavior change. We are studying the intersection between new user interface design strategies, computational tools for studying behavior change, computational learning algorithms for context recognition and planning, and behavioral and social psychology. We focus on the evaluation of algorithms and devices we create in real environments. Much of this work involves measuring and motivating physical activity and healthy eating.
One project developed a "GPS Thermostat" system that may encourage less energy use. Another investigated how point-of-decision prompting in a train station could impact behavior. A third looked at how to motivate less television watching (using a television remote control). |
We use paper prototyping and other user interface design strategies extensively when building prototype systems to deploy in research studies on motivating behavior change.
|
Maternal Stress and Children’s Obesity Risk (MATCH) Study
The goal of this project was to determine whether levels of stress among working mothers are related to increased obesity risk in their children. We used ecological momentary assessment to examine within-day mother-to-child stress processes that contribute to children’s long-term obesity risk in an accumulated manner over time. Working mothers and their children (9 to 11 years old) participated in six semi-annual assessments waves across three years using technology created at Northeastern. These data are now supporting ongoing work, including on new statistical methods for analyzing EMA data.
Collaborator(s): University of Southern California Medical School
Supported by: NIH/NHLBI
Understanding HIV Risk Environment for Youth in Supportive Housing
Supportive housing (SH) is a structural intervention that is being increasingly applied to homelessness among transition aged youth (TAY). Despite data indicating the positive effects of SH on housing stability, there are elements of SH that may promote risky behaviors that increase HIV rates. This study investigates the mechanisms that explain the association between SH and HIV risk behaviors. In addition, since research has not addressed whether SH yields the same outcomes for all TAY, we will investigate how individual factors such as age, gender, or sexual orientation moderate the effects of SH on TAY HIV risk behaviors. The goal of this investigation is to provide an empirical basis to guide the future development of SH for homeless TAY. This 3-year mixed methods study is guided by a robust social ecological conceptual framework of risk environment that refers to a space- whether social and/or physical-in which a variety of individual and contextual factors interact to influence HIV risk behaviors. For aim 1 (quantitative), we examine the impact of SH on TAY HIV risk behaviors (sexual risk and substance use) by recruiting 150 TAY living in SH along with a matched comparison group of 150 TAY who are currently homeless (total N=300). Ecological momentary assessment (EMA), which is considered the gold standard for capturing ecologically valid self-report, will be used to collect context specific predictive data on HIV risk behaviors. TAY will be prompted multiple times a day over a one week period to answer a series of surveys using mobile phone technology. For aim 2 (qualitative), we will investigate high and low risk environments within SH for TAY through the use an interactive, personalized geospatial map rendering of EMA responses (generated through mobile phone GPS) as an elicitation device for in-depth interviews. 60 participants will be purposively sampled to represent both high (n=30) and low (n=30) HIV risk taking. For aim 3, we will develop recommendations for designing SH for TAY that minimizes HIV risk though guideline development methods adopted by the World Health Organization. In addition to providing an empirical basis to guide the future development of SH for homeless TAY, the findings from this study can capitalize on TAY's widespread use of mobile phone technology and be used to design Ecological Momentary Interventions for both homeless and housed TAY.
Collaborator(s): Benjamin Henwood, University of Southern California Medical School
