Publications
Tran, H., V. Potter, U. Mazzucchelli, D. John and S. Intille (2026). "Towards practical, best practice video annotation to support human activity recognition." ARDUOUS 2025 CCIS 2706: 1-25. Student lead author. Best paper award.
Researchers need ground-truth activity annotations to train and evaluate wearable-sensor-based activity recognition models. Oftentimes, researchers establish ground truth by annotating the video recorded while someone engages in activity wearing sensors. The "gold-standard" video annotation practice requires two trained annotators independently annotating the same footage with a third domain expert resolving disagreements. Because such annotation is laborious, widely-used datasets have often been annotated using only a single annotator per video. Because the research community is moving towards collecting data of more complex behaviors from free-living people 24/7 and annotating more granular, fleeting activities, the annotation task grows even more challenging; the single-annotator approach may yield inaccuracies. We investigated a "silver-standard" approach: rather than using two independent annotation passes, a second annotator revises the work of the first annotator. The proposed approach reduced the total annotation time by 33% compared to the gold-standard approach, with near-equivalent annotation quality. The silver-standard label was in higher agreement with the gold-standard label than the single-annotator label, with Cohen's κ of 0.77 and 0.68 respectively on a 16.4 h video. The silver-standard labels also had higher inter-rater reliability than the single-annotator labels, with the respective mean Cohen's κ across six videos (92 h of total footage) of 0.79 and 0.68.
Cho, Y., S.-M. Chow, J. Li, W.-L. Wang, S. Wang, L. Ji, V. Chinchilli, S. Intille and D. GF (2025 (in-press)). "Understanding within- and between-individual compliance in mHealth: A joint modeling approach to non-random missingness." JMIR Mhealth Uhealth.
Background: Missing data are inevitable in mHealth research and driven by both within- and between-person variations in compliance levels. Not distinguishing these different sources can lead to biases in health behavior inferences. However, current missing data handling techniques do not address disentangling these distinct missingness mechanisms. Compared to missingness at random (MAR), missingness not at random (MNAR) is particularly concerning---often termed non-ignorable missingness. Objective: We demonstrate the utility of a joint modeling approach that simultaneously accommodates dynamics of health behavior changes as well as within- and between-person missingness mechanisms. We also evaluate how conflating these distinct contributors of (possibly non-ignorable) missingness affects the validity of health behavior inferences. We provide practical recommendations for building such joint models with empirical data. Methods: We applied a joint model on empirical data comprising one year of daily observations of affect (i.e., feeling energetic) reported through smartphone- based ecological momentary assessment (EMA) and smartwatch-tracked physical activity (PA). We implemented a joint modeling approach combining (1) a multilevel vector autoregressive (VAR) model for examining the reciprocal influences between daily affect and PA, and (2) a multilevel probit model for missingness mechanisms. As a sensitivity analysis, we compared the results from the proposed approach against other methods that examined health behavior changes without simultaneously modeling missingness mechanisms. Additionally, we validated the joint modeling approach through simulated data mirroring missing data patterns observed in empirical data: temporally clustered (e.g., consecutive days of) missingness and across-individual heterogeneity in compliance rates. Results: Sensitivity analysis indicated relative robustness of the autoregressive (AR) effects across missing data handling approaches, whereas cross-regressive (CR) effects could only be detected under the joint modeling, but not with methods that did not simultaneously model the missingness mechanism. Specifically, under the joint modeling, participants had higher levels of PA on days following a previous day with higher energy levels (95% CI=\[0.012, 0.049\]), and a higher level of PA on one day was associated with higher energy levels the next day (95% CI=\[0.006, 0.054\]). Furthermore, the missing data model revealed both MAR and MNAR missing mechanisms. For example, lower PA was linked with higher missingness in PA at the within-person level (95% CI=\[-1.528, -1.441\]). Employment status was associated with compliance in wearables data (95% CI=\[0.148, 0.574\]). Finally, simulation studies demonstrated that joint modeling improves the accuracy of the substantive model's estimate and identifies non-ignorable missing mechanisms effectively. Conclusions: We recommend utilizing joint modeling, particularly with multilevel decomposition to address non-ignorable missingness in mHealth studies collecting intensive longitudinal data. Simulation study showed joint modeling yielded results as accurate as those from fully observed data and enhanced understanding of within- and between-individual sources of missingness.
Crosley-Lyons, R., J. Li, W. L. Wang, S. D. Wang, J. Huh, D. Bae, S. S. Intille and G. F. Dunton (2025). "Exploring person-centred sleep and rest-activity cycle dynamics over 6 months." J Sleep Res: e14471.
Sleep and circadian characteristics are associated with health outcomes, but are often examined cross-sectionally or using variable-centred analyses. Person-centred longitudinal research is needed to identify combined effects of sleep and circadian characteristics while allowing for change over time. We aimed to classify individuals into sleep-circadian statuses (aim 1), determine whether they transitioned between statuses over time (aim 2), and explore associated covariates and health outcomes (aim 3). Young adults (N = 151) wore smartwatches continuously for 6 months. Sleep (total sleep time, wake after sleep onset) and circadian rest-activity cycle indicators (interdaily stability, intradaily variability, relative amplitude) were derived from acceleration data and aggregated into person-means for months 1, 3, and 6. These values were entered into a latent transition model for aims 1 and 2. Multinomial logistic regressions, ANOVA, and ANCOVA addressed aim 3. Four statuses were extracted (entropy = 0.88): optimal sleepers, restless sleepers, short sleepers, and nappers. 10%-13% of optimal sleepers and 21% of restless sleepers became nappers, 7%-18% of nappers transitioned to other statuses, and 94%-100% of short sleepers remained unchanged. Males were more likely than females to be short versus optimal sleepers (p \< 0.001). Restless sleepers had more physical dysfunction than nappers and short sleepers (p = 0.014, 0.022), while short sleepers reported more excessive sleepiness than optimal sleepers and nappers (p = 0.006, 0.060). This study identified four sleep-circadian statuses and found evidence for change over time. Our longitudinal person-centred approach could help inform the development of tailored diagnostic guidelines for sleep and circadian-related disorders that fluctuate within-individuals.
Dunton, G. F., W.-L. Wang, J. Li, S. Wang, D. Hedeker, S. S. Intille and A. J. Rothman (2025). "Prevalence of physical activity maintenance across a 12-month study: Comparison of accelerometer indicators." Journal of Physical Activity and Health: In press.
Background: Maintaining physical activity (PA) is critical for reducing disease risk. Yet, lack of consensus on how to define and operationalize PA maintenance has hindered surveillance efforts. We used longitudinal accelerometer data to compare how different ways of operationalizing PA maintenance impact PA maintenance prevalence estimates. Methods: Young adults (N = 173, ages 18-29) provided up to 12 months of PA data via smartwatch accelerometers. Nonsleep movement data were processed into 7-day moving averages of Monitor-Independent Movement Summary units. PA maintenance was operationalized using combinations of 3 accelerometer-based indicators: (1) threshold (ie, level of PA required: \[5.0-20.0 Monitor-Independent Movement Summary-units/min\], (2) duration (ie, time required above a threshold \[7-70 d\]), and (3) allowance (ie, time allowed below a threshold \[0-40 d\]). Outcomes included the prevalence of days, episodes, and number of participants classified into PA maintenance. Results: Increasing PA thresholds led to larger changes in PA maintenance prevalence outcomes than increasing durations or allowances. Greater changes in PA maintenance outcomes were observed when increasing thresholds up to about 12 Monitor-Independent Movement Summary-units/minute and allowances up to 7 days than when increasing above those points. Changes in PA maintenance outcomes were consistent across the entire range of durations. Conclusions: Threshold emerged as a more influential determinant of PA maintenance prevalence than duration or allowance, with greater changes across the lower range of thresholds. Validating these accelerometer-based indicators is a critical next step for establishing consensus regarding PA maintenance classification that can guide population-level surveillance.
Lakshminarayanan, R., A. Uppal, H. Le, J. C. Spilsbury and S. Intille (2025). "Detecting sleep disruptions in adolescents using context-sensitive ecological momentary assessment: A feasibility study." Pervasive Computing Technologies for Healthcare: 308-321. Student lead author
Adolescents are recommended to sleep at least 8--10 h per day. Inadequate sleep in adolescents is detrimental to their overall wellbeing and is linked to poor academic performance. Identifying causes of poor sleep in the sleep environment can help researchers and adolescents determine what changes need to be made to improve sleep quality. However, in-situ sleep monitoring is challenging because measurements cannot interfere with sleep, and people are poor at remembering what happens during the night. We report on the feasibility testing of an in-situ sleep monitoring application that uses passive sensing to drive context-sensitive ecological momentary assessments (EMAs) to help participants recall sleep disruptions when they wake up in the morning. Participants answered over 80% of EMAs delivered during the feasibility study and could recall meaningful reasons for over 40% of noise and motion events when they answered context-sensitive questions presented in the morning EMA. We discuss some challenges and future opportunities in sleep disruption detection.
Le, H., A. Choube, V. D. Swain, V. Mishra and S. Intille (2025). "A multi-agent LLM network for suggesting and correcting human activity and posture annotations." Proceedings of the GenAI4HS Workshop (Ubicomp 2025).
Accurate human activity recognition (HAR) is critical for health monitoring and behavior-aware systems. Developing reliable HAR models, however, requires large, high-quality labeled datasets that are challenging to collect in free-living settings. Although self-reports offer a practical solution for acquiring activity annotations, they are prone to recall biases, missing data, and human errors. Context-assisted recall can help participants remember their activities more accurately by providing visualizations of multiple data streams, but triangulating this information remains a burdensome and cognitively demanding task. In this work, we adapt GLOSS, a multi-agent LLM system that can triangulate self-reports and passive sensing data to assist participants in activity recall and annotation by suggesting the most likely activities. Our results show that GLOSS provides reasonable activity suggestions that align with human recall (63--75\\% agreement) and even effectively identifies and corrects common human annotation errors. These findings demonstrate the potential of LLM-powered, human-in-the-loop approaches to improve the quality and scalability of activity annotation in real-world HAR studies.
Le, H., V. Potter, A. Choube, R. Lakshminarayanan, V. Mishra and S. Intille (2025). "A context-assisted, semi-automated activity recall interface allowing uncertainty." Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 9(4): Article 186. Student lead author
Measuring activities and postures is an important area of research in ubiquitous computing, human-computer interaction, and personal health informatics. One approach that researchers use to collect large amounts of labeled data to develop models for activity recognition and measurement is asking participants to self-report their daily activities. Although participants can typically recall their sequence of daily activities, remembering the precise start and end times of each activity is significantly more challenging. ACAI is a novel, context-assisted ACtivity Annotation Interface that enables participants to efficiently label their activities by accepting or adjusting system-generated activity suggestions while explicitly expressing uncertainty about temporal boundaries. We evaluated ACAI using two complementary studies: a usability study with 11 participants and a two-week, free-living study with 14 participants. We compared our activity annotation system with the current gold-standard methods for activity recall in health sciences research: 24PAR and its computerized version, ACT24. Our system reduced annotation time and perceived effort while significantly improving data validity and fidelity compared to both standard human-supervised and unsupervised activity recall approaches. We discuss the limitations of our design and implications for developing adaptive, human-in-the-loop activity recognition systems used to collect self-report data on activity.
Le, H., V. Potter, R. Lakshminarayanan, V. Mishra and S. Intille (2025). "Feasibility and utility of multimodal micro ecological momentary assessment on a smartwatch." Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems: Article 1182. Student lead author
μEMAs allow participants to answer a short survey quickly with a tap on a smartwatch screen or a brief speech input. The short interaction time and low cognitive burden enable researchers to collect self-reports at high frequency (once every 5-15 minutes) while maintaining participant engagement. Systems with single input modality, however, may carry different contextual biases that could affect compliance. We combined two input modalities to create a multimodal-μEMA system, allowing participants to choose between speech or touch input to self-report. To investigate system usability, we conducted a seven-day field study where we asked 20 participants to label their posture and/or physical activity once every five minutes throughout their waking day. Despite the intense prompting interval, participants responded to 72.4% of the prompts. We found participants gravitated towards different modalities based on personal preferences and contextual states, highlighting the need to consider these factors when designing context-aware multimodal μEMA systems.
Ponnada, A., S. Wang, J. Li, W.-L. Wang, G. F. Dunton, D. Hedeker and S. S. Intille (2025). "Longitudinal user engagement with microinteraction ecological momentary assessment (μEMA)." Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 9(3): Article 121. Student lead author
Microinteraction ecological momentary assessment (μEMA) is a type of EMA that uses single-question prompts on a smartwatch to collect real-world self-reports. Smaller-scale studies show that μEMA yields higher response rates than EMA for up to 4 weeks. In this paper, we evaluated μEMA's longitudinal engagement in a 12-month study. Each participant completed EMA surveys (one smartphone prompt/hour for 96 days in 4-day bursts) and μEMA surveys (four smartwatch prompts/hour for the 270 days). Using data from 177 participants (1.37 million μEMA and 14.9K EMA surveys), we compared engagement across three groups: those who completed 12 months of EMA data collection(Completed), those who voluntarily withdrew after six months of EMA data collection (Withdrew), and those unenrolled by staff after six months of poor EMA response rates (Unenrolled). Compared to EMA, unenrolled participants were 2.25 times, those who withdrew were 1.65 times, and completed participants were 1.53 times more likely to answer μEMA prompts (p \< 0.001). Regardless of response rates, \|μEMA was perceived as less burdensome than EMA (p \< 0.001). These results suggest μEMA is a viable method for intensive longitudinal data collection, particularly for participants who find EMA unsustainable.
Potter, V., H. Le, U. H. Syeda, S. Intille and M. A. Borkin (2025). "An evaluation of temporal and categorical uncertainty on timelines: A case study in human activity recall visualizations." Proc. of IEEE VIS: Visualization & Visual Analytics 2025. Student lead author
Encoding uncertainty in timelines can provide more precise and informative visualizations (e.g., visual representations of unsure times or locations in event planning timelines). To evaluate the effectiveness of different temporal and categorical uncertainty representations on timelines, we conducted a mixed methods user study with 81 participants on uncertainty in activity recall timelines (ARTs). We find that participants' accuracy is better when temporal uncertainty is encoded using transparency instead of dashing, and that a participant's visual encoding preference does not always align with their performance (e.g., they performed better with a lesspreferred visual encoding technique). Additionally, qualitative findings show that existing biases of an individual alter their interpretation of ARTs. A copy of our studymaterials is available at [https://osf.io/98p6m/](https://osf.io/98p6m/).
Potter, V., H. Tran, D. Mobley, S. M. Bertisch, D. John and S. Intille (2025). "The Physical Activity Assessment Using Wearable Sensors (PAAWS) Dataset: Labeled laboratory and free-living accelerometer data." Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 9(4, Article 204 (December 2025)). Student lead author
Poor sleep and sedentary behavior patterns increase the risk of chronic diseases and negatively impact an individual's health and quality of life. Large-scale surveillance studies can unobtrusively measure free-living physical activities, sedentary behaviors, and sleep using wearable sensors; however, many human activity recognition algorithms cannot reliably detect activities in true free-living settings because they are trained on data collected in a controlled, lab setting. We describe the data collection protocol and present the first release of a multimodal, multi-sensor-site dataset (PAAWS R1). The PAAWS R1 release includes \~4 hours of semi-naturalistic activities from 252 individuals and \~7 days of 24-hour, free-living activities from 20 adults. We have annotated waking day activities using video to provide second-by-second, ground-truth labels capturing short, quickly changing bouts of activity with realistic activity transitions. Additionally, we have labeled up to two nights of sleep stages from PSG data collected during some nights of the free-living protocol. The PAAWS dataset enables researchers to directly compare activity recognition algorithms on the same participants' data across multiple collection protocols and days of free-living behaviors, encouraging convergence towards robust algorithms that could aid health research and drive novel mobile computing interventions and applications.
Carey, R. L., H. Le, D. L. Coffman, I. Nahum-Shani, M. Thirumalai, C. Hagen, L. A. Baehr, M. Schmidt-Read, M. S. R. Lamboy, S. A. Kolakowsky-Hayner, R. J. Marino, S. S. Intille and S. V. Hiremath (2024). "mHealth-based just-in-time adaptive intervention to improve the physical activity levels of individuals with spinal cord injury: Protocol for a randomized controlled trial." JMIR Res Protoc 13: e57699.
Background: The lack of regular physical activity (PA) in individuals with spinal cord injury (SCI) in the United States is an ongoing health crisis. Regular PA and exercise-based interventions have been linked with improved outcomes and healthier lifestyles among those with SCI. Providing people with an accurate estimate of their everyday PA level can promote PA. Furthermore, PA tracking can be combined with mobile health technology such as smartphones and smartwatches to provide a just-in-time adaptive intervention (JITAI) for individuals with SCI as they go about everyday life. A JITAI can prompt an individual to set a PA goal or provide feedback about their PA levels. Objective: The primary aim of this study is to investigate whether minutes of moderate-intensity PA among individuals with SCI can be increased by integrating a JITAI with a web-based PA intervention (WI) program. The WI program is a 14-week web-based PA program widely recommended for individuals with disabilities. A secondary aim is to investigate the benefit of a JITAI on proximal PA, defined as minutes of moderate-intensity PA within 120 minutes of a PA feedback prompt. Methods: Individuals with SCI (N=196) will be randomized to a WI arm or a WI+JITAI arm. Within the WI+JITAI arm, a microrandomized trial will be used to randomize participants several times a day to different tailored feedback and PA recommendations. Participants will take part in the 24-week study from their home environment in the community. The study has three phases: (1) baseline, (2) WI program with or without JITAI, and (3) PA sustainability. Participants will provide survey-based information at the initial meeting and at the end of weeks 2, 8, 16, and 24. Participants will be asked to wear a smartwatch every day for ≥12 hours for the duration of the study. Results: Recruitment and enrollment began in May 2023. Data analysis is expected to be completed within 6 months of finishing participant data collection. Conclusions: The JITAI has the potential to achieve long-term PA performance by delivering tailored, just-in-time feedback based on the person's actual PA behavior rather than a generic PA recommendation. New insights from this study may guide intervention designers to develop engaging PA interventions for individuals with disability.
Do, B., D. Hedeker, W.-L. Wang, T. B. Mason, B. R. Belcher, K. A. Miller, A. J. Rothman, S. S. Intille and G. F. Dunton (2024). "Investigating the day-level associations between affective variability and physical activity using ecological momentary assessment." Psychology of Sport and Exercise 70: 102542.
Background Understanding affect as a determinant of physical activity has gained increased attention in health behavior research. Fluctuations in affect intensity from moment-to-moment (i.e., affective variability) may interfere with cognitive and regulatory processes, making it difficult to engage in goal-directed behaviors such as physical activity. Preliminary evidence indicates that those with greater trait-level affective variability engage in lower levels of habitual physical activity. However, the extent to which daily fluctuations in affect variability are associated with same-day physical activity levels is unknown. This study used ecological momentary assessment (EMA) to investigate day-level associations between affective variability (i.e., within-subject variance) and physical activity. Methods Young adults (N = 231, M = 23.58 ± 3.02 years) provided three months of smartphone-based EMA and smartwatch-based activity data. Every two weeks, participants completed a 4-day EMA measurement burst (M = 5.17 ± 1.28 bursts per participant). Bursts consisted of hourly randomly-prompted EMA surveys assessing momentary positive-activated (happy, energetic), positive-deactivated (relaxed), negative-activated (tense, stressed), and negative-deactivated (sad, fatigued) affect. Participants continuously wore a smartwatch to measure physical activity across the three months. Mixed-effects location scale modeling examined the day-level associations of affective variability (i.e., positive-activated, positive-deactivated, negative-activated, and negative-deactivated) and physical activity, controlling for covariates such as mean levels of affect, between-subject effects of physical activity, time of day, day of week, day in study, and smartwatch wear time. Results There were 41,546 completed EMA surveys (M = 182.22 ± 69.82 per participant) included in the analyses. Above and beyond mean levels of affect, greater day-level variability in positive-activated affect was associated with greater physical activity on that same day compared to other days (τ = 0.01, p \< .001), whereas greater day-level variability in negative-deactivated affect was associated with less physical activity on that same day compared to other days (τ = −0.01, p \< .001). Day-level variability in positive-deactivated affect or negative-activated affect were not associated with day-level physical activity (ps \> .05) Conclusions Individuals were less active on days with greater variability in feeling sad and fatigued but more active on days with greater variability in feeling happy and energetic. Understanding the dynamic relationships of affective variability with day-level physical activity can strengthen physical activity interventions by considering how these processes differ within individuals and unfold within the context of daily life. Future research should examine causal pathways between affective variability and physical activity across the day.
Le, H., R. Lakshminarayanan, J. Li, V. Mishra and S. Intille (2024). "Collecting self-reported physical activity and posture data using audio-based ecological momentary assessment." Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 8(3): Article 111. Student lead author
μEMA is a data collection method that prompts research participants with quick, answer-at-a-glance, single-multiple-choice self-report behavioral questions, thus enabling high-temporal-density self-report of up to four times per hour when implemented on a smartwatch. However, due to the small watch screen, μEMA is better used to select among 2 to 5 multiple-choice answers versus allowing the collection of open-ended responses. We introduce an alternative and novel form of micro-interaction self-report using speech input - audio-μEMA- where a short beep or vibration cues participants to verbally report their behavioral states, allowing for open-ended, temporally dense self-reports. We conducted a one-hour usability study followed by a within-subject, 6-day to 21-day free-living feasibility study in which participants self-reported their physical activities and postures once every 2 to 5 minutes. We qualitatively explored the usability of the system and identified factors impacting the response rates of this data collection method. Despite being interrupted 12 to 20 times per hour, participants in the free-living study were highly engaged with the system, with an average response rate of 67.7% for audio-μEMA for up to 14 days. We discuss the factors that impacted feasibility; some implementation, methodological, and participant challenges we observed; and important considerations relevant to deploying audio-μEMA in real-time activity recognition systems.
Li, J., A. Ponnada, W.-L. Wang, G. Dunton and S. Intille (2024). "Ask less, learn more: Adapting ecological momentary assessment survey length by modeling question-answer information gain." Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 8(4): Article 166. Student lead author
Ecological momentary assessment (EMA) is an approach to collect self-reported data repeatedly on mobile devices in natural settings. EMAs allow for temporally dense, ecologically valid data collection, but frequent interruptions with lengthy surveys on mobile devices can burden users, impacting compliance and data quality. We propose a method that reduces the length of each EMA question set measuring interrelated constructs, with only modest information loss. By estimating the potential information gain of each EMA question using question-answer prediction models, this method can prioritize the presentation of the most informative question in a question-by-question sequence and skip uninformative questions. We evaluated the proposed method by simulating question omission using four real-world datasets from three different EMA studies. When compared against the random question omission approach that skips 50% of the questions, our method reduces imputation errors by 15%-52%. In surveys with five answer options for each question, our method can reduce the mean survey length by 34%-56% with a real-time prediction accuracy of 72%-95% for the skipped questions. The proposed method may either allow more constructs to be surveyed without adding user burden or reduce response burden for more sustainable longitudinal EMA data collection.
Wang, S., L. Hatzinger, J. Morales, M. Hewus, S. Intille and G. Dunton (2024). "Burden and inattentive responding in a 12-month intensive longitudinal study: A qualitative analysis." JMIR Form Res 8: e52165.
Background: Intensive longitudinal data (ILD) collection methods have gained popularity in social and behavioral research as a tool to better understand behavior and experiences over time with reduced recall bias. Engaging participants in these studies over multiple months and ensuring high data quality are crucial but challenging due to the potential burden of repeated measurements. It is suspected that participants may engage in inattentive responding (IR) behavior to combat burden, but the processes underlying this behavior are unclear as previous studies have focused on the barriers to compliance rather than the barriers to providing high-quality data. Objective: This study aims to broaden researchers' knowledge about IR during ILD studies using qualitative analysis and uncover the underlying IR processes to aid future hypothesis generation. Methods: We explored the process of IR by conducting semistructured qualitative exit interviews with 31 young adult participants (aged 18-29 years) who completed a 12-month ILD health behavior study with daily evening smartphone-based ecological momentary assessment (EMA) surveys and 4-day waves of hourly EMA surveys. The interviews assessed participants' motivations, the impact of time-varying contexts, changes in motivation and response patterns over time, and perceptions of attention check questions (ACQs) to understand participants' response patterns and potential factors leading to IR. Results: Thematic analysis revealed 5 overarching themes on factors that influence participant engagement: (1) friends and family also had to tolerate the frequent surveys, (2) participants tried to respond to surveys quickly, (3) the repetitive nature of surveys led to neutral responses, (4) ACQs within the surveys helped to combat overly consistent response patterns, and (5) different motivations for answering the surveys may have led to different levels of data quality. Conclusions: This study aimed to examine participants' perceptions of the quality of data provided in an ILD study to contribute to the field's understanding of engagement. These findings provide insights into the complex process of IR and participant engagement in ILD studies with EMA. The study identified 5 factors influencing IR that could guide future research to improve EMA survey design. The identified themes offer practical implications for researchers and study designers, including the importance of considering social context, the consideration of dynamic motivations, and the potential benefit of including ACQs as a technique to reduce IR and leveraging the intrinsic motivators of participants. By incorporating these insights, researchers might maximize the scientific value of their multimonth ILD studies through better data collection protocols.
Xu, X., B. Yao, Ziqi Yang, S. Zhang, E. Rogers, S. Intille, N. Shara, G. Gao and D. Wang (2024). "Talk2Care: Facilitating asynchronous patient-provider communication with large-language-model." Proceedings of the 2024 AAAI Fall Symposia 4(1): 146-151.
Despite the plethora of telehealth applications to assist home-based older adults and healthcare providers, basic messaging and phone calls are still the most common communication methods, which suffer from limited availability, information loss, and process inefficiencies. One promising solution to facilitate patient-provider communication is to leverage large language models (LLMs) with their powerful natural conversation and summarization capability. However, there is a limited understanding of LLMs' role during the communication. We first conducted two interview studies with both older adults (N=10) and healthcare providers (N=9) to understand their needs and opportunities for LLMs in patient-provider asynchronous communication. Based on the insights, we built an LLM-powered communication system, Talk2Care, and designed interactive components for both groups: (1) For older adults, we leveraged the convenience and accessibility of voice assistants (VAs) and built an LLM-powered conversational interface for effective information collection. (2) For health providers, we built an LLM-based dashboard to summarize and present important health information based on older adults' conversations with the VA. We further conducted two user studies with older adults and providers to evaluate the usability of the system. The results showed that Talk2Care could facilitate the communication process, enrich the health information collected from older adults, and considerably save providers' efforts and time. We envision our work as an initial exploration of LLMs' capability in the intersection of healthcare and interpersonal communication.
Yang, Z., X. Xu, B. Yao, E. Rogers, S. Zhang, S. Intille, N. Shara, G. G. Gao and D. Wang (2024). "Talk2Care: An LLM-based voice assistant for communication between healthcare providers and older adults." Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 8(2): Article 73. Distinguished paper award
Despite the plethora of telehealth applications to assist home-based older adults and healthcare providers, basic messaging and phone calls are still the most common communication methods, which suffer from limited availability, information loss, and process inefficiencies. One promising solution to facilitate patient-provider communication is to leverage large language models (LLMs) with their powerful natural conversation and summarization capability. However, there is a limited understanding of LLMs' role during the communication. We first conducted two interview studies with both older adults (N=10) and healthcare providers (N=9) to understand their needs and opportunities for LLMs in patient-provider asynchronous communication. Based on the insights, we built an LLM-powered communication system, Talk2Care, and designed interactive components for both groups: (1) For older adults, we leveraged the convenience and accessibility of voice assistants (VAs) and built an LLM-powered conversational interface for effective information collection. (2) For health providers, we built an LLM-based dashboard to summarize and present important health information based on older adults' conversations with the VA. We further conducted two user studies with older adults and providers to evaluate the usability of the system. The results showed that Talk2Care could facilitate the communication process, enrich the health information collected from older adults, and considerably save providers' efforts and time. We envision our work as an initial exploration of LLMs' capability in the intersection of healthcare and interpersonal communication.
Canori, A., R. Lakshminarayanan, M. Nunn, M. Schmidt-Read, S. S. Intille and S. V. Hiremath (2023). "Potential of social engagement for overcoming barriers to physical activity in individuals with spinal cord injury." Journal of Rehabilitation and Assistive Technologies Engineering 10: 20556683231185755. Student co-lead author
IntroductionMany barriers to physical activity (PA) exist for individuals with spinal cord injury (SCI). Social engagement may improve motivation to perform PA, which in turn may increase PA levels. This pilot study investigates how social engagement facilitated by mobile technology may reduce lack of motivation as a barrier to PA in individuals with SCI and demonstrates design implications for future technologies.MethodsA user-needs survey was conducted with participants in the community. We recruited 26 participants (16 individuals with SCI and 10 family members or peers). A participatory design process using semi-structured interviews was used to identify themes relating to PA barriers.ResultsOne theme related to PA barriers was lack of PA-focused forums to connect with peers. Participants with SCI considered connecting with other individuals with SCI more motivating than connecting with their family members. Another key finding was that participants with SCI did not perceive that personal fitness trackers were targeted towards wheelchair-based activities.ConclusionsEngagement and communication with peers who have similar functional mobility levels and life experiences can potentially improve motivation for PA; however, PA-motivational platforms are not tailored towards wheelchair-users. Our preliminary findings show that some individuals with SCI are not satisfied with current mobile-technologies for wheelchair-based PA.
Chow, S. M., I. Nahum-Shani, J. T. Baker, D. Spruijt-Metz, N. B. Allen, R. P. Auerbach, G. F. Dunton, N. P. Friedman, S. S. Intille, P. Klasnja, B. Marlin, M. K. Nock, S. L. Rauch, M. Pavel, S. Vrieze, D. W. Wetter, E. M. Kleiman, T. R. Brick, H. Perry and D. L. Wolff-Hughes (2023). "The ILHBN: Challenges, opportunities, and solutions from harmonizing data under heterogeneous study designs, target populations, and measurement protocols." Translational Behavioral Medicine 13(1): 7-16.
The ILHBN is funded by the National Institutes of Health to collaboratively study the interactive dynamics of behavior, health, and the environment using Intensive Longitudinal Data (ILD) to (a) understand and intervene on behavior and health and (b) develop new analytic methods to innovate behavioral theories and interventions. The heterogenous study designs, populations, and measurement protocols adopted by the seven studies within the ILHBN created practical challenges, but also unprecedented opportunities to capitalize on data harmonization to provide comparable views of data from different studies, enhance the quality and utility of expensive and hard-won ILD, and amplify scientific yield. The purpose of this article is to provide a brief report of the challenges, opportunities, and solutions from some of the ILHBN's cross-study data harmonization efforts. We review the process through which harmonization challenges and opportunities motivated the development of tools and collection of metadata within the ILHBN. A variety of strategies have been adopted within the ILHBN to facilitate harmonization of ecological momentary assessment, location, accelerometer, and participant engagement data while preserving theory-driven heterogeneity and data privacy considerations. Several tools have been developed by the ILHBN to resolve challenges in integrating ILD across multiple data streams and time scales both within and across studies. Harmonization of distinct longitudinal measures, measurement tools, and sampling rates across studies is challenging, but also opens up new opportunities to address cross-cutting scientific themes of interest. Health behavior changes, such as prevention of suicidal thoughts and behaviors, smoking, drug use, and alcohol use; and the promotion of mental health, sleep, and physical activities, and decreases in sedentary behavior, are difficult to sustain. The ILHBN is a cooperative agreement network funded jointly by seven participating units within the National Institutes of Health to collaboratively study how factors that occur in individuals' everyday life and in their natural environment influence the success of positive health behavior changes. This article discusses how information collected using smartphones, wearables, and other devices can provide helpful active and passive reflections of the participants' extent of risk and resources at the moment for an extended period of time. However, successful engagement and retention of participants also require tailored adaptations of study designs, measurement tools, measurement intervals, study span, and device choices that create hurdles in integrating (harmonizing) data from multiple studies. We describe some of the challenges, opportunities, and solutions that emerged from harmonizing intensive longitudinal data under heterogeneous study and participant characteristics within the ILHBN, and share some tools and recommendations to facilitate future data harmonization efforts.
Hester, J., H. Le, S. Intille and E. Meier (2023). "A feasibility study on the use of audio-based ecological momentary assessment with persons with aphasia." ASSETS '23: Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility: Article 55. Student lead author
We describe a smartphone/smartwatch system to evaluate anomia in individuals with aphasia by using audio-recording-based ecological momentary assessments. The system delivers object-naming assessments to a participant's smartwatch, whereby a prompt signals the availability of images of these objects on the watch screen. Participants attempt to speak the names of the images that appear on the watch display out loud and into the watch as they go about their lives. We conducted a three-week feasibility study with six participants with mild to moderate aphasia. Participants were assigned to either a nine-item (four prompts per day with nine images) or single-item (36 prompts per day with one image each) ecological momentary assessment protocol. Compliance in recording an audio response to a prompt was approximately 80% for both protocols. Qualitative analysis of the participants' interviews suggests that the participants felt capable of completing the protocol, but opinions about using a smartwatch were mixed. We review participant feedback and highlight the importance of considering a population's specific cognitive or motor impairments when designing technology and training protocols.
Keadle, S. K., J. Martinez, S. J. Strath, J. Sirard, D. John, S. Intille, D. Arguello, M. Amalbert-Birriel, R. Barnett, B. Thapa-Chhetry, M. Cox, J. Chase, E. Dooley, R. Marcotte, A. Tolas and J. W. Staudemayer (2023). "Evaluation of within- and between-site agreement for direct observation of physical behavior across four research groups." Journal for the Measurement of Physical Behaviour 6(3): 176-184.
Direct observation (DO) is a widely accepted ground-truth measure, but the field lacks standard operational definitions. Research groups develop project-specific annotation platforms, limiting the utility of DO if labels are not consistent. Purpose: The purpose was to evaluate within- and between-site agreement for DO taxonomies (e.g., activity intensity category) across four independent research groups who have used video-recorded DO. Methods : Each site contributed video files (508 min) and had two trained research assistants annotate the shared video files according to their existing annotation protocols. The authors calculated (a) within-site agreement for the two coders at the same site expressed as intraclass correlation and (b) between-site agreement, the proportion of seconds that agree between any two coders regardless of site. Results: Within-site agreement at all sites was good--excellent for both activity intensity categories (intraclass correlation range: .82--.9) and posture/whole-body movement (intraclass correlation range: .77--.98). Between-site agreement for intensity categories was 94.6% for sedentary, 80.9% for light, and 82.8% for moderate--vigorous. Three of the four sites had common labels for eight posture/whole-body movements and had within-site agreements of 94.5% and between-site agreements of 86.1%. Conclusions: Distinct research groups can annotate key features of physical behavior with good-to-excellent interrater reliability. Operational definitions are provided for core metrics for researchers to consider in future studies to facilitate between-study comparisons and data pooling, enabling the deployment of deep learning approaches to wearable device algorithm calibration.
Dunton, G. F., A. M. Leventhal, A. L. Rebar, B. Gardner, S. S. Intille and A. J. Rothman (2022). "Towards consensus in conceptualizing and operationalizing physical activity maintenance." Psychology of Sport and Exercise: 102214.
Recognized challenges in promoting long-term physical activity maintenance may be due to inconsistencies in the conceptualization and operationalization of behavior maintenance terminology in physical activity research. The overall goal of this paper is to propose a framework and agenda for the development of a common set of terms, definitions, and measures for physical activity maintenance concepts that can be widely tested and evaluated. To initiate this effort, this paper (1) provides an overview of conceptual and operational definitions of physical activity maintenance used in the empirical literature; (2) evaluates whether behavior maintenance terms used in addiction science can be translated to physical activity, (3) recommends research directions for developing consensus definitions of physical activity maintenance; and (4) proposes a conceptual model of physical activity maintenance with inflection points that require operational definitions to be decided upon through consensus efforts in the field. Consensus over the conceptualization and operationalization of physical activity maintenance is needed to draw conclusions regarding which policies and programs are best able to promote long-term behavior change.
Dunton, G. F., W.-L. Wang, S. S. Intille, E. Dzubur, A. Ponnada and D. Hedeker (2022). "How acute affect dynamics impact longitudinal changes in physical activity among children." Journal of Behavioral Medicine 45(3): 451-460.
Research examined how acute affect dynamics, including stability and context-dependency, contribute to changes in children's physical activity levels as they transition from late-childhood to early-adolescence. Children (N = 151) (ages 8--12 years at baseline) participated in an ecological momentary assessment and accelerometry study with six semi-annual bursts (7 days each) across three years. A two-stage mixed-effects multiple location-scale model tested random intercept, variance, and slope estimates for positive affect as predictors of moderate-to-vigorous physical activity (MVPA). Multi-year declines in MVPA were greater for children who had greater subject-level variance in positive affect. Children who experienced more positive affect when alone did not experience steeper declines in physical activity. Interventions aiming for long-term modifications in children's physical activity may focus on buffering the effects of within-day fluctuations in affect or tailoring programs to fit the needs of "acute dynamic process phenotypes."
Lakshminarayanan, R., A. Canori, A. Ponnada, M. Nunn, M. Schmidt-Read, S. Hiremath and S. Intille (2022). "Exploring opportunities to improve physical activity in individuals with spinal cord injury using context-aware messaging." Proceedings of the ACM on Human-Computer Interaction 6(CSCW2): Article 515. Student lead author
Spinal cord injury (SCI) affects the mobility of 250,000 people per year worldwide. Physical activity (PA) in individuals with SCI is positively associated with improved mental and physical health outcomes. Mobile technologies have been developed to motivate individuals with SCI to increase PA using activity tracking and real-time feedback. We conducted semi-structured interviews and participatory design sessions with 15 manual wheelchair users with SCI and eight of their family members/friends to investigate user impressions of future technologies that might use computer-mediated, sensor-triggered communication to motivate PA. We assessed barriers to PA and how context-aware communication could help overcome them. Participants with SCI expressed that PA tracking and communication technologies must be tailored to their specific needs. Further analysis revealed that context-aware messaging could help participants with SCI connect with others to initiate timely conversations about overcoming PA barriers, and to provide encouragement to meet their PA goals. We discuss opportunities to empower individuals with SCI with regards to PA using tailored, context-aware communication.
Ponnada, A., J. Li, S. Wang, W.-L. Wang, B. Do, G. F. Dunton and S. S. Intille (2022). "Contextual biases in microinteraction ecological momentary assessment (μEMA) non-response." Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6(1): 1--24. Student lead author, Distinguished Paper Award (Top 4% Vol 6) (2023)
Ecological momentary assessment (EMA) is used to gather in-situ self-report on behaviors using mobile devices. Microinteraction EMA (μEMA), is a type of EMA where each survey is only one single question that can be answered with a glanceable microinteraction on a smartwatch. Prior work shows that even when μEMA interrupts far more frequently than smartphone-EMA, μEMA yields higher response rates with lower burden. We examined the contextual biases associated with non-response of μEMA prompts on a smartwatch. Based on prior work on EMA non-response and smartwatch use, we identified 10 potential contextual biases from three categories: temporal (time of the day, parts of waking day, day of the week, and days in study), device use (screen state, charging status, battery mode, and phone usage), and activity (wrist motion and location). We used data from a longitudinal study where 131 participants (Mean age 22.9 years, SD = 3.0) responded to μEMA surveys on a smartwatch for at least six months. Using mixed-effects logistic regression, we found that all temporal, activity/mobility, and device use variables had a statistically significant (p\<0.001) association with momentary μEMA non-response. We discuss the implication of these results for future use of context-aware μEMA methodology.
Ponnada, A., S. Wang, D. Chu, B. Do, G. Dunton and S. Intille (2022). "Intensive longitudinal data collection using microinteraction ecological momentary assessment: Pilot and preliminary results." JMIR Formative Research 6(2): e32772. Student lead author
BACKGROUND: Ecological momentary assessment (EMA) uses mobile technology to enable in situ self-report data collection on behaviors and states. In a typical EMA study, participants are prompted several times a day to answer sets of multiple-choice questions. Although the repeated nature of EMA reduces recall bias, it may induce participation burden. There is a need to explore complementary approaches to collecting in situ self-report data that are less burdensome yet provide comprehensive information on an individual's behaviors and states. A new approach, microinteraction EMA (muEMA), restricts EMA items to single, cognitively simple questions answered on a smartwatch with single-tap assessments using a quick, glanceable microinteraction. However, the viability of using muEMA to capture behaviors and states in a large-scale longitudinal study has not yet been demonstrated. OBJECTIVE: This paper describes the muEMA protocol currently used in the Temporal Influences on Movement & Exercise (TIME) Study conducted with young adults, the interface of the muEMA app used to gather self-report responses on a smartwatch, qualitative feedback from participants after a pilot study of the muEMA app, changes made to the main TIME Study muEMA protocol and app based on the pilot feedback, and preliminary muEMA results from a subset of active participants in the TIME Study. METHODS: The TIME Study involves data collection on behaviors and states from 246 individuals; measurements include passive sensing from a smartwatch and smartphone and intensive smartphone-based hourly EMA, with 4-day EMA bursts every 2 weeks. Every day, participants also answer a nightly EMA survey. On non-EMA burst days, participants answer muEMA questions on the smartwatch, assessing momentary states such as physical activity, sedentary behavior, and affect. At the end of the study, participants describe their experience with EMA and muEMA in a semistructured interview. A pilot study was used to test and refine the muEMA protocol before the main study. RESULTS: Changes made to the muEMA study protocol based on pilot feedback included adjusting the single-question selection method and smartwatch vibrotactile prompting. We also added sensor-triggered questions for physical activity and sedentary behavior. As of June 2021, a total of 81 participants had completed at least 6 months of data collection in the main study. For 662,397 muEMA questions delivered, the compliance rate was 67.6% (SD 24.4%) and the completion rate was 79% (SD 22.2%). CONCLUSIONS: The TIME Study provides opportunities to explore a novel approach for collecting temporally dense intensive longitudinal self-report data in a sustainable manner. Data suggest that muEMA may be valuable for understanding behaviors and states at the individual level, thus possibly supporting future longitudinal interventions that require within-day, temporally dense self-report data as people go about their lives.
Semborski, S., B. Henwood, B. Redline, E. Dzubur, T. Mason and S. Intille (2022). "Feasibility and acceptability of ecological momentary assessment with young adults who are currently or were formerly homeless: Mixed methods study." JMIR Formative Research 6(3): e33387.
Background: Ecological momentary assessment (EMA) has been used with young people experiencing homelessness to gather information on contexts associated with homelessness and risk behavior in real time and has proven feasible in this population. However, the extent to which EMA may affect the attitudes or behaviors of young adults who are currently or were formerly homeless and are residing in supportive housing has not been well investigated. Objective: This study aims to describe the feedback regarding EMA study participation from young adults who are currently or were formerly homeless and examine the reactivity to EMA participation and compliance. Methods: This mixed methods study used cross-sectional data collected before and after EMA, intensive longitudinal data from a 7-day EMA prompting period, and focus groups of young adults who are currently or were formerly homeless in Los Angeles, California, between 2017 and 2019. Results: Qualitative data confirmed the quantitative findings. Differences in the experience of EMA between young adults who are currently or were formerly homeless were found to be related to stress or anxiety, interference with daily life, difficulty charging, behavior change, and honesty in responses. Anxiety and depression symptomatology decreased from before to after EMA; however, compliance was not significantly associated with this decrease. Conclusions: The results point to special considerations when administering EMA to young adults who are currently or were formerly homeless. EMA appears to be slightly more burdensome for young adults who are currently homeless than for those residing in supportive housing, which are nuances to consider in the study design. The lack of a relationship between study compliance and symptomatology suggests low levels of reactivity.
Thapa-Chhetry, B., D. J. Arguello, D. John and S. Intille (2022). "Detecting sleep and non-wear in 24-hour wrist accelerometer data from the National Health and Nutrition Examination Survey." Medicine & Science in Sports & Exercise 54(11): 1936-1946. Student lead author
INTRODUCTION: Estimating physical activity, sedentary behavior, and sleep from wrist-worn accelerometer data requires reliable detection of sensor non-wear and sensor wear during both sleep and wake. PURPOSE: To develop an algorithm that simultaneously identifies sensor wake-wear, sleep-wear, and non-wear in 24-hour wrist-accelerometer data collected with or without filtering. METHODS: Using sensor data labeled with polysomnography (N = 21) and directly observed wake-wear data (N = 31) from healthy adults, and non-wear data from sensors left at various locations in a home (N = 20), we developed an algorithm to detect non-wear, sleep-wear, and wake-wear for 'idle sleep mode' (ISM) filtered data collected in the 2011-2014 National Health and Nutrition Examination Survey. The algorithm was then extended to process original raw data collected from devices without ISM filtering. Both algorithms were further validated using a polysomnography-based sleep and wake-wear dataset (N = 22) and diary-based wake-wear and non-wear labels from healthy adults (N = 23). Classification performance (F1-scores) was compared to four alternative approaches. RESULTS: F1-score of the ISM-based algorithm on the training dataset using leave-one-subject-out cross-validation was 0.95 ± 0.13. Validation on the two independent datasets yielded F1-scores of 0.84 ± 0.60 for the dataset with sleep-wear and wake-wear and 0.94 ± 0.04 for the dataset with wake-wear and non-wear. F1 score when using original, raw data was 0.96 ± 0.08 for the training datasets and 0.86 ± 0.18 and 0.97 ± 0.04 for the two independent validation datasets. The algorithm performed comparably or better than the alternative approaches on the datasets. CONCLUSIONS: A novel machine-learning algorithm was designed to recognize wake-wear, sleep-wear, and non-wear in 24-hour wrist-worn accelerometer data that is applicable for ISM-filtered data or original raw data.
Wang, S., S. Intille, A. Ponnada, B. Do, A. Rothman and G. Dunton (2022). "Investigating microtemporal processes underlying health behavior adoption and maintenance: Protocol for an intensive longitudinal observational study." JMIR Research Protocols 11(7): e36666.
Background. Young adulthood (ages 18-29 years) is marked by substantial weight gain, leading to increased lifetime risks of chronic diseases. Engaging in sufficient levels of physical activity and sleep, and limiting sedentary time are important contributors to the prevention of weight gain. Dual-process models of decision-making and behavior that delineate reflective (ie, deliberative, slow) and reactive (ie, automatic, fast) processes shed light on different mechanisms underlying the adoption versus maintenance of these energy-balance behaviors. However, reflective and reactive processes may unfold at different time scales and vary across people. Objective. This paper describes the study design, recruitment, and data collection procedures for the Temporal Influences on Movement and Exercise (TIME) study, a 12-month intensive longitudinal data collection study to examine real-time microtemporal influences underlying the adoption and maintenance of physical activity, sedentary behavior, and sleep. Methods. Intermittent ecological momentary assessment (eg, intentions, self-control) and continuous, sensor-based passive monitoring (eg, location, phone/app use, activity levels) occur using smartwatches and smartphones. Data analyses will combine idiographic (person-specific, data-driven) and nomothetic (generalizable, theory-driven) approaches to build models that may predict within-subject variation in the likelihood of behavior "episodes" (eg, ≥10 minutes of physical activity, ≥120 minutes of sedentary time, ≥7 hours sleep) and "lapses" (ie, not attaining recommended levels for ≥7 days) as a function of reflective and reactive factors. Results. The study recruited young adults across the United States (N=246). Rolling recruitment began in March 2020 and ended August 2021. Data collection will continue until August 2022. Conclusions. Results from the TIME study will be used to build more predictive health behavior theories, and inform personalized behavior interventions to reduce obesity and improve public health.
Yi, L., S. D. Wang, D. Chu, A. Ponnada, S. S. Intille and G. F. Dunton (2022). "Examining whether physical activity location choices were associated with weekly physical activity maintenance across 13 months of the COVID-19 pandemic in emerging adults." Journal of Physical Activity and Health 19(6): 446-455.
BACKGROUND: Recent studies have shown potentially detrimental effects of the COVID-19 pandemic on physical activity (PA) in emerging adults (ages 18-29 y). However, studies that examined the effects of COVID-19 on PA location choices and maintenance for this age group remain limited. The current study investigated changes in PA location choices across 13 months during the pandemic and their associations with PA maintenance in this population. METHODS: Emerging adults (N = 197) living in the United States completed weekly survey on personal smartphones (May 2020-June 2021) regarding PA location choices and maintenance. Mixed-effects models examined the main effects of PA location choice and its interaction with weeks into the pandemic on participants' PA maintenance. RESULTS: On a given week, participants performing PA on roads/sidewalks or at parks/open spaces were 1½ and 2 times as likely to maintain PA levels, respectively. Moreover, after September 2021, weeks when individuals performed PA on roads/sidewalks had a protective effect on PA maintenance. CONCLUSIONS: Performing PA on roads/sidewalks and at parks/open spaces was associated with PA maintenance during the COVID-19 pandemic. PA promotion and intervention efforts for emerging adults during large-scale disruptions to daily life should focus on providing programmed activities in open spaces to maintain their PA levels.
Yun, H. S., S. Zhou, E. Kimani, S. Olafsson, T. K. O'Leary, D. Parmar, J. Hoffman, S. Intille, M. Paasche-Orlow and T. Bickmore (2022). "Techno-spiritual engagement: Mechanisms for improving uptake of mHealth apps designed for church members." Joint Proceedings of the ACM IUI Workshops: 130-138.
Keeping users engaged with mHealth applications is important but difficult to achieve. We describe the development of a smartphone-based application designed to promote health and wellness in church communities, along with mechanisms explicitly designed to maintain engagement. We evaluated religiously tailored techno-spiritual engagement mechanisms, including a prayer posting wall, pastor announcements, an embodied conversational agent for dialogue-based scriptural reflections and health coaching, and tailored push notifications. We conducted a four-week pilot study with 25 participants from two churches, measuring high levels of participant acceptance and satisfaction with all features of the application. Engagement with the app was higher for users considered to be more religious and correlated with the number of notifications received. Our findings demonstrate that our tailored mechanisms can increase engagement with an mHealth app.
Dunton, G. F., A. J. Rothman, A. M. Leventhal and S. S. Intille (2021). "How intensive longitudinal data can stimulate advances in health behavior maintenance theories and interventions." Translational Behavioral Medicine 11(1): 281-286.
Interventions that promote long-term maintenance of behaviors such as exercise, healthy eating, and avoidance of tobacco and excessive alcohol are critical to reduce noncommunicable disease burden. Theories of health behavior maintenance tend to address reactive (i.e., automatic) or reflective (i.e., deliberative) decision-making processes, but rarely both. Progress in this area has been stalled by theories that say little about when, why, where, and how reactive and reflective systems interact to promote or derail a positive health behavior change. In this commentary, we discuss factors influencing the timing and circumstances under which an individual may shift between the two systems such as (a) limited availability of psychological assets, (b) interruption in exposure to established contextual cues, and (c) lack of intrinsic or appetitive motives. To understand the putative factors that regulate the interface between these systems, research methods are needed that are able to capture properties such as (a) fluctuation over short periods of time, (b) change as a function of time, (c) context dependency, (d) implicit and physiological channels, and (e) idiographic phenomenology. These properties are difficult to assess with static, cross-sectional, laboratory-based, or retrospective research methods. We contend that intensive longitudinal data (ILD) collection and analytic strategies such as smartphone and sensor-based real-time activity and location monitoring, ecological momentary assessment (EMA), machine learning, and systems modeling are well-positioned to capture and interpret within-person shifts between reactive and reflective systems underlying behavior maintenance. We conclude with examples of how ILD can accelerate the development of theories and interventions to sustain health behavior over the long term.
Ponnada, A., S. Cooper, Q. Tang, B. Thapa-Chhetry, J. A. Miller, D. John and S. Intille (2021). "Signaligner Pro: A tool to explore and annotate multi-day raw accelerometer data." IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops): 475-480. Student lead author
Human activity recognition using wearable accelerometers can enable in-situ detection of physical activities to support novel human-computer interfaces. Many of the machinelearning-based activity recognition algorithms require multiperson, multi-day, carefully annotated training data with precisely marked start and end times of the activities of interest. To date, there is a dearth of usable tools that enable researchers to conveniently visualize and annotate multiple days of highsampling-rate raw accelerometer data. Thus, we developed Signaligner Pro, an interactive tool to enable researchers to conveniently explore and annotate multi-day high-sampling rate raw accelerometer data. The tool visualizes high-sampling-rate raw data and time-stamped annotations generated by existing activity recognition algorithms and human annotators; the annotations can then be directly modified by the researchers to create their own, improved, annotated datasets. In this paper, we describe the tool's features and implementation that facilitate convenient exploration and annotation of multi-day data and demonstrate its use in generating activity annotations.
Ponnada, A., B. Thapa-Chhetry, J. Manjourides and S. Intille (2021). "Measuring criterion validity of microinteraction ecological momentary assessment (Micro-EMA): Exploratory pilot study with physical activity measurement." JMIR mHealth and Uhealth 9(3): e23391. Student lead author
Background: Ecological momentary assessment (EMA) is an in situ method of gathering self-report on behaviors using mobile devices. In typical phone-based EMAs, participants are prompted repeatedly with multiple-choice questions, often causing participation burden. Alternatively, microinteraction EMA (micro-EMA or μEMA) is a type of EMA where all the self-report prompts are single-question surveys that can be answered using a 1-tap glanceable microinteraction conveniently on a smartwatch. Prior work suggests that μEMA may permit a substantially higher prompting rate than EMA, yielding higher response rates and lower participation burden. This is achieved by ensuring μEMA prompt questions are quick and cognitively simple to answer. However, the validity of participant responses from μEMA self-report has not yet been formally assessed. Objective: In this pilot study, we explored the criterion validity of μEMA self-report on a smartwatch, using physical activity (PA) assessment as an example behavior of interest. Methods: A total of 17 participants answered 72 μEMA prompts each day for 1 week using a custom-built μEMA smartwatch app. At each prompt, they self-reported whether they were doing sedentary, light/standing, moderate/walking, or vigorous activities by tapping on the smartwatch screen. Responses were compared with a research-grade activity monitor worn on the dominant ankle simultaneously (and continuously) measuring PA. Results: Participants had an 87.01% (5226/6006) μEMA completion rate and a 74.00% (5226/7062) compliance rate taking an average of only 5.4 (SD 1.5) seconds to answer a prompt. When comparing μEMA responses with the activity monitor, we observed significantly higher (P\<.001) momentary PA levels on the activity monitor when participants self-reported engaging in moderate+vigorous activities compared with sedentary or light/standing activities. The same comparison did not yield any significant differences in momentary PA levels as recorded by the activity monitor when the μEMA responses were randomly generated (ie, simulating careless taps on the smartwatch). Conclusions: For PA measurement, high-frequency μEMA self-report could be used to capture information that appears consistent with that of a research-grade continuous sensor for sedentary, light, and moderate+vigorous activity, suggesting criterion validity. The preliminary results show that participants were not carelessly answering μEMA prompts by randomly tapping on the smartwatch but were reporting their true behavior at that moment. However, more research is needed to examine the criterion validity of μEMA when measuring vigorous activities.
Yang, C. H., J. P. Maher, A. Ponnada, E. Dzubur, R. Nordgren, S. Intille, D. Hedeker and G. F. Dunton (2021). "An empirical example of analysis using a two-stage modeling approach: Within-subject association of outdoor context and physical activity predicts future daily physical activity levels." Transl Behav Med 11(4): 912-920.
People differ from each other to the extent to which momentary factors, such as context, mood, and cognitions, influence momentary health behaviors. However, statistical models to date are limited in their ability to test whether the association between two momentary variables (i.e., subject-level slopes) predicts a subject-level outcome. This study demonstrates a novel two-stage statistical modeling strategy that is capable of testing whether subject-level slopes between two momentary variables predict subject-level outcomes. An empirical case study application is presented to examine whether there are differences in momentary moderate-to-vigorous physical activity (MVPA) levels between the outdoor and indoor context in adults and whether these momentary differences predict mean daily MVPA levels 6 months later. One hundred and eight adults from a multiwave longitudinal study provided 4 days of ecological momentary assessment (during baseline) and accelerometry data (both at baseline and 6 month follow-up). Multilevel data were analyzed using an open-source program (MixWILD) to test whether momentary strength between outdoor context and MVPA during baseline was associated with average daily MVPA levels measured 6 months later. During baseline, momentary MVPA levels were higher in outdoor contexts as compared to indoor contexts (b = 0.07, p \< .001). Participants who had more momentary MVPA when outdoors (vs. indoors) during baseline (i.e., a greater subject-level slope) had higher daily MVPA at the 6 month follow-up (b = 0.09, p \< .05). This empirical example shows that the subject-level momentary association between specific context (i.e., outdoors) and health behavior (i.e., physical activity) may contribute to overall engagement in that behavior in the future. The demonstrated two-stage modeling approach has extensive applications in behavioral medicine to analyze intensive longitudinal data collected from wearable sensors and mobile devices.
Canori, A., A. M. Amiri, B. Thapa-Chhetry, M. A. Finley, M. Schmidt-Read, M. R. Lamboy, S. S. Intille and S. V. Hiremath (2020). "Relationship between pain, fatigue, and physical activity levels during a technology-based physical activity intervention." The Journal of Spinal Cord Medicine 44(4): 549-556. Student lead author
Objective: The majority of individuals with spinal cord injury (SCI) experience chronic pain. Chronic pain can be difficult to manage because of variability in the underlying pain mechanisms. More insight regarding the relationship between pain and physical activity (PA) is necessary to understand pain responses during PA. The objective of this study is to explore possible relationships between PA levels and secondary conditions including pain and fatigue.Design: Prospective cohort analysis of a pilot study.Setting: Community.Participants: Twenty individuals with SCI took part in the study, and sixteen completed the study.Interventions: Mobile-health (mHealth) based PA intervention for two-months during the three-month study.Outcome measures: Chronic Pain Grade Scale (CPGS) questionnaire, The Wheelchair User?s Shoulder Pain Index (WUSPI), Fatigue Severity Scale (FSS), and PA levels measured by the mHealth system.Results: A positive linear relationship was found between light-intensity PA and task-specific pain. However, the relationship between moderate-intensity PA and pain interference was best represented by a curvilinear relationship (polynomial regression of second order). Light-intensity PA showed positive, linear correlation with fatigue at baseline. Moderate-intensity PA was not associated with fatigue during any phase of the study.Conclusion: Our results indicated that PA was associated with chronic pain, and the relationship differed based on intensity and amount of PA performed. Further research is necessary to refine PA recommendations for individuals with SCI who experience chronic pain.Trial registration: ClinicalTrials.gov identifier: NCT03773692.
Dzubur, E., A. Ponnada, R. Nordgren, C. H. Yang, S. Intille, G. Dunton and D. Hedeker (2020). "MixWILD: A program for examining the effects of variance and slope of time-varying variables in intensive longitudinal data." Behavior Research Methods 52(4): 1403-1427. Students lead authors
The use of intensive sampling methods, such as ecological momentary assessment (EMA), is increasingly prominent in medical research. However, inferences from such data are often limited to the subject-specific mean of the outcome and between-subject variance (i.e., random intercept), despite the capability to examine within-subject variance (i.e., random scale) and associations between covariates and subject-specific mean (i.e., random slope). MixWILD (Mixed model analysis With Intensive Longitudinal Data) is statistical software that tests the effects of subject-level parameters (variance and slope) of time-varying variables, specifically in the context of studies using intensive sampling methods, such as ecological momentary assessment. MixWILD combines estimation of a stage 1 mixed-effects location-scale (MELS) model, including estimation of the subject-specific random effects, with a subsequent stage 2 linear or binary/ordinal logistic regression in which values sampled from each subject's random effect distributions can be used as regressors (and then the results are aggregated across replications). Computations within MixWILD were written in FORTRAN and use maximum likelihood estimation, utilizing both the expectation-maximization (EM) algorithm and a Newton-Raphson solution. The mean and variance of each individual's random effects used in the sampling are estimated using empirical Bayes equations. This manuscript details the underlying procedures and provides examples illustrating standalone usage and features of MixWILD and its GUI. MixWILD is generalizable to a variety of data collection strategies (i.e., EMA, sensors) as a robust and reproducible method to test predictors of variability in level 1 outcomes and the associations between subject-level parameters (variances and slopes) and level 2 outcomes.
Madden, D. R., C. Nok Lam, B. Redline, E. Dzubur, H. Rhoades, S. S. Intille, G. F. Dunton and B. Henwood (2020). "Real-time data collection to examine relations between physical activity and affect in adults with mental illness." Journal of Sport & Exercise Psychology 42(5): 386-393.
Adults with serious mental illness engage in limited physical activity, which contributes to significant health disparities. This study explored the use of both ecological momentary assessments (EMAs) and activity trackers in adults with serious mental illness to examine the bidirectional relationship between activity and affect with multilevel modeling. Affective states were assessed up to seven times per day using EMA across 4 days. The participants (n = 20) were equipped with a waist-worn accelerometer to measure moderate to vigorous physical activity. The participants had a mean EMA compliance rate of 88.3%, and over 90% of completed EMAs were matched with 30-min windows of accelerometer wear. The participants who reported more positive affect than others had a higher probability of engaging in moderate to vigorous physical activity. Engaging in more moderate to vigorous physical activity than one's usual was associated with more negative affect. This study begins to address the effect of momentary mood on physical activity in a population of adults that is typically difficult to reach.
Tang, Q., D. John, B. Thapa-Chhetry, D. J. Arguello and S. Intille (2020). "Posture and physical activity detection: Impact of number of sensors and feature type." Medicine & Science in Sports & Exercise 52(8): 1834-1845. Student lead author
Studies using wearable sensors to measure posture, physical activity (PA), and sedentary behavior typically use a single sensor worn on the ankle, thigh, wrist, or hip. Although the use of single sensors may be convenient, using multiple sensors is becoming more practical as sensors miniaturize. Purpose: We evaluated the effect of single-site versus multisite motion sensing at seven body locations (both ankles, wrists, hips, and dominant thigh) on the detection of physical behavior recognition using a machine learning algorithm. We also explored the effect of using orientation versus orientation-invariant features on performance. Methods: Performance (F1 score) of PA and posture recognition was evaluated using leave-one-subject-out cross-validation on a 42-participant data set containing 22 physical activities with three postures (lying, sitting, and upright). Results: Posture and PA recognition models using two sensors had higher F1 scores (posture, 0.89 ± 0.06; PA, 0.53 ± 0.08) than did models using a single sensor (posture, 0.78 ± 0.11; PA, 0.43 ± 0.03). Models using two nonwrist sensors for posture recognition (F1 score, 0.93 ± 0.03) outperformed two-sensor models including one or two wrist sensors (F1 score, 0.85 ± 0.06). However, two-sensor models for PA recognition with at least one wrist sensor (F1 score, 0.60 ± 0.05) outperformed other two-sensor models (F1 score, 0.47 ± 0.02). Both posture and PA recognition F1 scores improved with more sensors (up to seven; 0.99 for posture and 0.70 for PA), but with diminishing performance returns. Models performed best when including orientation-based features. Conclusions: Researchers measuring posture should consider multisite sensing using at least two nonwrist sensors, and researchers measuring PA should consider multisite sensing using at least one wrist sensor and one nonwrist sensor. Including orientation-based features improved both posture and PA recognition.
Tang, Q., A. Ponnada and S. Intille (2020). Towards personal hand hygiene detection in free-living using wearable devices. Machine Learning for Mobile Health NeurIPS Workshop. Student lead author
The COVID-19 outbreak demonstrates the need for measurement of hand hygiene behaviors such as handwashing and face touching to prevent the spread of infectious diseases. Wearable technologies and machine-learning-based algorithms can be used to automatically detect these behaviors. In this work, we demonstrate a recurrent neural network with a set of local-extrema-based features for detecting hand hygiene behaviors (handwashing and face touching activities simultaneously) using data from inertial sensors (i.e., accelerometer, magnetometer, and gyroscope) on the wrist(s). The training and validation dataset were gathered from ten individuals; each person provided 60 min of data (sampled at 100 Hz) while performing 12 steps of handwashing, 8 variations of face touching, and 7 variations of other face-to-head gestures across six sessions. With 10 min of person-specific training data, the real-time algorithm achieved its best performance (F1-score of 0.88 for handwashing steps and 0.80 for face touching) using leave-one-session-out validation. We also describe a pilot evaluation on six-hour, free-living waking-day datasets of two participants annotated via front-facing video.
Henwood, B. F., B. Redline, E. Dzubur, D. R. Madden, H. Rhoades, G. F. Dunton, E. Rice, S. Semborski, Q. Tang and S. S. Intille (2019). "Investigating health risk environments in housing programs for young adults: Protocol for a geographically explicit ecological momentary assessment study." JMIR Res Protoc 8(1): e12112.
Background: Young adults who experience homelessness are exposed to environments that contribute to risk behavior. However, few studies have examined how access to housing may affect the health risk behaviors of young adults experiencing homelessness. Objective: This paper describes the Log My Life study that uses an innovative, mixed-methods approach based on geographically explicit ecological momentary assessment (EMA) through cell phone technology to understand the risk environment of young adults who have either enrolled in housing programs or are currently homeless. Methods: For the quantitative arm, study participants age 18-27 respond to momentary surveys via a smartphone app that collects geospatial information repeatedly during a 1-week period. Both EMAs (up to 8 per day) and daily diaries are prompted to explore within-day and daily variations in emotional affect, context, and health risk behavior, while also capturing infrequent risk behaviors such as sex in exchange for goods or services. For the qualitative arm, a purposive subsample of participants who indicated engaging in risky behaviors are asked to complete an in-depth qualitative interview using an interactive, personalized geospatial map rendering of EMA responses. Results: Recruitment began in June of 2017. To date, 170 participants enrolled in the study. Compliance with EMA and daily diary surveys was generally high. In-depth qualitative follow-ups have been conducted with 15 participants. We expect to recruit 50 additional participants and complete analyses by September of 2019. Conclusions: Mixing the quantitative and qualitative arms in this study will provide a more complete understanding of differences in risk environments between homeless and housed young adults. Furthermore, this approach can improve recall bias and enhance ecological validity. International Registered Report Identifier (IRRID): DERR1-10.2196/12112
Hiremath, S. V., A. M. Amiri, B. Thapa-Chhetry, G. Snethen, M. Schmidt-Read, M. Ramos-Lamboy, D. L. Coffman and S. S. Intille (2019). "Mobile health-based physical activity intervention for individuals with spinal cord injury in the community: A pilot study." PLoS ONE 14(10): e0223762.
Low levels of physical activity (PA) and high levels of sedentary behavior in individuals with spinal cord injury (SCI) have been associated with secondary conditions such as pain, fatigue, weight gain, and deconditioning. One strategy for promoting regular PA is to provide people with an accurate estimate of everyday PA level. The objective of this research was to use a mobile health-based PA measurement system to track PA levels of individuals with SCI in the community and provide them with a behavior-sensitive, just-in-time-adaptive intervention (JITAI) to improve their PA levels. The first, second, and third phases of the study, each with a duration of one month, involved collecting baseline PA levels, providing near-real-time feedback on PA level (PA Feedback), and providing PA Feedback with JITAI, respectively. PA levels in terms of energy expenditure in kilocalories, and minutes of light- and moderate- or vigorous-intensity PA were assessed by an activity monitor during the study. Twenty participants with SCI took part in this research study with a mean (SD) age of 39.4 (12.8) years and 12.4 (12.5) years since injury. Sixteen participants completed the study. Sixteen were male, 16 had paraplegia, and 12 had complete injury. Within-participant comparisons indicated that only two participants had higher energy expenditure (\>10%) or lower energy expenditure (\<-10%) during PA Feedback with JITAI compared to the baseline. However, eleven participants (69.0%) had higher light- and/or moderate-intensity PA during PA Feedback with JITAI compared to the baseline. To our knowledge, this is the first study to test a PA JITAI for individuals with SCI that responds automatically to monitored PA levels. The results of this pilot study suggest that a sensor-enabled mobile JITAI has potential to improve PA levels of individuals with SCI. Future research should investigate the efficacy of JITAI through a clinical trial.
John, D., Q. Tang, F. Albinali and S. Intille (2019). "An open-source monitor-independent movement summary for accelerometer data processing." Journal for the Measurement of Physical Behaviour 2(4): 268-281.
Background: Physical behavior researchers using motion sensors often use acceleration summaries to visualize, clean, and interpret data. Such output is dependent on device specifications (e.g., dynamic range, sampling rate) and/or are proprietary, which invalidate cross-study comparison of findings when using different devices. This limits flexibility in selecting devices to measure physical activity, sedentary behavior, and sleep. Purpose: Develop an open-source, universal acceleration summary metric that accounts for discrepancies in raw data among research and consumer devices. Methods: We used signal processing techniques to generate a Monitor-Independent Movement Summary unit (MIMS-unit) optimized to capture normal human motion. Methodological steps included raw signal harmonization to eliminate inter-device variability (e.g., dynamic g-range, sampling rate), bandpass filtering (0.2--5.0 Hz) to eliminate non-human movement, and signal aggregation to reduce data to simplify visualization and summarization. We examined the consistency of MIMS-units using orbital shaker testing on eight accelerometers with varying dynamic range (±2 to ±8 g) and sampling rates (20--100 Hz), and human data (N = 60) from an ActiGraph GT9X. Results: During shaker testing, MIMS-units yielded lower between-device coefficient of variations than proprietary ActiGraph and ENMO acceleration summaries. Unlike the widely used ActiGraph activity counts, MIMS-units were sensitive in detecting subtle wrist movements during sedentary behaviors. Conclusions: Open-source MIMS-units may provide a means to summarize high-resolution raw data in a device-independent manner, thereby increasing standardization of data cleaning and analytical procedures to estimate selected attributes of physical behavior across studies.
Ponnada, A., S. Cooper, B. Thapa-Chhetry, J. A. Miller, D. John and S. Intille (2019). "Designing videogames to crowdsource accelerometer data annotation for activity recognition research." Proceedings of the Annual Symposium on Computer-Human Interaction in Play: 135-147. Student lead author
Human activity recognition using wearable accelerometers can enable in-situ detection of physical activities to support novel human-computer interfaces and interventions. However, developing valid algorithms that use accelerometer data to detect everyday activities often requires large amounts of training datasets, precisely labeled with the start and end times of the activities of interest. Acquiring annotated data is challenging and timeconsuming. Applied games, such as human computation games (HCGs) have been used to annotate images, sounds, and videos to support advances in machine learning using the collective effort of "non-expert game players." However, their potential to annotate accelerometer data has not been formally explored. In this paper, we present two proof-ofconcept, web-based HCGs aimed at enabling game players to annotate accelerometer data. Using results from pilot studies with Amazon Mechanical Turk players, we discuss key challenges, opportunities, and, more generally, the potential of using applied videogames for annotating raw accelerometer data to support activity recognition research.
Arguello, D., K. Andersen, A. Morton, P. S. Freedson, S. S. Intille and D. John (2018). "Validity of proximity sensor-based wear-time detection using the ActiGraph GT9X." Journal of Sports Sciences 36(13): 1502-1507. Student lead author
Purpose: To investigate the performance of proximity-sensor-based wear-time detection using the GT9X under laboratory and free-living settings. Methods: Fifty-two volunteers (23.2±3.8 y; 23.2±3.7 kg/m2) participated in either a laboratory or a freeliving protocol. Participants in the lab wore and removed a wrist-worn GT9X on 3-5 occasions during a 3-hour directly observed activity protocol. The 2-day free-living protocol used an independent temperature sensor and self-report as the reference to determine if a wrist and hip-worn GT9X accurately determines wear (i.e., sensitivity) and non-wear (i.e., specificity). Free-living estimates of wear/non-wear were also compared to the Troiano 2007 and Choi 2012 wear/non-wear algorithms. Results: In lab, sensitivity and specificity of the wrist-worn GT9X in detecting total minutes of wearon and off was 93% and 49%, respectively. The GT9X detected wear-off more often than wear-on, but with a greater margin of error (4.8±11.6 vs. 1.4±1.4 min). In the freeliving protocol, wrist and hip-worn GT9X's yielded sensitivity and specificity of 72 and 90% and 84 and 92%, respectively. GT9X estimations had inferior sensitivity but superior specificity to Troiano 2007 and Choi 2012 algorithms. Conclusions: Due to inaccuracies, it may not be advisable to singularly use the current proximity-sensorbased wear-time detection method to detect wear-time.
Dunton, G. F., A. M. Leventhal, A. J. Rothman and S. S. Intille (2018). "Affective response during physical activity: Within-subject differences across phases of behavior change." Health Psychology 37(10): 915-923.
Objective: Affective response during physical activity may be a key factor reinforcing future behavior. However, little is known about how affective responses during physical activity may differ across phases of behavior change. This study used real-time Ecological Momentary Assessment (EMA) to examine within-subject differences in affective response during physical activity in daily life as individuals transitioned across phases of behavior change. Method: A sample of 115 adults (M = 41.0 years, 74% female) participated in an intensive longitudinal study with measurement bursts at 0, 6, and 12-months. Each burst consisted of 8 randomly-prompted EMA occasions per day across 4 days. EMA self-report items assessed current activity level (i.e., physical activity or nonphysical activity), and positive and negative affect. Questionnaires measured phase of behavior change (e.g., preaction \[no regular physical activity\], action \[regular physical activity \<6 months\], and maintenance \[regular physical activity ≥6 months\]) at each burst. Three-level (Level-1 = occasion, Level-2 = burst, Level-3 = person) linear regression models tested Phase of Change (Level-2, within-subject) × Physical Activity Level (Level-1, within-subject) interactions controlling for day of week, time of day, and sex. Results: Positive affective response during physical activity (vs. nonphysical activity) was higher when individuals were in preaction phases (vs. action). Negative affective response during physical activity (vs. nonphysical activity) was lower when individuals were in the maintenance phase (vs. action). Conclusions: Long-term maintenance of physical activity may be particularly challenging, given the lack of positive reinforcement that is thought to be needed to sustain behavior
Dzubur, E., J. Huh, J. P. Maher, S. S. Intille and G. F. Dunton (2018). "Response patterns and intra-dyadic factors related to compliance with ecological momentary assessment among mothers and children." Translational Behavioral Medicine 8(2): 233-242.
Ecological momentary assessment (EMA) is a real-time sampling strategy that may address limitations in health research, such as the inability to examine how processes unfold on a daily basis. However, EMA studies are prone to limited data availability due to difficulties in implementing sophisticated protocols and systematic non-compliance with prompts, resulting in biased estimates and limited statistical power. The objectives of this study were to describe the availability of data, to examine response patterns, and to analyze factors related to EMA prompt compliance in a dyadic EMA study with mothers and children. Participants (N = 404) each received up to eight EMA prompts (i.e., audible pings) per day for a total of 7 days. Each EMA survey consisted of items assessing affect, perceived stress, and social context. Participants responded to approximately 80% (range: 3.4%-100%) of prompted EMA surveys, and completed 92.6% of surveys once started. Mothers and children identifying as Hispanic, as well as mothers in lower-income households, were less likely to comply with any given EMA prompt. Participant dyads were more likely to comply with prompts when they were together. Understanding factors related to systematic EMA prompt non-compliance is an important step to reduce the likelihood of biased estimates and improve statistical power. Socioeconomic factors may impede mothers' compliance with EMA protocols. Furthermore, mothers' presence and involvement may enhance children's compliance with EMA protocols.
Lin, P.-H., S. Grambow, S. Intille, J. Gallis, T. Lazenka, H. Bosworth, C. Voils, G. Bennett, B. Batch, J. Allen, L. Corsino, C. Tyson and L. Svetkey (2018). "The association between engagement and weight loss through personal coaching and cell phone interventions in young adults: Randomized controlled trial." JMIR mHealth and uHealth 6(10): e10471.
Background: Understanding how engagement in mobile health (mHealth) weight loss interventions relates to weight change may help develop effective intervention strategies. Objective: This study aims to examine the (1) patterns of participant engagement overall and with key intervention components within each intervention arm in the Cell Phone Intervention For You (CITY) trial; (2) associations of engagement with weight change; and (3) participant characteristics related to engagement. Methods: The CITY trial tested two 24-month weight loss interventions. One was delivered with a smartphone app (cell phone) containing 24 components (weight tracking, etc) and included prompting by the app in predetermined frequency and forms. The other was delivered by a coach via monthly calls (personal coaching) supplemented with limited app components (18 overall) and without any prompting by the app. Engagement was assessed by calculating the percentage of days each app component was used and the frequency of use. Engagement was also examined across 4 weight change categories: gained (≥2%), stable (±2%), mild loss (≥2% to \<5%), and greater loss (≥5%). Results: Data from 122 cell phone and 120 personal coaching participants were analyzed. Use of the app was the highest during month 1 for both arms; thereafter, use dropped substantially and continuously until the study end. During the first 6 months, the mean percentage of days that any app component was used was higher for the cell phone arm (74.2%, SD 20.1) than for the personal coaching arm (48.9%, SD 22.4). The cell phone arm used the apps an average of 5.3 times/day (SD 3.1), whereas the personal coaching participants used them 1.7 times/day (SD 1.2). Similarly, the former self-weighed more than the latter (57.1% days, SD 23.7 vs 32.9% days, SD 23.3). Furthermore, the percentage of days any app component was used, number of app uses per day, and percentage of days self-weighed all showed significant differences across the 4 weight categories for both arms. Pearson correlation showed a negative association between weight change and the percentage of days any app component was used (cell phone: r=−.213; personal coaching: r=−.319), number of apps use per day (cell phone: r=−.264; personal coaching: r=−.308), and percentage of days self-weighed (cell phone: r=−.297; personal coaching: r=−.354). None of the characteristics examined, including age, gender, race, education, income, energy expenditure, diet quality, and hypertension status, appeared to be related to engagement. Conclusions: Engagement in CITY intervention was associated with weight loss during the first 6 months. Nevertheless, engagement dropped substantially early on for most intervention components. Prompting may be helpful initially. More flexible and less intrusive prompting strategies may be needed during different stages of an intervention to increase or sustain engagement. Future studies should explore the motivations for engagement and nonengagement to determine meaningful levels of engagement required for effective intervention.
Maher, J. P., J. Huh, S. Intille, D. Hedeker and G. F. Dunton (2018). "Greater variability in daily physical activity is associated with poorer mental health profiles among obese adults." Mental Health and Physical Activity 14: 74-81.
Research is inconclusive about whether physical activity (PA) should be performed every day or performed less frequently but in longer bouts to obtain mental health benefits. The current study examined the extent to which day-to-day variability in PA is associated with adults' mental health, and if this association differed by Body Mass Index (BMI). Adults ( = 116) completed three waves of data collection (each lasting 4 days) during which participants completed a questionnaire assessing mental health (life satisfaction, depressive symptoms, perceived stress), wore a waist accelerometer, and had height and weight measured. This study employed a novel two-stage data analysis approach using the standalone program MIXWILD. The first-stage model partitioned mean level as well as between-subject and within-subject variances in daily PA by estimating a random location (subject-level mean) and a random scale (subject-level variability) for daily PA. In the second-stage, these random subject effects for daily PA along with their interactions with BMI were used as predictors for subject-level mental health outcomes. Associations between subject-level variability in daily PA and mental health outcomes significantly differed depending on adults' BMI (life satisfaction: β = −0.05, p \< 0.05; depressive symptoms: β = 0.03, p \< 0.05; perceived stress: β = 0.04, p \< 0.01). Greater day-to-day variability in PA was associated with poorer mental health in adults with higher BMI values as compared to adults with lower BMI. For individuals with high BMI values, inconsistent activity patterns may have consequences that diminish mental health. Strategies that promote consistency in daily PA may be useful for individuals with high BMI to enhance mental health.
Mannini, A. and S. Intille (2018). "Classifier personalization for activity recognition using wrist accelerometers." IEEE Journal of Biomedical and Health Informatics 23(4): 1585-1594. Student lead author
Inter-subject variability in accelerometer-based activity recognition may significantly affect classification accuracy, limiting a reliable extension of methods to new users. In this work we propose an approach for personalizing classification rules to a single person. We demonstrate that the method improves activity detection from wrist-worn accelerometer data on a four-class recognition problem of interest to the exercise science community, where classes are ambulation, cycling, sedentary, and other. We extend a previously published activity classification method based on support vector machines so that it estimates classification uncertainty. Uncertainty is used to drive data label requests from the user, and the resulting label information is used to update the classifier. Two different datasets - one from 33 adults with 26 activity types, and another from 20 youth with 23 activity types - were used to evaluate the method using leave-one-subject-out and leave-one-group-out cross validation. The new method improved overall recognition accuracy up to 11% on average, with some large person-specific improvements (ranging from -2% to +36%). The proposed method is suitable for online implementation supporting real-time recognition systems.
Millstein, R. A., N. M. Oreskovic, L. M. Quintiliani, P. James and S. Intille (2018). "The need for local, multidisciplinary collaborations to promote advances in physical activity research and policy change: The creation of the Boston Physical Activity Resource Collaborative (BPARC)." Journal of Physical Activity Research 3(2): 74-77.
This commentary describes the development, vision, and initial progress of the newly-founded Boston Physical Activity Resource Collaborative (BPARC). Our aims are to move the field of physical activity forward, with broader dissemination and translation, by creating a local Boston and Massachusetts hub for researchers, practitioners, advocates, and policymakers. Participants come from multiple academic and medical centers, local advocacy groups, and government agencies, all of whom are working on components of physical activity promotion. We have had initial success in collaborating on study design, methodology, and grant applications. Future endeavors aim to produce streamlined methods and products with maximal impact for the field of physical activity research, policy, and practice.
Spruijt-Metz, D., C. K. F. Wen, B. M. Bell, S. Intille, J. S. Huang and T. Baranowski (2018). "Advances and controversies in diet and physical activity measurement in youth." American Journal of Preventative Medicine 55(4): e81-e91.
Technological advancements in the past decades have improved dietary intake and physical activity measurements. This report reviews current developments in dietary intake and physical activity assessment in youth. Dietary intake assessment has relied predominantly on self-report or image-based methods to measure key aspects of dietary intake (e.g., food types, portion size, eating occasion), which are prone to notable methodologic (e.g., recall bias) and logistic (e.g., participant and researcher burden) challenges. Although there have been improvements in automatic eating detection, artificial intelligence, and sensor-based technologies, participant input is often needed to verify food categories and portions. Current physical activity assessment methods, including self-report, direct observation, and wearable devices, provide researchers with reliable estimations for energy expenditure and bodily movement. Recent developments in algorithms that incorporate signals from multiple sensors and technology-augmented self-reporting methods have shown preliminary efficacy in measuring specific types of activity patterns and relevant contextual information. However, challenges in detecting resistance (e.g., in resistance training, weight lifting), prolonged physical activity monitoring, and algorithm (non)equivalence remain to be addressed. In summary, although dietary intake assessment methods have yet to achieve the same validity and reliability as physical activity measurement, recent developments in wearable technologies in both arenas have the potential to improve current assessment methods.
Jones, M., A. Taylor, Y. Liao, S. S. Intille and G. F. Dunton (2017). "Real-time subjective assessment of psychological stress: Associations with objectively-measured physical activity levels." Psychology of Sport and Exercise 31: 79-87.
Psychosocial stress may be a factor in the link between physical activity and obesity. This study examines how the daily experience of psychosocial stress influences physical activity levels and weight status in adults. This study reports temporally ordered relationships between sedentary, light, and moderate-to-vigorous physical activity levels and real-time reports of subjective psychosocial stress levels. Adults (n = 105) wore an accelerometer and participated in an ecological momentary assessment (EMA) of stress by answering prompts on a mobile phone several times per day over 4 days. Subjective stress was negatively related to sedentary activity in the minutes immediately preceding and immediately following an EMA prompt. Light activity was positively associated with a subsequent EMA report of higher stress, but there were no observed associations between stress and moderate-to-vigorous activity. Real-time stress reports and accelerometer readings for the same 4-day period showed no association. Nor were there associations between real-time stress reports and weight status. •Subjective psychosocial stress measured in real-time.•Lower sedentary activity was related to higher subjective stress in real-time.•Higher light activity was associated with higher subjective stress in real-time.•Real-time stress measurement identifies relationships that traditional approaches may miss.
Maher, J. P., R. E. Rhodes, E. Dzubur, J. Huh, S. Intille and G. F. Dunton (2017). "Momentary assessment of physical activity intention-behavior coupling in adults." Translational Behavioral Medicine 7(4): 709-718.
Research attempting to elucidate physical activity (PA) intention-behavior relations has focused on differences in long-term behavior forecasting between people. However, regular PA requires a repeated performance on a daily or within-daily basis. An empirical case study application is presented using intensive longitudinal data from a study of PA in adults to (a) describe the extent to which short-term intention-behavior coupling occurs and (b) explore time-varying predictors of intention formation and short-term intention-behavior coupling. Adults (n = 116) participated in three 4-day waves of ecological momentary assessment (EMA). Each day, participants received EMA questionnaires assessing short-term PA intentions and wore accelerometers to assess whether they engaged in \>/=10 min of moderate-to-vigorous physical activity (MVPA) in the 3-hour period after each EMA prompt. Concurrent affective states and contexts were also assessed through EMA. Participants reported having short-term intentions to engage in PA in 41% of EMA prompts. However, participants only engaged in \>/=10 min of MVPA following 16% of the prompts that short-term PA intentions were reported indicating an intention-behavior gap of 84%. Odds of intentions followed by PA were greater on occasions when individuals reported higher levels of positive affect than was typical for them. This study is the first to take an EMA approach to describe short-term intention-behavior coupling in adults. Results suggest that adults have difficulty translating intentions into behavior at the momentary level, more so than over longer timescales, and that positive affect may be a key to successfully translating intentions into behavior.
Mannini, A., M. Rosenberger, W. L. Haskell, A. M. Sabatini and S. S. Intille (2017). "Activity recognition in youth using single accelerometer placed at wrist or ankle." Medicine & Science in Sports & Exercise 49(4): 801-812. Student lead author
PURPOSE: State-of-the-art methods for recognizing human activity using raw data from body-worn accelerometers have primarily been validated with data collected from adults. This study applies a previously available method for activity classification using wrist or ankle accelerometer to data sets collected from both adults and youth. METHODS: An algorithm for detecting activity from wrist-worn accelerometers, originally developed using data from 33 adults, is tested on a data set of 20 youth (age, 13 +/- 1.3 yr). The algorithm is also extended by adding new features required to improve performance on the youth data set. Subsequent tests on both the adult and youth data were performed using crossed tests (training on one group and testing on the other) and leave-one-subject-out cross-validation. RESULTS: The new feature set improved overall recognition using wrist data by 2.3% for adults and 5.1% for youth. Leave-one-subject-out cross-validation accuracy performance was 87.0% (wrist) and 94.8% (ankle) for adults, and 91.0% (wrist) and 92.4% (ankle) for youth. Merging the two data sets, overall accuracy was 88.5% (wrist) and 91.6% (ankle). CONCLUSIONS: Previously available methodological approaches for activity classification in adults can be extended to youth data. Including youth data in the training phase and using features designed to capture information on the activity fragmentation of young participants allows a better fit of the methodological framework to the characteristics of activity in youth, improving its overall performance. The proposed algorithm differentiates ambulation from sedentary activities that involve gesturing in wrist data, such as that being collected in large surveillance studies.
Ponnada, A., C. Haynes, D. Maniar, J. Manjourides and S. Intille (2017). "Microinteraction ecological momentary assessment response rates: Effect of microinteractions or the smartwatch?" Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1(3): 1-16. Student lead author
Mobile-based ecological-momentary-assessment (EMA) is an in-situ measurement methodology where an electronic device prompts a person to answer questions of research interest. EMA has a key limitation: interruption burden. Microinteraction-EMA(µEMA) may reduce burden without sacrificing high temporal density of measurement. In µEMA, all EMA prompts can be answered with 'at a glance' microinteractions. In a prior 4-week pilot study comparing standard EMA delivered on a phone (phone-EMA) vs. µEMA delivered on a smartwatch (watch-µEMA), watch-µEMA demonstrated higher response rates and lower perceived burden than phone-EMA, even when the watch-µEMA interruption rate was 8 times more than phone-EMA. A new 4-week dataset was gathered on smartwatch-based EMA (i.e., watch-EMA with 6 back-to-back, multiple-choice questions on a watch) to compare whether the high response rates of watch-µEMA previously observed were a result of using microinteractions, or due to the novelty and accessibility of the smartwatch. No statistically significant differences in compliance, completion, and first-prompt response rates were observed between phone-EMA and watch-EMA. However, watch-µEMA response rates were significantly higher than watch-EMA. This pilot suggests that (1) the high compliance and low burden previously observed in watch-µEMA is likely due to the microinteraction question technique, not simply the use of the watch versus the phone, and that (2) compliance with traditional EMA (with long surveys) may not improve simply by moving survey delivery from the phone to a smartwatch.
Spilsbury, J. C., S. R. Patel, N. Morris, A. Ehyaei and S. S. Intille (2017). "Household chaos and sleep-disturbing behavior of family members: Results of a pilot study of African American early adolescents." Sleep Health 3(2): 84-89.
AbstractBackground Although disorganized, chaotic households have been linked to poorer sleep outcomes, how household chaos actually manifests itself in the behaviors of others around the bedtime of a child or adolescent is not well understood. Objective To determine whether household chaos was associated with specific, nightly sleep-disturbing activities of adolescents' family members. Design Longitudinal study. Participants Twenty-six African American or multiethnic early adolescent (ages 11-12 years) and parent dyads, recruited from local schools and social-service agencies in greater Cleveland, OH. Measurements Over 14 days, each night at bedtime, adolescents identified family-member activities keeping them awake or making it difficult to sleep by using a smart phone--administered survey. Household organization was assessed via parent-completed, validated instruments. A generalized linear mixed model examined associations between each activity and household-organization measures. Results Adjusted for the effect of school being in session the next day, an increasingly chaotic household was associated with increased odds of household members disturbing adolescents' efforts to fall asleep by watching TV/listening to music (odds ratio \[OR\] = 1.8, 95% confidence interval \[CI\] = 1.2-3.2), phoning/texting (OR = 1.7, 95% CI =1.2-2.9), or having friends/relatives over visiting at the home (OR = 1.6, 95% CI =1.0-3.0). Conversely, a more chaotic household was associated with decreased odds of adolescents reporting that "nothing" was keeping them awake or making it more difficult to sleep (OR = 0.6, 95% CI =0.4-0.8). Enforced sleep rules were inconsistently associated with sleep-disturbing behaviors. Conclusion Improving early-adolescent sleep may benefit from considering the nighttime behavior of all household members and encouraging families to see that improving early-adolescent sleep requires the household's participation.
Dunton, G. F., E. Dzubur and S. S. Intille (2016). "Feasibility and performance test of a real-time sensor-informed context-sensitive ecological momentary assessment to capture physical activity." Journal of Medical Internet Research 18(6): e106.
Background: Objective physical activity monitors (eg, accelerometers) have high rates of nonwear and do not provide contextual information about behavior. Objective: This study tested performance and value of a mobile phone app that combined objective and real-time self-report methods to measure physical activity using sensor-informed context-sensitive ecological momentary assessment (CS-EMA). Methods: The app was programmed to prompt CS-EMA surveys immediately after 3 types of events detected by the mobile phone's built-in motion sensor: (1) Activity (ie, mobile phone movement), (2) No-Activity (ie, mobile phone nonmovement), and (3) No-Data (ie, mobile phone or app powered off). In addition, the app triggered random (ie, signal-contingent) ecological momentary assessment (R-EMA) prompts (up to 7 per day). A sample of 39 ethnically diverse high school students in the United States (aged 14-18, 54% female) tested the app over 14 continuous days during nonschool time. Both CS-EMA and R-EMA prompts assessed activity type (eg, reading or doing homework, eating or drinking, sports or exercising) and contextual characteristics of the activity (eg, location, social company, purpose). Activity was also measured with a waist-worn Actigraph accelerometer. Results: The average CS-EMA + R-EMA prompt compliance and survey completion rates were 80.5% and 98.5%, respectively. More moderate-to-vigorous intensity physical activity was recorded by the waist-worn accelerometer in the 30 minutes before CS-EMA activity prompts (M=5.84 minutes) than CS-EMA No-Activity (M=1.11 minutes) and CS-EMA No-Data (M=0.76 minute) prompts (P's\<.001). Participants were almost 5 times as likely to report going somewhere (ie, active or motorized transit) in the 30 minutes before CS-EMA Activity than R-EMA prompts (odds ratio=4.91, 95% confidence interval=2.16-11.12). Conclusions: Mobile phone apps using motion sensor-informed CS-EMA are acceptable among high school students and may be used to augment objective physical activity data collected from traditional waist-worn accelerometers.
Hiremath, S. V., S. S. Intille, A. Kelleher, R. A. Cooper and D. Ding (2016). "Estimation of energy expenditure for wheelchair users using a physical activity monitoring system." Archives of Physical Medicine and Rehabilitation 97(7): 1146-1153.
Objective: To develop and evaluate energy expenditure (EE) estimation models for a physical activity monitoring system (PAMS) in manual wheelchair users with spinal cord injury (SCI). Design: Cross-sectional study. Setting: University-based laboratory environment, a semistructured environment at the National Veterans Wheelchair Games, and the participants' home environments. Participants: Volunteer sample of manual wheelchair users with SCI (N=45). Intervention: Participants were asked to perform 10 physical activities (PAs) of various intensities from a list. The PAMS consists of a gyroscope-based wheel rotation monitor (G-WRM) and an accelerometer device worn on the upper arm or on the wrist. Criterion EE using a portable metabolic cart and raw sensor data from PAMS were collected during each of these activities. Main outcome measures: Estimated EE using custom models for manual wheelchair users based on either the G-WRM and arm accelerometer (PAMS-Arm) or the G-WRM and wrist accelerometer (PAMS-Wrist). Results: EE estimation performance for the PAMS-Arm (average error ± SD: -9.82%±37.03%) and PAMS-Wrist (-5.65%±32.61%) on the validation dataset indicated that both PAMS-Arm and PAMS-Wrist were able to estimate EE for a range of PAs with \<10% error. Moderate to high intraclass correlation coefficients (ICCs) indicated that the EE estimated by PAMS-Arm (ICC3,1=.82, P\<.05) and PAMS-Wrist (ICC3,1=.89, P\<.05) are consistent with the criterion EE. Conclusions: Availability of PA monitors can assist wheelchair users to track PA levels, leading toward a healthier lifestyle. The new models we developed can estimate PA levels in manual wheelchair users with SCI in laboratory and community settings.
Intille, S., C. Haynes, D. Maniar, A. Ponnada and J. Manjourides (2016). μEMA: Microinteraction-based ecological momentary assessment (EMA) using a smartwatch. Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16), Association for Computing Machinery, New York, NY, United States.
Ecological Momentary Assessment (EMA) is a method of in situ data collection for assessment of behaviors, states, and contexts. Questions are prompted during everyday life using an individual's mobile device, thereby reducing recall bias and increasing validity over other self-report methods such as retrospective recall. We describe a microinteraction-based EMA method ("micro" EMA, or μEMA) using smartwatches, where all EMA questions can be answered with a quick glance and a tap \-- nearly as quickly as checking the time on a watch. A between-subjects, 4-week pilot study was conducted where μEMA on a smartwatch (n=19) was compared with EMA on a phone (n=14). Despite an =8 times increase in the number of interruptions, μEMA had a significantly higher compliance rate, completion rate, and first prompt response rate, and μEMA was perceived as less distracting. The temporal density of data collection possible with μEMA could prove useful in ubiquitous computing studies.
Maher, J. P., E. Dzubur, J. Huh, S. Intille and G. F. Dunton (2016). "Within-day time-varying associations between behavioral cognitions and physical activity in adults." Journal of Sport & Exercise Psychology 38(4): 423-434.
This study used time-varying effect modeling to examine time-of-day differences in how behavioral cognitions predict subsequent physical activity (PA). Adults (N = 116) participated in three 4-day "bursts" of ecological momentary assessment (EMA). Participants were prompted with eight EMA questionnaires per day assessing behavioral cognitions (i.e., intentions, self-efficacy, outcome expectations) and wore an accelerometer during waking hours. Subsequent PA was operationalized as accelerometer-derived minutes of moderate- or vigorousintensity PA in the 2 hr following the EMA prompt. On weekdays, intentions positively predicted subsequent PA in the morning (9:25 a.m.-11:45 a.m.) and in the evening (8:15 p.m.-10:00 p.m.). Self-efficacy positively predicted subsequent PA on weekday evenings (7:35 p.m.-10:00 p.m.). Outcome expectations were unrelated to subsequent PA on weekdays. On weekend days, behavior cognitions and subsequent PA were unrelated regardless of time of day. This study identifies windows of opportunity and vulnerability for motivation-based PA interventions aiming to deliver intervention content within the context of adults' daily lives.
Rodgers, R. F., W. Pernal, A. Matsumoto, M. Shiyko, S. Intille and D. L. Franko (2016). "Capitalizing on mobile technology to support healthy eating in ethnic minority college students." Journal of American College Health 64(2): 125-132.
Objective: To evaluate the capacity of a mobile technology-based intervention to support healthy eating among ethnic minority female students. Participants: Forty-three African American and Hispanic female students participated in a 3-week intervention between January and May 2013. Methods: Participants photographed their meals using their smart phone camera and received motivational text messages 3 times a day. At baseline, postintervention, and 10 weeks after the intervention, participants reported on fruit, vegetable, and sugar-sweetened beverage consumption. Participants were also weighed at baseline. Results: Among participants with body mass index (BMI) ≥25, fruit and vegetable consumption increased with time (p \< .01). Among participants with BMI \<21, consumption of fruit decreased (p \< .05), whereas the consumption of vegetables remained stable. No effects were found for sugar-sweetened beverage consumption. Conclusion: Mobile technology-based interventions could facilitate healthy eating among female ethnic minority college students, particularly those with higher BMI.
Bickmore, T., R. Asadi, A. Ehyaei, H. Fell, L. Henault, S. Intille, L. Quintiliani, A. Shamekhi, H. Trinh, K. Waite, C. Shanahan and M. Paasche-Orlow (2015). "Context-awareness in a persistent hospital companion agent." Proceedings of the Fifteenth International Conference on Intelligent Virtual Agents (IVA 2015)(LNAI 9238): 332-342.
We describe the design and preliminary evaluation of a virtual agent that provides continual bedside companionship and a range of health, information, and entertainment functions to hospital patients during their stay. The agent system uses sensors to enable it to be aware of events in the hospital room and the status of the patient, in order to provide context-sensitive health counseling. Patients in the pilot study responded well to having the agent in their rooms for 1--3 days and engaged in 9.4 conversations per day with the agent on average, using all available functions.
Dunton, G., E. Dzubur, M. Li, J. Huh, S. Intille and R. McConnell (2015). "Momentary assessment of psychosocial stressors, context, and asthma symptoms in Hispanic adolescents." Behavior Modification 40(1-2): 257-280.
The current study used a novel real-time data capture strategy, ecological momentary assessment (EMA), to examine whether within-day variability in stress and context leads to exacerbations in asthma symptomatology in the everyday lives of ethnic minority adolescents. Low-income Hispanic adolescents (N = 20; 7th-12th grade; 54% male) with chronic asthma completed 7 days of EMA on smartphones, with an average of five assessments per day during non-school time. EMA surveys queried about where (e.g., home, outdoors) and with whom (e.g., alone, with friends) participants were at the time of the prompt. EMA surveys also assessed over the past few hours whether participants had experienced specific stressors (e.g., being teased, arguing with anyone), asthma symptoms (e.g., wheezing, coughing), or used an asthma inhaler. Multilevel models tested the independent relations of specific stressors and context to subsequent asthma symptoms adjusting for age, gender, and chronological day in the study. Being outdoors, experiencing disagreements with parents, teasing, and arguing were associated with more severe self-reported asthma symptoms in the next few hours (ps \< .05). Being alone and having too much to do were unrelated to the experience of subsequent self-reported asthma symptoms. Using a novel real-time data capture strategy, results provide preliminary evidence that being outdoors and experiencing social stressors may induce asthma symptoms in low-income Hispanic children and adolescents with chronic asthma. The results of this preliminary study can serve as a basis for larger epidemiological and intervention studies.
Dunton, G. F., Y. Liao, E. Dzubur, A. M. Leventhal, J. Huh, T. Gruenewald, G. Margolin, C. Koprowski, E. Tate and S. Intille (2015). "Investigating within-day and longitudinal effects of maternal stress on children's physical activity, dietary intake, and body composition: Protocol for the MATCH study." Contemporary Clinical Trials 43: 142-154.
Parental stress is an understudied factor that may compromise parenting practices related to children's dietary intake, physical activity, and obesity. However, studies examining these associations have been subject to methodological limitations, including cross-sectional designs, retrospective measures, a lack of stress biomarkers, and the tendency to overlook momentary etiologic processes occurring within each day. This paper describes the recruitment, data collection, and data analytic protocols for the MATCH (Mothers And Their Children's Health) study, a longitudinal investigation using novel real-time data capture strategies to examine within-day associations of maternal stress with children's physical activity and dietary intake, and how these effects contribute to children's obesity risk. In the MATCH study, 200 mothers and their 8 to 12year-old children are participating in 6 semi-annual assessment waves across 3years. At each wave, measures for mother--child dyads include: (a) real-time Ecological Momentary Assessment (EMA) of self-reported daily psychosocial stressors (e.g., work at a job, family demands), feeling stressed, perceived stress, parenting practices, dietary intake, and physical activity with time and location stamps (b) diurnal salivary cortisol patterns, accelerometer-monitored physical activity, and 24-hour dietary recalls (c) retrospective questionnaires of sociodemographic, cultural, family, and neighborhood covariates and (d) height, weight, and waist circumference. Putative within-day and longitudinal effects of maternal stress on children's dietary intake, physical activity, and body composition will be tested through multilevel modeling and latent growth curve models, respectively. The results will inform interventions that help mothers reduce the negative effects of stress on weight-related parenting practices and children's obesity risk.
Dunton, G. F., Y. Liao, S. Intille, J. Huh and A. Leventhal (2015). "Momentary assessment of contextual influences on affective response during physical activity." Health Psychology 34(12): 1145-1153.
Objective: Higher positive and lower negative affective response during physical activity may reinforce motivation to engage in future activity. However, affective response during physical activity is typically examined under controlled laboratory conditions. This research used ecological momentary assessment (EMA) to examine social and physical contextual influences on momentary affective response during physical activity in naturalistic settings. Method: Participants included 116 adults (mean age = 40.3 years, 73% female) who completed 8 randomly prompted EMA surveys per day for 4 days across 3 semiannual waves. EMA surveys measured current activity level, social context, and physical context. Participants also rated their current positive and negative affect. Multilevel models assessed whether momentary physical activity level moderated differences in affective response across contexts controlling for day of the week, time of day, and activity intensity (measured by accelerometer). Results: The Activity Level × Alone interaction was significant for predicting positive affect (β = −0.302, SE = 0.133, p = .024). Greater positive affect during physical activity was reported when with other people (vs. alone). The Activity Level × Outdoors interaction was significant for predicting negative affect (β = −0.206, SE = 0.097, p = .034). Lower negative affect during physical activity was reported outdoors (vs. indoors). Conclusions: Being with other people may enhance positive affective response during physical activity, and being outdoors may dampen negative affective response during physical activity.
Hiremath, S. V., S. S. Intille, A. Kelleher, R. A. Cooper and D. Ding (2015). "Detection of physical activities using a physical activity monitor system for wheelchair users." Medical Engineering & Physics 37(1): 68-76.
Availability of physical activity monitors for wheelchair users can potentially assist these individuals to track regular physical activity (PA), which in turn could lead to a healthier and more active lifestyle. Therefore, the aim of this study was to develop and validate algorithms for a physical activity monitoring system (PAMS) to detect wheelchair based activities. The PAMS consists of a gyroscope based wheel rotation monitor (G-WRM) and an accelerometer device (wocket) worn on the upper arm or on the wrist. A total of 45 persons with spinal cord injury took part in the study, which was performed in a structured university-based laboratory environment, a semi-structured environment at the National Veterans Wheelchair Games, and in the participants' home environments. Participants performed at least ten PAs, other than resting, taken from a list of PAs. The classification performance for the best classifiers on the testing dataset for PAMS-Arm (G-WRM and wocket on upper arm) and PAMS-Wrist (G-WRM and wocket on wrist) was 89.26% and 88.47%, respectively. The outcomes of this study indicate that multi-modal information from the PAMS can help detect various types of wheelchair-based activities in structured laboratory, semi-structured organizational, and unstructured home environments.
Lin, P. H., S. Intille, G. Bennett, H. B. Bosworth, L. Corsino, C. Voils, S. Grambow, T. Lazenka, B. C. Batch, C. Tyson and L. P. Svetkey (2015). "Adaptive intervention design in mobile health: Intervention design and development in the Cell Phone Intervention for You trial." Clinical Trials 12(6): 634-645.
BACKGROUND/AIMS: The obesity epidemic has spread to young adults, and obesity is a significant risk factor for cardiovascular disease. The prominence and increasing functionality of mobile phones may provide an opportunity to deliver longitudinal and scalable weight management interventions in young adults. The aim of this article is to describe the design and development of the intervention tested in the Cell Phone Intervention for You study and to highlight the importance of adaptive intervention design that made it possible. The Cell Phone Intervention for You study was a National Heart, Lung, and Blood Institute-sponsored, controlled, 24-month randomized clinical trial comparing two active interventions to a usual-care control group. Participants were 365 overweight or obese (body mass index \>/= 25 kg/m2) young adults. METHODS: Both active interventions were designed based on social cognitive theory and incorporated techniques for behavioral self-management and motivational enhancement. Initial intervention development occurred during a 1-year formative phase utilizing focus groups and iterative, participatory design. During the intervention testing, adaptive intervention design, where an intervention is updated or extended throughout a trial while assuring the delivery of exactly the same intervention to each cohort, was employed. The adaptive intervention design strategy distributed technical work and allowed introduction of novel components in phases intended to help promote and sustain participant engagement. Adaptive intervention design was made possible by exploiting the mobile phone's remote data capabilities so that adoption of particular application components could be continuously monitored and components subsequently added or updated remotely. RESULTS: The cell phone intervention was delivered almost entirely via cell phone and was always-present, proactive, and interactive-providing passive and active reminders, frequent opportunities for knowledge dissemination, and multiple tools for self-tracking and receiving tailored feedback. The intervention changed over 2 years to promote and sustain engagement. The personal coaching intervention, alternatively, was primarily personal coaching with trained coaches based on a proven intervention, enhanced with a mobile application, but where all interactions with the technology were participant-initiated. CONCLUSION: The complexity and length of the technology-based randomized clinical trial created challenges in engagement and technology adaptation, which were generally discovered using novel remote monitoring technology and addressed using the adaptive intervention design. Investigators should plan to develop tools and procedures that explicitly support continuous remote monitoring of interventions to support adaptive intervention design in long-term, technology-based studies, as well as developing the interventions themselves.
Mannini, A., A. M. Sabatini and S. S. Intille (2015). "Accelerometry-based recognition of the placement sites of a wearable sensor." Pervasive and Mobile Computing 21: 62-74. Student lead author
This work describes an automatic method to recognize the position of an accelerometer worn on five different parts of the body: ankle, thigh, hip, arm and wrist from raw accelerometer data. Automatic detection of body position of a wearable sensor would enable systems that allow users to wear sensors flexibly on different body parts or permit systems that need to automatically verify sensor placement. The two-stage location detection algorithm works by first detecting time periods during which candidates are walking (regardless of where the sensor is positioned). Then, assuming that the data refer to walking, the algorithm detects the position of the sensor. Algorithms were validated on a dataset that is substantially larger than in prior work, using a leave-one-subject-out cross-validation approach. Correct walking and placement recognition were obtained for 97.4% and 91.2% of classified data windows, respectively.
Pickering, T. A., J. Huh, S. Intille, Y. Liao, M. A. Pentz and G. F. Dunton (2015). "Physical activity and variation in momentary behavioral cognitions: An ecological momentary assessment study." Journal of Physical Activity and Health 13(3): 344-351.
BACKGROUND: Decisions to perform moderate to vigorous physical activity (MVPA) involve behavioral cognitive processes that may differ within individuals depending on the situation. METHODS: Ecological momentary assessment (EMA) was used to examine the relationships of momentary behavioral cognitions (i.e., self-efficacy, outcome expectancy, intentions) with MVPA (measured by accelerometer). A sample of 116 adults (M=40.3 years, 72.4% female) provided real-time EMA responses via mobile phones across four days. Multilevel models tested whether momentary behavioral cognitions differed across contexts, and were associated with subsequent MVPA. Mixed-effects location scale models examined whether subject-level means and within-subject variances in behavioral cognitions were associated with average daily MVPA. RESULTS: Momentary behavioral cognitions differed across contexts for self-efficacy (p=.007) but not for outcome expectancy (p=.53) or intentions (p=.16). Momentary self-efficacy, intentions, and their interaction predicted MVPA within the subsequent two hours (p's\<.01). Average daily MVPA was positively associated with within-subject variance in momentary self-efficacy and intentions for physical activity (p's\<.05). CONCLUSIONS: While momentary behavioral cognitions are related to subsequent MVPA, adults with higher average MVPA have more variation in physical activity self-efficacy and intentions. Performing MVPA may depend more on how much behavioral cognitions vary across the day than whether they are generally high or low.
Rodgers, R. F., D. L. Franko, M. Shiyko, S. Intille, K. Wilson, D. O'Carroll, M. Lovering, A. Matsumoto, A. Iannuccilli, S. Luk and H. Shoemaker (2015). "Exploring healthy eating among ethnic minority students using mobile technology: Feasibility and adherence." Health Informatics Journal 22(3): 440-450.
Interventions aiming to help ethnically diverse emerging adults engage in healthy eating have had limited success. The aim of this study was to assess the feasibility of and adherence to an intervention capitalizing on mobile technology to improve healthy eating. Participants created an online photo food journal and received motivational text messages three times a day. Satisfaction with the intervention was assessed, as were control variables including depression and body dissatisfaction. In addition, weight and height were measured. Levels of adherence to the photo food journal were high with approximately two photos posted a day at baseline. However, adherence rates decreased over the course of the study. Body dissatisfaction positively predicted adherence, while body mass index negatively predicted study satisfaction. Mobile technology provides innovative avenues for healthy eating interventions. Such interventions appear acceptable and feasible for a short period; however, more work is required to evaluate their viability regarding long-term engagement.
Spruijt-Metz, D., E. Hekler, N. Saranummi, S. Intille, I. Korhonen, W. Nilsen, D. Rivera, B. Spring, S. Michie, D. Asch, A. Sanna, V. Salcedo, R. Kukakfa and M. Pavel (2015). "Building new computational models to support health behavior change and maintenance: New opportunities in behavioral research." Translational Behavioral Medicine 5(3): 335-346.
Adverse and suboptimal health behaviors and habits are responsible for approximately 40 % of preventable deaths, in addition to their unfavorable effects on quality of life and economics. Our current understanding of human behavior is largely based on static "snapshots" of human behavior, rather than ongoing, dynamic feedback loops of behavior in response to ever-changing biological, social, personal, and environmental states. This paper first discusses how new technologies (i.e., mobile sensors, smartphones, ubiquitous computing, and cloud-enabled processing/computing) and emerging systems modeling techniques enable the development of new, dynamic, and empirical models of human behavior that could facilitate just-in-time adaptive, scalable interventions. The paper then describes concrete steps to the creation of robust dynamic mathematical models of behavior including: (1) establishing "gold standard" measures, (2) the creation of a behavioral ontology for shared language and understanding tools that both enable dynamic theorizing across disciplines, (3) the development of data sharing resources, and (4) facilitating improved sharing of mathematical models and tools to support rapid aggregation of the models. We conclude with the discussion of what might be incorporated into a "knowledge commons," which could help to bring together these disparate activities into a unified system and structure for organizing knowledge about behavior.
Svetkey, L. P., B. C. Batch, P.-H. Lin, S. S. Intille, L. Corsino, C. C. Tyson, H. B. Bosworth, S. C. Grambow, C. Voils, C. Loria, J. A. Gallis, J. Schwager and G. B. Bennett (2015). "Cell phone Intervention for You (CITY): A randomized, controlled trial of behavioral weight loss intervention for young adults using mobile technology." Obesity(23): 2133-2141.
Objectives: To determine the effect on weight of two Mobile technology-based (mHealth) behavioral weight loss interventions in young adults. Methods: Randomized, controlled comparative effectiveness trial in 18-35 year olds with BMI \> 25 kg/m2 (overweight/obese), with participants randomized to 24 months of mHealth intervention delivered by interactive smartphone application on a cell phone (CP); personal coaching enhanced by smartphone self-monitoring (PC); or Control. Results: The 365 randomized participants had mean baseline BMI of 35 kg/m2. Final weight was measured in 86% of participants. CP was not superior to Control at any measurement point. PC participants lost significantly more weight than Controls at 6 months (net effect -1.92 kg \[CI -3.17, -0.67\], p=0.003), but not at 12 and 24 months. Conclusions: Despite high intervention engagement and study retention, the inclusion of behavioral principles and tools in both interventions, and weight loss in all treatment groups, CP did not lead to weight loss and PC did not lead to sustained weight loss relative to control. Although mHealth solutions offer broad dissemination and scalability, the CITY results sound a cautionary note concerning intervention delivery by mobile applications. Effective intervention may require the efficiency of mobile technology, the social support and human interaction of personal coaching, and an adaptive approach to intervention design.
Batch, B. C., C. Tyson, J. Bagwell, L. Corsino, S. Intille, P. H. Lin, T. Lazenka, G. Bennett, H. B. Bosworth, C. Voils, S. Grambow, A. Sutton, R. Bordogna, M. Pangborn, J. Schwager, K. Pilewski, C. Caccia, J. Burroughs and L. P. Svetkey (2014). "Weight loss intervention for young adults using mobile technology: design and rationale of a randomized controlled trial - Cell Phone Intervention for You (CITY)." Contemporary Clinical Trials 37(2): 333-341.
BACKGROUND: The obesity epidemic has spread to young adults, leading to significant public health implications later in adulthood. Intervention in early adulthood may be an effective public health strategy for reducing the long-term health impact of the epidemic. Few weight loss trials have been conducted in young adults. It is unclear what weight loss strategies are beneficial in this population. PURPOSE: To describe the design and rationale of the NHLBI-sponsored Cell Phone Intervention for You (CITY) study, which is a single center, randomized three-arm trial that compares the impact on weight loss of 1) a behavioral intervention that is delivered almost entirely via cell phone technology (Cell Phone group); and 2) a behavioral intervention delivered mainly through monthly personal coaching calls enhanced by self-monitoring via cell phone (Personal Coaching group), each compared to 3) a usual care, advice-only control condition. METHODS: A total of 365 community-dwelling overweight/obese adults aged 18-35 years were randomized to receive one of these three interventions for 24 months in parallel group design. Study personnel assessing outcomes were blinded to group assignment. The primary outcome is weight change at 24 \[corrected\] months. We hypothesize that each active intervention will cause more weight loss than the usual care condition. Study completion is anticipated in 2014. CONCLUSIONS: If effective, implementation of the CITY interventions could mitigate the alarming rates of obesity in young adults through promotion of weight loss. ClinicalTrial.gov: NCT01092364.
Dunton, G. F., E. Dzubur, K. Kawabata, B. Yanez, B. Bo and S. Intille (2014). "Development of a smartphone application to measure physical activity using sensor-assisted self-report." Frontiers in Public Health 2(12): 1-13.
Introduction: Despite the known advantages of objective physical activity monitors (e.g., accelerometers), these devices have high rates of non-wear, which leads to missing data. Objective activity monitors are also unable to capture valuable contextual information about behavior. Adolescents recruited into physical activity surveillance and intervention studies will increasingly have smartphones, which are miniature computers with built-in motion sensors.
Goodwin, M. S., M. Haghighi, Q. Tang, M. Akcakaya, D. Erdogmus and S. Intille (2014). "Moving towards a real-time system for automatically recognizing stereotypical motor movements in individuals on the autism spectrum using wireless accelerometry." Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: 861-872.
This paper extends previous work automatically detecting stereotypical motor movements (SMM) in individuals on the autism spectrum. Using three-axis accelerometer data obtained through wearable wireless sensors, we compare recognition results for two different classifiers \-- Support Vector Machine and Decision Tree \-- in combination with different feature sets based on time-frequency characteristics of accelerometer data. We use data collected from six individuals on the autism spectrum who participated in two different studies conducted three years apart in classroom settings, and observe an average accuracy across all participants over time ranging from 81.2% (TPR: 0.91; FPR: 0.21) to 99.1% (TPR: 0.99; FPR: 0.01) for all combinations of classifiers and feature sets. We also provide analyses of kinematic parameters associated with observed movements in an attempt to explain classifier-feature specific performance. Based on our results, we conclude that real-time, person-dependent, adaptive algorithms are needed in order to accurately and consistently measure SMM automatically in individuals on the autism spectrum over time in real-word settings.
Liao, Y., S. Intille, J. Wolch, M. A. Pentz and G. F. Dunton (2014). "Understanding the physical and social contexts of children's nonschool sedentary behavior: An ecological momentary assessment study." Journal of Physical Activity and Health 11(3): 588-595.
BACKGROUND: Research on children's sedentary behavior has relied on recall-based self-report or accelerometer methods, which do not assess the context of such behavior. PURPOSE: This study used ecological momentary assessment (EMA) to determine where and with whom children's sedentary behavior occurs during their nonschool time. METHODS: Children (N = 120) ages 9-13 years (51% male, 33% Hispanic) wore mobile phones that prompted surveys (20 total) for 4 days. Surveys measured current activity (eg, exercise, watching TV), physical location (eg, home, outdoors), and social company (eg, family, friends). RESULTS: Children engaged in a greater percentage of leisure-oriented (eg, watching TV) than productive (eg, reading, doing homework) sedentary behavior (70% vs 30%, respectively). Most of children's sedentary activity occurred at home (85%). Children's sedentary activity took place most often with family members (58%). Differences in physical context of sedentary behavior were found for older vs. younger children (P \< .05). Type of sedentary behavior differed by gender, racial/ethnic group, and social context (P \< .05). CONCLUSION: Children may prefer or have greater opportunities to be sedentary in some contexts than others. Research demonstrates the potential for using EMA to capture real-time information about children's sedentary behavior during their nonschool time.
Liao, Y., S. S. Intille and G. F. Dunton (2014). "Using ecological momentary assessment to understand where and with whom adults' physical and sedentary activity occur." International Journal of Behavioral Medicine 22(1): 51-61.
PURPOSE: This study used Ecological Momentary Assessment (EMA), a real-time self-report strategy, to describe the physical and social contexts of adults' physical activity and sedentary activity during their everyday lives and to determine whether these patterns and relationships differ for men and women. METHODS: Data from 114 adults were collected through mobile phones across 4 days. Eight electronic EMA surveys were randomly prompted each day asking about current activities (e.g., physical or sedentary activity), physical and social contexts, and perceived outdoor environmental features (e.g., greenness/vegetation, safety, and traffic). All participants also wore accelerometers during this period to objectively measure moderate-to-vigorous physical activity (MVPA) and sedentary activity. RESULTS: Home was the most common physical context for EMA-reported physical and sedentary activity. Most of these activities occurred when participants were alone. When alone, the most commonly EMA-reported physical activity and sedentary activity was walking and reading/using computer, respectively. When in outdoor home locations (e.g., yard and driveway) women demonstrated higher levels of MVPA, whereas men demonstrated higher levels of MVPA when in outdoor park settings (ps \< .05). Men but not women demonstrated higher levels of MVPA in settings with a greater degree of perceived greenness and vegetation (p \< .05). CONCLUSIONS: The current study shows how EMA via mobile phones and accelerometers can be combined to offer an innovative approach to assess the contexts of adults' daily physical and sedentary activity. Future studies could consider utilizing this method in more representative samples to gather context-specific information to inform the development of physical activity interventions.
Tang, Q., D. J. Vidrine, E. Crowder and S. S. Intille (2014). "Automated detection of puffing and smoking with wrist accelerometers." 8th International Conference on Pervasive Computing Technologies for Healthcare: 80-87. Student lead author
Real-time, automatic detection of smoking behavior could lead to novel measurement tools for smoking research and "just-in-time" interventions that may help people quit, reducing preventable deaths. This paper discusses the use of machine learning with wrist accelerometer data for automatic puffing and smoking detection. A two-layer smoking detection model is proposed that incorporates both low-level time domain features and high-level smoking topography such as inter-puff intervals and puff frequency to detect puffing then smoking. On a pilot dataset of 6 individuals observed for 11.8 total hours in real-life settings performing complex tasks while smoking, the model obtains a cross validation F1-score of 0.70 for puffing detection and 0.79 for smoking detection over all participants, and a mean F1-score of 0.75 for puffing detection with user-specific training data. Unresolved challenges that must still be addressed in this activity detection domain are discussed.
Corsino, L., P. H. Lin, B. C. Batch, S. Intille, S. C. Grambow, H. B. Bosworth, G. G. Bennett, C. Tyson, L. P. Svetkey and C. I. Voils (2013). "Recruiting young adults into a weight loss trial: report of protocol development and recruitment results." Contemp Clin Trials 35(2): 1-7.
Obesity has spread to all segments of the U.S. population. Young adults, aged 18-35 years, are rarely represented in clinical weight loss trials. We conducted a qualitative study to identify factors that may facilitate recruitment of young adults into a weight loss intervention trial. Participants were 33 adults aged 18-35 years with BMI \>/=25 kg/m(2). Six group discussions were conducted using the nominal group technique. Health, social image, and "self" factors such as emotions, self-esteem, and confidence were reported as reasons to pursue weight loss. Physical activity, dietary intake, social support, medical intervention, and taking control (e.g. being motivated) were perceived as the best weight loss strategies. Incentives, positive outcomes, education, convenience, and social support were endorsed as reasons young adults would consider participating in a weight loss study. Incentives, advertisement, emphasizing benefits, and convenience were endorsed as ways to recruit young adults. These results informed the Cellphone Intervention for You (CITY) marketing and advertising, including message framing and advertising avenues. Implications for recruitment methods are discussed.
Mannini, A., S. S. Intille, M. Rosenberger, A. M. Sabatini and W. Haskell (2013). "Activity recognition using a single accelerometer placed at the wrist or ankle." Medicine & Science in Sports & Exercise 45(11): 2193-2203. Student lead author
PURPOSE: Large physical activity surveillance projects such as the UK Biobank and NHANES are using wrist-worn accelerometer-based activity monitors that collect raw data. The goal is to increase wear time by asking subjects to wear the monitors on the wrist instead of the hip, and then to use information in the raw signal to improve activity type and intensity estimation. The purposes of this work was to obtain an algorithm to process wrist and ankle raw data and to classify behavior into four broad activity classes: ambulation, cycling, sedentary, and other activities. METHODS: Participants (N = 33) wearing accelerometers on the wrist and ankle performed 26 daily activities. The accelerometer data were collected, cleaned, and preprocessed to extract features that characterize 2-, 4-, and 12.8-s data windows. Feature vectors encoding information about frequency and intensity of motion extracted from analysis of the raw signal were used with a support vector machine classifier to identify a subject's activity. Results were compared with categories classified by a human observer. Algorithms were validated using a leave-one-subject-out strategy. The computational complexity of each processing step was also evaluated. RESULTS: With 12.8-s windows, the proposed strategy showed high classification accuracies for ankle data (95.0%) that decreased to 84.7% for wrist data. Shorter (4 s) windows only minimally decreased performances of the algorithm on the wrist to 84.2%. CONCLUSIONS: A classification algorithm using 13 features shows good classification into the four classes given the complexity of the activities in the original data set. The algorithm is computationally efficient and could be implemented in real time on mobile devices with only 4-s latency.
Mannini, A., A. Sabatini and S. Intille (2013). "Human gait detection from wrist-worn accelerometer data." Gait & Posture(37): S26-S27. Student lead author
In this study we propose a system capable of detecting walking and running events from different activities that could include wrist motion.
Rosenberger, M. E., W. L. Haskell, F. Albinali, S. Mota, J. Nawyn and S. Intille (2013). "Estimating activity and sedentary behavior from an accelerometer on the hip or wrist." Medicine & Science in Sports & Exercise 45(5): 964-975. Student lead author
PURPOSE: Previously, the National Health and Examination Survey measured physical activity with an accelerometer worn on the hip for 7 d but recently changed the location of the monitor to the wrist. This study compared estimates of physical activity intensity and type with an accelerometer on the hip versus the wrist. METHODS: Healthy adults (n = 37) wore triaxial accelerometers (Wockets) on the hip and dominant wrist along with a portable metabolic unit to measure energy expenditure during 20 activities. Motion summary counts were created, and receiver operating characteristic (ROC) curves were then used to determine sedentary and activity intensity thresholds. Ambulatory activities were separated from other activities using the coefficient of variation of the counts. Mixed-model predictions were used to estimate activity intensity. RESULTS: The ROC for determining sedentary behavior had greater sensitivity and specificity (71% and 96%) at the hip than at the wrist (53% and 76%), as did the ROC for moderate- to vigorous-intensity physical activity on the hip (70% and 83%) versus the wrist (30% and 69%). The ROC for the coefficient of variation associated with ambulation had a larger AUC at the hip compared to the wrist (0.83 and 0.74). The prediction model for activity energy expenditure resulted in an average difference of 0.55 +/- 0.55 METs on the hip and 0.82 +/- 0.93 METs on the wrist. CONCLUSIONS: Methods frequently used for estimating activity energy expenditure and identifying activity intensity thresholds from an accelerometer on the hip generally do better than similar data from an accelerometer on the wrist. Accurately identifying sedentary behavior from a lack of wrist motion presents significant challenges.
Albinali, F., M. S. Goodwin and S. S. Intille (2012). "Detecting stereotypical motor movements in the classroom using accelerometry and pattern recognition algorithms." Pervasive and Mobile Computing 8: 103-114. Postdoc lead author
Individuals with Autism Spectrum Disorders (ASD) frequently engage in stereotyped and repetitive motor movements. Automatically detecting these movements using comfortable, miniature wireless sensors could advance autism research and enable new intervention tools for the classroom that help children and their caregivers monitor, understand, and cope with this potentially problematic class of behavior. We present activity recognition results for stereotypical hand flapping and body rocking using accelerometer data collected wirelessly from six children with ASD repeatedly observed by experts in real classroom settings. An overall recognition accuracy of 88.6% (TP: 0.85; FP: 0.08) was achieved using three sensors. We also present pilot work in which non-experts use software on mobile phones to annotate stereotypical motor movements for classifier training. Preliminary results indicate that non-expert annotations for training can be as effective as expert annotations. Challenges encountered when applying machine learning to this domain, as well as implications for the development of real-time classroom interventions and research tools are discussed.
Dunton, G. F., S. S. Intille, J. Wolch and M. A. Pentz (2012). "Children's perceptions of physical activity environments captured through ecological momentary assessment: A validation study." Prev Med 55(2): 119-121.
OBJECTIVE: This study used ecological momentary assessment (EMA) to investigate whether children's perceptions of physical activity (PA) settings correspond with (1) parents' perceptions of neighborhood characteristics (convergent construct validity) and (2) children's level of PA in those settings (concurrent criterion validity). METHODS: Low-to-middle income, ethnically-diverse children (N=108) (ages 9-13) living in Southern California participated in 8 days of EMA during non-school time. EMA measured current activity type (e.g., sports/exercise, TV watching) and perceptions of the current setting (i.e., vegetation, traffic, safety). The Neighborhood Environment Walkability Survey (NEWS) assessed parents' perceptions of neighborhood characteristics. EMA responses were time-matched to moderate-to-vigorous physical activity (MVPA) (measured by accelerometer) in the 30 min before and after each EMA survey. Data were collected in 2009-2010. RESULTS: Children's perceptions of vegetation and traffic in PA settings corresponded with parents' perceptions of the aesthetics (OR=2.21, 95% CI=1.04-4.73) and traffic (OR=2.64, 95% CI=1.31-5.30) in neighborhood environment, respectively. MVPA minutes were higher in settings perceived by children to have less traffic (beta=3.47, p\<.05). CONCLUSIONS: This work provides initial support for the construct and criterion validity of EMA-based measures of children's perceptions of their PA environments.
Dunton, G. F., S. S. Intille, J. Wolch and M. A. Pentz (2012). "Investigating the impact of a smart growth community on the contexts of children's physical activity using ecological momentary assessment." Health Place 18(1): 76-84.
This quasi-experimental research used Ecological Momentary Assessment with electronic surveys delivered through mobile phones to determine whether children change the type of contexts (i.e., settings) where they engage in physical activity after a recent move to a smart growth (SG) community in the U.S. as compared to children living in conventional low-to-medium density U.S. suburban communities (controls). SG vs. control children engaged in a greater proportion of physical activity bouts with friends, a few blocks from home, and at locations to which they walked. Over six months, the proportion of physical activity bouts reported at home (indoors) and in high traffic locations decreased among SG but not control children. Six-month increases in daily moderate-to-vigorous physical activity did not significantly differ by group. Children might have altered the type of contexts where they engage in physical activity after moving to SG communities, yet more time may be necessary for these changes to impact overall physical activity.
Dunton, G. F., K. Kawabata, S. Intille, J. Wolch and M. A. Pentz (2012). "Assessing the social and physical contexts of children's leisure-time physical activity: An ecological momentary assessment study." Am J Health Promot 26(3): 135-142.
PURPOSE: To use Ecological Momentary Assessment with mobile phones to describe where and with whom children's leisure-time physical activity occurs. DESIGN: Repeated assessments across 4 days (Friday-Monday) during nonschool time (20 total). SETTING: Chino, California, and surrounding communities. SUBJECTS: Primarily low to middle income children (N =121; aged 9-13 years; x =11.0 years, SD =1.2 years; 52% male, 38% Hispanic/Latino). MEASURES: Electronic surveys measured current activity (e.g., active play/sports/exercise, watching TV/movies), social company (e.g., family, friends, alone), physical location (e.g., home, outdoors, school), and other perceived contextual features (e.g., safety, traffic, vegetation, distance from home). Analysis . Multilevel linear and multinomial logistic regression. RESULTS: Most of children's physical activity occurred outdoors (away from home) (42%), followed by at home (indoors) (30%), front/backyard (at home) (8%), someone else's house (8%), at a gym/recreation center (3%), and other locations (9%). Children's physical activity took place most often with multiple categories of people together (e.g., friends and family) (39%), followed by family members only (32%), alone (15%), and with friends only (13%). Age, weight status, income, and racial/ethnic differences in physical activity contexts were observed. CONCLUSIONS: The most frequently reported contexts for children's leisure time physical activity were outdoors and with family members and friends together.
Dunton, G. F., Y. Liao, K. Kawabata and S. Intille (2012). "Momentary assessment of adults' physical activity and sedentary behavior: Feasibility and validity." Frontiers in Psychology 3: 260.
Introduction: Mobile phones are ubiquitous and easy to use, and thus have the capacity to collect real-time data from large numbers of people. Research tested the feasibility and validity of an Ecological Momentary Assessment (EMA) self-report protocol using electronic surveys on mobile phones to assess adults' physical activity and sedentary behaviors. Methods: Adults (N = 110; 73% female, 30% Hispanic, 62% overweight/obese) completed a 4-day signal-contingent EMA protocol (Saturday-Tuesday) with eight surveys randomly spaced throughout each day. EMA items assessed current activity (e.g., Watching TV/Movies, Reading/Computer, Physical Activity/Exercise). EMA responses were time-matched to minutes of moderate-to-vigorous physical activity (MVPA) and sedentary activity (SA) measured by accelerometer immediately before and after each EMA prompt. Results: Unanswered EMA prompts had greater MVPA (+/-15 min) than answered EMA prompts (p = 0.029) for under/normal weight participants, indicating that activity level might influence the likelihood of responding. The 15-min. intervals before versus after the EMA-reported physical activity (n = 296 occasions) did not differ in MVPA (p \> 0.05), suggesting that prompting did not disrupt physical activity. SA decreased after EMA-reported sedentary behavior (n = 904 occasions; p \< 0.05) for overweight and obese participants. As compared with other activities, EMA-reported physical activity and sedentary behavior had significantly greater MVPA and SA, respectively, in the +/-15 min of the EMA prompt (ps \< 0.001), providing evidence for criterion validity. Conclusion: Findings generally support the acceptability and validity of a 4-day signal-contingent EMA protocol using mobile phones to measure physical activity and sedentary behavior in adults. However, some MVPA may be missed among underweight and normal weight individuals.
Intille, S. S., J. Lester, J. F. Sallis and G. Duncan (2012). "New horizons in sensor development." Medicine & Science in Sports & Exercise 44(S24-31).
Background: Accelerometry and other sensing technologies are important tools for physical activity measurement. Engineering advances have allowed developers to transform clunky, uncomfortable, and conspicuous monitors into relatively small, ergonomic, and convenient research tools. New devices can be used to collect data on overall physical activity and, in some cases, posture, physiological state, and location, for many days or weeks from subjects during their everyday lives. In this review article, we identify emerging trends in several types of monitoring technologies and gaps in the current state of knowledge. Best practices: The only certainty about the future of activity-sensing technologies is that researchers must anticipate and plan for change. We propose a set of best practices that may accelerate adoption of new devices and increase the likelihood that data being collected and used today will be compatible with new data sets and methods likely to appear on the horizon. Future directions: We describe several technology-driven trends, ranging from continued miniaturization of devices that provide gross summary information about activity levels and energy expenditure to new devices that provide highly detailed information about the specific type, amount, and location of physical activity. Some devices will take advantage of consumer technologies, such as mobile phones, to detect and respond to physical activity in real time, creating new opportunities in measurement, remote compliance monitoring, data-driven discovery, and intervention.
Rodes, C. E., S. N. Chillrud, W. L. Haskell, S. S. Intille, F. Albinali and M. E. Rosenberger (2012). "Predicting adult pulmonary ventilation volume and wearing compliance by on-board accelerometry during personal level exposure assessments." Atmospheric Environment 57: 126-137.
Background: Metabolic functions typically increase with human activity, but optimal methods to characterize activity levels for real-time predictions of ventilation volume (l/min) during exposure assessments have not been available. Could tiny, triaxial accelerometers be incorporated into personal level monitors to define periods of acceptable wearing compliance, and allow the exposures (μg/m3) to be extended to potential doses in μg/min/kg of body weight? Objectives: In a pilot effort, we tested: 1) whether appropriately-processed accelerometer data could be utilized to predict compliance and in linear regressions to predict ventilation volumes in real time as an on-board component of personal level exposure sensor systems, and 2) whether locating the exposure monitors on the chest in the breathing zone, provided comparable accelerometric data to other locations more typically utilized (waist, thigh, wrist, etc.). Methods: Prototype exposure monitors from RTI International and Columbia University were worn on the chest by a pilot cohort of adults while conducting an array of scripted activities (all \<10 METS), spanning common recumbent, sedentary, and ambulatory activity categories. Referee Wocket accelerometers that were placed at various body locations allowed comparison with the chest-located exposure sensor accelerometers. An Oxycon Mobile mask was used to measure oral-nasal ventilation volumes in-situ. For the subset of participants with complete data (n= 22), linear regressions were constructed (processed accelerometric variable versus ventilation rate) for each participant and exposure monitor type, and Pearson correlations computed to compare across scenarios. Results: Triaxial accelerometer data were demonstrated to be adequately sensitive indicators for predicting exposure monitor wearing compliance. Strong linear correlations (R values from 0.77 to 0.99) were observed for all participants for both exposure sensor accelerometer variables against ventilation volume for recumbent, sedentary, and ambulatory activities with MET values \~\<6. The RTI monitors mean R value of 0.91 was slightly higher than the Columbia monitors mean of 0.86 due to utilizing a 20 Hz data rate instead of a slower 1 Hz rate. A nominal mean regression slope was computed for the RTI system across participants and showed a modest RSD of +/-36.6%. Comparison of the correlation values of the exposure monitors with the Wocket accelerometers at various body locations showed statistically identical regressions for all sensors at alternate hip, ankle, upper arm, thigh, and pocket locations, but not for the Wocket accelerometer located at the dominant-side wrist location (R=0.57; p=0.016). Conclusions: Even with a modest number of adult volunteers, the consistency and linearity of regression slopes for all subjects were very good with excellent within-person Pearson correlations for the accelerometer versus ventilation volume data. Computing accelerometric standard deviations allowed good sensitivity for compliance assessments even for sedentary activities. These pilot findings supported the hypothesis that a common linear regression is likely to be usable for a wider range of adults to predict ventilation volumes from accelerometry data over a range of low to moderate energy level activities. The predicted volumes would then allow real-time estimates of potential dose, enabling more robust panel studies. The poorer correlation in predicting ventilation rate for an accelerometer located on the wrist suggested that this location should not be considered for predictions of ventilation volume.
Dunton, G. F., Y. Liao, S. Intille, J. Wolch and M. A. Pentz (2011). "Physical and social contextual influences on children's leisure-time physical activity: An ecological momentary assessment study." Journal of Physical Activity and Health 8 Suppl 1: S103-108.
BACKGROUND: This study used real-time electronic surveys delivered through mobile phones, known as Ecological Momentary Assessment (EMA), to determine whether level and experience of leisure-time physical activity differ across children's physical and social contexts. METHODS: Children (N = 121; ages 9 to 13 years; 52% male, 32% Hispanic/Latino) participated in 4 days (Fri.-Mon.) of EMA during nonschool time. Electronic surveys (20 total) assessed primary activity (eg, active play/sports/exercise), physical location (eg, home, outdoors), social context (eg, friends, alone), current mood (positive and negative affect), and enjoyment. Responses were time-matched to the number of steps and minutes of moderate-to-vigorous physical activity (MVPA; measured by accelerometer) in the 30 minutes before each survey. RESULTS: Mean steps and MVPA were greater outdoors than at home or at someone else's house (all P \< .05). Steps were greater with multiple categories of company (eg, friends and family together) than with family members only or alone (all P \< .05). Enjoyment was greater outdoors than at home or someone else's house (all P \< .05). Negative affect was greater when alone and with family only than friends only (all P \< .05). CONCLUSION: Results describing the value of outdoor and social settings could inform context-specific interventions in this age group.
Dunton, G. F., Y. Liao, S. S. Intille, D. Spruijt-Metz and M. Pentz (2011). "Investigating children's physical activity and sedentary behavior using ecological momentary assessment with mobile phones." Obesity (Silver Spring) 19(6): 1205-1212.
The risk of obesity during childhood can be significantly reduced through increased physical activity and decreased sedentary behavior. Recent technological advances have created opportunities for the real-time measurement of these behaviors. Mobile phones are ubiquitous and easy to use, and thus have the capacity to collect data from large numbers of people. The present study tested the feasibility, acceptability, and validity of an electronic ecological momentary assessment (EMA) protocol using electronic surveys administered on the display screen of mobile phones to assess children's physical activity and sedentary behaviors. A total of 121 children (ages 9-13, 51% male, 38% at risk for overweight/overweight) participated in EMA monitoring from Friday afternoon to Monday evening during children's nonschool time, with 3-7 surveys/day. Items assessed current activity (e.g., watching TV/movies, playing video games, active play/sports/exercising). Children simultaneously wore an Actigraph GT2M accelerometer. EMA survey responses were time-matched to total step counts and minutes of moderate-to-vigorous physical activity (MVPA) occurring in the 30 min before each EMA survey prompt. No significant differences between answered and unanswered EMA surveys were found for total steps or MVPA. Step counts and the likelihood of 5+ min of MVPA were significantly higher during EMA-reported physical activity (active play/sports/exercising) vs. sedentary behaviors (reading/computer/homework, watching TV/movies, playing video games, riding in a car) (P \< 0.001). Findings generally support the acceptability and validity of a 4-day EMA protocol using mobile phones to measure physical activity and sedentary behavior in children during leisure time.
Goodwin, M. S., S. S. Intille, F. Albinali and W. F. Velicer (2011). "Automated detection of stereotypical motor movements." Journal of Autism and Developmental Disorders 41(6): 770-782.
To overcome problems with traditional methods for measuring stereotypical motor movements in persons with Autism Spectrum Disorders (ASD), we evaluated the use of wireless three-axis accelerometers and pattern recognition algorithms to automatically detect body rocking and hand flapping in children with ASD. Findings revealed that, on average, pattern recognition algorithms correctly identified approximately 90% of stereotypical motor movements repeatedly observed in both laboratory and classroom settings. Precise and efficient recording of stereotypical motor movements could enable researchers and clinicians to systematically study what functional relations exist between these behaviors and specific antecedents and consequences. These measures could also facilitate efficacy studies of behavioral and pharmacologic interventions intended to replace or decrease the incidence or severity of stereotypical motor movements.
Intille, S. S., F. Albinali, S. Mota, B. Kuris, P. Botana and W. L. Haskell (2011). "Design of a wearable physical activity monitoring system using mobile phones and accelerometers." Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC): 3636-3639.
This paper describes the motivation for, and overarching design of, an open-source hardware and software system to enable population-scale, longitudinal measurement of physical activity and sedentary behavior using common mobile phones. The "Wockets" data collection system permits researchers to collect raw motion data from participants who wear multiple small, comfortable sensors for 24 hours per day, including during sleep, and monitor data collection remotely.
Albinali, F., S. S. Intille, W. Haskell and M. Rosenberger (2010). "Using wearable activity type detection to improve physical activity energy expenditure estimation." Proceedings of the 12th International Conference on Ubiquitous Computing: 311-320. Postdoc lead author. Best paper nominee
Accurate, real-time measurement of energy expended during everyday activities would enable development of novel health monitoring and wellness technologies. A technique using three miniature wearable accelerometers is presented that improves upon state-of-the-art energy expenditure (EE) estimation. On a dataset acquired from 24 subjects performing gym and household activities, we demonstrate how knowledge of activity type, which can be automatically inferred from the accelerometer data, can improve EE estimates by more than 15% when compared to the best estimates from other methods.
Albinali, F., M. S. Goodwin and S. S. Intille (2009). "Recognizing stereotypical motor movements in the laboratory and classroom: A case study with children on the autism spectrum." Proceedings of the 11th International Conference on Ubiquitous Computing: 71-80. Postdoc lead author. Best paper award
Individuals with Autism Spectrum Disorders (ASD) frequently engage in stereotyped and repetitive motor movements. Automatically detecting these movements in real-time using comfortable, miniature wireless sensors could advance autistic research and enable new intervention tools for the classroom that help children and their caregivers monitor and cope with this potentially problematic class of behavior. We present activity recognition results for stereotypical hand flapping and body rocking using data collected from six children with ASD repeatedly observed in both laboratory and classroom settings. In the classroom, an overall recognition accuracy of 88.6% (TP: 0.85; FP: 0.08) was achieved using three sensors. Challenges encountered when applying machine learning to this domain, as well as implications for the development of real-time classroom interventions and research tools, are discussed.
Gupta, M., S. S. Intille and K. Larson (2009). "Adding GPS-control to traditional thermostats: An exploration of potential energy savings and design challenges." Proceedings of the Seventh International Conference on Pervasive Computing LNCS 5538: 95-114. Student lead author
Although manual and programmable home thermostats can save energy when used properly, studies have shown that over 40% of U.S. homes may not use energy-saving temperature setbacks when homes are unoccupied. We propose a system for augmenting these thermostats using just-in-time heating and cooling based on travel-to-home distance obtained from location-aware mobile phones. Analyzing GPS travel data from 8 participants (8-12 weeks each) and heating and cooling characteristics from 5 homes, we report results of running computer simulations estimating potential energy savings from such a device. Using a GPS-enabled thermostat might lead to savings of as much as 7% for some households that do not regularly use the temperature setback afforded by manual and programmable thermostats. Significantly, these savings could be obtained without requiring any change in occupant behavior or comfort level, and the technology could be implemented affordably by exploiting the ubiquity of mobile phones. Additional savings may be possible with modest context-sensitive prompting. We report on design considerations identified during a pilot test of a fully-functional implementation of the system.
Intille, S. S., J. Nawyn, B. Logan and G. D. Abowd (2009). "Developing shared home behavior datasets to advance HCI and ubiquitous computing research." Proceedings of the 27th International Conference on Human Factors in Computing Systems: 4763-4766.
Researchers in human-computer interaction and allied fields are increasingly interested in using new sensing capabilities to create context-aware interfaces and devices for the home. Data from sensors worn on the body or installed in an environment can be used by algorithms to infer what activities the home occupant may be engaged in and enable applications to respond accordingly. This one-day CHI'09 workshop would convene a multidisciplinary group of researchers to discuss strategies for creating community resources that might accelerate research on development of home technologies. In particular, the participants will discuss how to collaboratively gather high quality synchronized data streams from real homes, as well as qualitative material about home occupants and their behaviors. The resultant datasets could facilitate work on context modeling and enable researchers in other areas of HCI to explore contextual factors influencing the use of technology in naturalistic settings. The outcome of the workshop will be a community index of existing shared datasets of home behavior and guidelines for those interested in creating and disseminating new datasets.
Goodwin, M. S., W. F. Velicer and S. S. Intille (2008). "Telemetric monitoring in the behavior sciences." Behavior Research Methods 40(1): 328-341.
This article reviews recent advances in telemetrics, a class of wireless information systems technology that can collect and transmit a wide variety of behavioral and environmental data remotely. Telemetrics include wearable computers that weave on-body sensors into articles of clothing, ubiquitous computers that embed sensors and transmitters seamlessly into the environment, and handheld devices, such as mobile phones and personal digital assistants, that can record cognitive and affective states. Examples of telemetric applications are provided to illustrate how this technology has been used in the behavioral sciences to unobtrusively and repeatedly gather physiological, behavioral, environmental, cognitive, and affective data in natural settings. Special issues relating to privacy and confidentiality, practical considerations, and statistical and measurement challenges when telemetrics are used are also discussed.
Kaushik, P., S. S. Intille and K. Larson (2008). "Observations from a case study on user adaptive reminders for medication adherence." Proceedings of the Second International Conference on Pervasive Computing Technologies for Healthcare: 250-253. Student lead author
We present the design and exploratory evaluation of a sensor-driven adaptive reminder system for home medical tasks. Our prototype implementation consists of a mobile reminder delivery device and ambient sensors for determining opportune moments for reminder delivery. A volunteer used the prototype in a residential research facility while adhering to a regimen of simulated medical tasks for ten days. Based on this case study, including direct observation of individual alert-action sequences, we make four recommendations for designers of context-sensitive adaptive reminder systems.
Kaushik, P., S. S. Intille and K. Larson (2008). "User-adaptive reminders for home-based medical tasks. A case study." Methods of Information in Medicine 47(3): 203-207. Student lead author
OBJECTIVES: We present a prototype adaptive reminder system for home-based medical tasks. The system consists of a mobile device for reminder presentation and ambient sensors to determine opportune moments for reminder delivery. Our objective was to study interaction with the prototype under naturalistic living conditions and gain insight into factors affecting the long-term acceptability of context-sensitive reminder systems for the home setting. METHODS: A volunteer participant used the prototype in a residential research facility while adhering to a regimen of simulated medical tasks for ten days. Some reminders were scheduled at fixed times during the day and some were automatically time-shifted based on sensor data. We made a complete video and sensor record of the stay. Finally, the participant commented about his experiences with the system in a debriefing interview. RESULTS: Based on this case study, including direct observation of individual alert-action sequences, we make four recommendations for designers of context-sensitive adaptive reminder systems. Captured metrics suggest that adaptive reminders led to faster reaction times and were perceived by the participant as being more useful. CONCLUSIONS: The evaluation of context-sensitive systems that overlap into domestic lives is challenging. We believe that the ideal experiment is to deploy such systems in real homes and assess performance longitudinally. This case study in an instrumented live-in facility is a step toward that long-term goal.
Beaudin, J. S., S. S. Intille, E. Munguia Tapia, R. Rockinson and M. Morris (2007). "Context-sensitive microlearning of foreign language vocabulary on a mobile device." Proceedings of the European Ambient Intelligence Conference 2007 LNCS 4794: 55-72. Student lead author
We explore the use of ubiquitous sensing in the home for context-sensitive microlearning. To assess how users would respond to frequent and brief learning interactions tied to context, a sensor-triggered mobile phone application was developed, with foreign language vocabulary as the learning domain. A married couple used the system in a home environment, during the course of everyday activities, for a four-week study period. Built-in and stick-on multi-modal sensors detected the participants' interactions with hundreds of objects, furniture, and appliances. Sensor activations triggered the audio presentation of English and Spanish phrases associated with object use. Phrases were presented on average 57 times an hour; this intense interaction was found to be acceptable even after extended use. Based on interview feedback, we consider design attributes that may have reduced the interruption burden and helped sustain user interest, and which may be applicable to other context-sensitive, always-on systems.
Logan, B., J. Healey, M. Philipose, E. Munguia Tapia and S. S. Intille (2007). "A long-term evaluation of sensing modalities for activity recognition." Proceedings of the International Conference on Ubiquitous Computing LNCS 4717: 483--500.
We study activity recognition using 104 hours of annotated data collected from a person living in an instrumented home. The home contained over 900 sensor inputs, including wired reed switches, current and water flow inputs, object and person motion detectors, and RFID tags. Our aim was to compare different sensor modalities on data that approached "real world" conditions, where the subject and annotator were unaffiliated with the authors. We found that 10 infra-red motion detectors outperformed the other sensors on many of the activities studied, especially those that were typically performed in the same location. However, several activities, in particular "eating" and "reading" were difficult to detect, and we lacked data to study many fine-grained activities. We characterize a number of issues important for designing activity detection systems that may not have been as evident in prior work when data was collected under more controlled conditions.
Tapia, E. M., S. S. Intille, W. Haskell, K. Larson, J. Wright, A. King and R. Friedman (2007). "Real-time recognition of physical activities and their intensities using wireless accelerometers and a heart rate monitor." Proceedings of the Tenth IEEE International Symposium on Wearable Computers : ISWC 2006: 1-4. Student lead author. (ISWC) Ten-Year Impact Award
In this paper, we present a real-time algorithm for automatic recognition of not only physical activities, but also, in some cases, their intensities, using five triaxial wireless accelerometers and a wireless heart rate monitor. The algorithm has been evaluated using datasets consisting of 30 physical gymnasium activities collected from a total of 21 people at two different labs. On these activities, we have obtained a recognition accuracy performance of 94.6% using subject-dependent training and 56.3% using subjectindependent training. The addition of heart rate data improves subject-dependent recognition accuracy only by 1.2% and subject-independent recognition only by 2.1%. When recognizing activity type without differentiating intensity levels, we obtain a subjectindependent performance of 80.6%. We discuss why heart rate data has such little discriminatory power.
Tapia, E. M., S. S. Intille and K. Larson (2007). "Portable wireless sensors for object usage sensing in the home: Challenges and practicalities." Proceedings of the European Ambient Intelligence Conference 2007 LNCS 4794: 19-37. Student lead author
A low-cost kit of stick-on wireless sensors that transmit data indicating whenever various objects are being touched or used might aid ubiquitous computing research efforts on rapid prototyping, context-aware computing,and ultra-dense object sensing, among others. Ideally, the sensors would besmall, easy-to-install, and affordable. The sensors would reliably recognize when specific objects are manipulated, despite vibrations produced by the usage of nearby objects and environmental noise. Finally, the sensors would operate continuously for several months, or longer. In this paper, we discuss the challenges and practical aspects associated with creating such "object usage" sensors. We describe the existing technologies used to recognize object usage and then present the design and evaluation of a new stick-on, wireless object usage sensor. The device uses (1) a simple classification rule tuned to differentiate real object usage from adjacent vibrations and noise in real-time based on data collected from a real home, and (2) two complimentary sensors to obtain good battery performance. Results of testing 168 of the sensors in an instrumented home for one month of normal usage are reported as well as results from a 4-hour session of a person busily cooking and cleaning in the home, where every object usage interaction was annotated and analyzed.
Beaudin, J. S., S. S. Intille and M. E. Morris (2006). "To track or not to track: User reactions to concepts in longitudinal health monitoring." Journal of Medical Internet Research 8(4): e29. Student lead author
BACKGROUND: Advances in ubiquitous computing, smart homes, and sensor technologies enable novel, longitudinal health monitoring applications in the home. Many home monitoring technologies have been proposed to detect health crises, support aging-in-place, and improve medical care. Health professionals and potential end users in the lay public, however, sometimes question whether home health monitoring is justified given the cost and potential invasion of privacy. OBJECTIVE: The aim of the study was to elicit specific feedback from health professionals and laypeople about how they might use longitudinal health monitoring data for proactive health and well-being. METHODS: Interviews were conducted with 8 health professionals and 26 laypeople. Participants were asked to evaluate mock data visualization displays that could be generated by novel home monitoring systems. The mock displays were used to elicit reactions to longitudinal monitoring in the home setting as well as what behaviors, events, and physiological indicators people were interested in tracking. RESULTS: Based on the qualitative data provided by the interviews, lists of benefits of and concerns about health tracking from the perspectives of the practitioners and laypeople were compiled. Variables of particular interest to the interviewees, as well as their specific ideas for applications of collected data, were documented. CONCLUSIONS: Based upon these interviews, we recommend that ubiquitous "monitoring" systems may be more readily adopted if they are developed as tools for personalized, longitudinal self-investigation that help end users learn about the conditions and variables that impact their social, cognitive, and physical health.
Intille, S. S., K. Larson, E. Munguia Tapia, J. Beaudin, P. Kaushik, J. Nawyn and R. Rockinson (2006). "Using a live-in laboratory for ubiquitous computing research." Proceedings of PERVASIVE 2006 LNCS 3968: 349-365.
Ubiquitous computing researchers are increasingly turning to sensor-enabled "living laboratories" for the study of people and technologies in settings more natural than a typical laboratory. We describe the design and operation of the PlaceLab, a new live-in laboratory for the study of ubiquitous technologies in home settings. Volunteer research participants individually live in the PlaceLab for days or weeks at a time, treating it as a temporary home. Meanwhile, sensing devices integrated into the fabric of the architecture record a detailed description of their activities. The facility generates sensor and observational datasets that can be used for research in ubiquitous computing and other fields where domestic contexts impact behavior. We describe some of our experiences constructing and operating the living laboratory, and we detail a recently generated sample dataset, available online to researchers.
Munguia Tapia, E., S. S. Intille, L. Lopez and K. Larson (2006). "The design of a portable kit of wireless sensors for naturalistic data collection." Proceedings of PERVASIVE 2006 LNCS 3968: 117-134. Student lead author
In this paper, we introduce MITes, a flexible kit of wireless sensing devices for pervasive computing research in natural settings. The sensors have been optimized for ease of use, ease of installation, affordability, and robustness to environmental conditions in complex spaces such as homes. The kit includes six environmental sensors: movement, movement tuned for object-usage-detection, light, temperature, proximity, and current sensing in electric appliances. The kit also includes five wearable sensors: onbody acceleration, heart rate, ultra-violet radiation exposure, RFID reader wristband, and location beacons. The sensors can be used simultaneously with a single receiver in the same environment. This paper describes our design goals and results of the evaluation of some of the sensors and their performance characteristics. Also described is how the kit is being used for acquisition of data in non-laboratory settings where real-time multi-modal sensor information is acquired simultaneously from several sensors worn on the body and up to several hundred sensors distributed in an environment.
Nawyn, J., S. S. Intille and K. Larson (2006). "Embedding behavior modification strategies into consumer electronic devices." Proceedings of UbiComp 2006 LNCS 4206: 297-314. Student lead author
Ubiquitous computing technologies create new opportunities for preventive healthcare researchers to deploy behavior modification strategies outside of clinical settings. In this paper, we describe how strategies for motivating behavior change might be embedded within usage patterns of a typical electronic device. This interaction model differs substantially from prior approaches to behavioral modification such as CD-ROMs: sensor-enabled technology can drive interventions that are timelier, tailored, subtle, and even fun. To explore these ideas, we developed a prototype system namedViTo. On one level, ViTo functions as a universal remote control for a home entertainment system. The interface of this device, however, is designed in such a way that it may unobtrusively promote a reduction in the user's television viewing while encouraging an increase in the frequency and quantity of non-sedentary activities. The design of ViTo demonstrates how a variety of behavioral science strategies for motivating behavior change can be carefully woven into the operation of a common consumer electronic device. Results of an exploratory evaluation of a single participant using the system in an instrumented home facility are presented.
Ho, J. and S. S. Intille (2005). "Using context-aware computing to reduce the perceived burden of interruptions from mobile devices." Proceedings of CHI 2005 Connect: Conference on Human Factors in Computing Systems: 909 - 918. Student lead author
The potential for sensor-enabled mobile devices to proactively present information when and where users need it ranks among the greatest promises of ubiquitous computing. Unfortunately, mobile phones, PDAs, and other computing devices that compete for the user's attention can contribute to interruption irritability and feelings of information overload. Designers of mobile computing interfaces, therefore, require strategies for minimizing the perceived interruption burden of proactively delivered messages. In this work, a context-aware mobile computing device was developed that automatically detects postural and ambulatory activity transitions in real time using wireless accelerometers. This device was used to experimentally measure the receptivity to interruptions delivered at activity transitions relative to those delivered at random times. Messages delivered at activity transitions were found to be better received, thereby suggesting a viable strategy for context-aware message delivery in sensor-enabled mobile computing devices.
Intille, S. S., K. Larson, J. S. Beaudin, J. Nawyn, E. Munguia Tapia and P. Kaushik (2005). "A living laboratory for the design and evaluation of ubiquitous computing interfaces." Extended Abstracts of the 2005 Conference on Human Factors in Computing Systems: 1941 - 1944.
We introduce the PlaceLab, a new "living laboratory" for the study of ubiquitous technologies in home settings. The PlaceLab is a tool for researchers developing context-aware and ubiquitous interaction technologies. It complements more traditional data gathering instruments and methods, such as home ethnography and laboratory studies. We describe the data collection capabilities of the laboratory and current examples of its use.
Morris, M., S. S. Intille and J. S. Beaudin (2005). "Embedded Assessment: Overcoming barriers to early detection with pervasive computing." Proceedings of Pervasive 2005: 333-346.
Embedded assessment leverages the capabilities of pervasive computing to advance early detection of health conditions. In this approach, technologies embedded in the home setting are used to establish personalized baselines against which later indices of health status can be compared. Our ethnographic and concept feedback studies suggest that adoption of such health technologies among end users will be increased if monitoring is woven into preventive and compensatory health applications, such that the integrated system provides value beyond assessment. We review health technology advances in the three areas of monitoring, compensation, and prevention. We then define embedded assessment in terms of these three components. The validation of pervasive computing systems for early detection involves unique challenges due to conflicts between the exploratory nature of these systems and the validation criteria of medical research audiences. We discuss an approach for demonstrating value that incorporates ethnographic observation and new ubiquitous computing tools for behavioral observation in naturalistic settings such as the home.
Patrick, K., S. Intille and M. Zabinski (2005). "An ecological framework for cancer communication: Implications for research." Journal of Medical Internet Research 7(3): e23.
The field of cancer communication has undergone a major revolution as a result of the Internet. As recently as the early 1990s, face-to-face, print, and the telephone were the dominant methods of communication between health professionals and individuals in support of the prevention and treatment of cancer. Computer-supported interactive media existed, but this usually required sophisticated computer and video platforms that limited availability. The introduction of point-and-click interfaces for the Internet dramatically improved the ability of non-expert computer users to obtain and publish information electronically on the Web. Demand for Web access has driven computer sales for the home setting and improved the availability, capability, and affordability of desktop computers. New advances in information and computing technologies will lead to similarly dramatic changes in the affordability and accessibility of computers. Computers will move from the desktop into the environment and onto the body. Computers are becoming smaller, faster, more sophisticated, more responsive, less expensive, and\--essentially\--ubiquitous. Computers are evolving into much more than desktop communication devices. New computers include sensing, monitoring, geospatial tracking, just-in-time knowledge presentation, and a host of other information processes. The challenge for cancer communication researchers is to acknowledge the expanded capability of the Web and to move beyond the approaches to health promotion, behavior change, and communication that emerged during an era when language- and image-based interpersonal and mass communication strategies predominated. Ecological theory has been advanced since the early 1900s to explain the highly complex relationships among individuals, society, organizations, the built and natural environments, and personal and population health and well-being. This paper provides background on ecological theory, advances an Ecological Model of Internet-Based Cancer Communication intended to broaden the vision of potential uses of the Internet for cancer communication, and provides some examples of how such a model might inform future research and development in cancer communication.
Bao, L. and S. S. Intille (2004). "Activity recognition from user-annotated acceleration data." Pervasive Computing LNCS 3001(3001): 1-17. Student lead author. Ubicomp/Pervasive 10 Year Impact Award
In this work, algorithms are developed and evaluated to detect physical activities from data acquired using .ve small biaxial accelerometers worn simultaneously on di.erent parts of the body. Acceleration data was collected from 20 subjects without researcher supervision or observation. Subjects were asked .rst to perform a sequence of everyday tasks but not told speci.cally where or how to do them. Many tasks were performed outside of the laboratory setting. Mean, energy, frequency-domain entropy, and correlation of acceleration data was calculated, and decision table, nearest neighbor, decision tree, and Naive Bayesian classi.ers were tested on these features using leave-one-subject-out validation. Decision tree classi.ers showed the best performance recognizing everyday activities such as walking, watching TV, and vacuuming with an overall accuracy rate of 84%. The classi.er captures conjunctions in acceleration feature values that e.ectively discriminate activities. This is the .rst work to investigate performance of recognition algorithms with multiple accelerometers on 20 activities using datasets annotated by the subjects themselves. We also show that with just two biaxial accelerometers -- thigh and wrist -- the recognition rate dropped only 3.3%.
Beaudin, J. S., E. Munguia Tapia and S. S. Intille (2004). "Lessons learned using ubiquitous sensors for data collection in real homes." Extended Abstracts of the 2004 Conference on Human Factors in Computing Systems: 1359-1362. Student lead author
Interface design for the home requires a realistic understanding of the complexity and richness of the human activities that go on there; it is our goal to develop tools that enable HCI investigation in actual home environments. We have developed a kit of ubiquitous sensing devices and over the past year have conducted a series of studies installing a large number of sensors, of diverse types, in multiple homes of participants not affiliated with the research team. As we deployed our portable kit outside the laboratory, we encountered unanticipated study design and technology requirements that will affect the continued development of the kit itself. We offer practical tips we have learned from our experience and describe how we are applying them to the design of our next generation of sensors.
Intille, S. S. (2004). "A new research challenge: Persuasive technology to motivate healthy aging." Transactions on Information Technology in Biomedicine 8(3): 235-237.
Healthcare systems in developed countries are experiencing severe financial stress as age demographics shift upward, leading to a larger percentage of older adults needing care. One way to potentially reduce or slow spiraling medical costs is to use technology, not only to cure sickness, but also to promote well- ness throughout all stages of life, thereby avoiding or deferring expensive medical treatments. Ubiquitous computing and context- aware algorithms offer a new healthcare opportunity and a new set of research challenges: exploiting emerging consumer electronic devices to motivate healthy behavior as people age by presenting "just-in-time" information at points of decision and behavior.
Intille, S. S. (2004). "Ubiquitous computing technology for just-in-time motivation of behavior change." Proceedings of Medinfo 11(Pt) 2: 1434-1437.
This paper describes a vision of health care where "just-in-time" user interfaces are used to transform people from passive to active consumers of health care. Systems that use computational pattern recognition to detect points of decision, behavior, or consequences automatically can present motivational messages to encourage healthy behavior at just the right time. Further, new ubiquitous computing and mobile computing devices permit information to be conveyed to users at just the right place. In combination, computer systems that present messages at the right time and place can be developed to motivate physical activity and healthy eating. Computational sensing technologies can also be used to measure the impact of the motivational technology on behavior.
Intille, S. S., L. Bao, E. Munguia Tapia and J. Rondoni (2004). "Acquiring in situ training data for context-aware ubiquitous computing applications." Proceedings of CHI 2004 Connect: Conf. on Human Factors in Computing Systems: 1-9.
Ubiquitous, context-aware computer systems may ultimately enable computer applications that naturally and usefully respond to a user's everyday activity. Although new algorithms that can automatically detect context from wearable and environmental sensor systems show promise, many of the most flexible and robust systems use probabilistic detection algorithms that require extensive libraries of training data with labeled examples. In this paper, we describe the need for such training data and some challenges we have identified when trying to collect it while testing three context-detection systems for ubiquitous computing and mobile applications.
Munguia Tapia, E., S. S. Intille and K. Larson (2004). "Activity recognition in the home setting using simple and ubiquitous sensors." Proceedings of PERVASIVE 2004 LNCS 3001: 158-175. Student lead author. Ubicomp/Pervasive 10 Year Impact Award
In this work, a system for recognizing activities in the home setting using a set of small and simple state-change sensors is introduced. The sensors are designed to be "tape on and forget" devices that can be quickly and ubiquitously installed in home environments. The proposed sensing system presents an alternative to sensors that are sometimes perceived as invasive, such as cameras and microphones. Unlike prior work, the system has been deployed in multiple residential environments with non-researcher occupants. Preliminary results on a small dataset show that it is possible to recognize activities of interest to medical professionals such as toileting, bathing, and grooming with detection accuracies ranging from 25% to 89% depending on the evaluation criteria used.
Munguia Tapia, E., N. Marmasse, S. S. Intille and K. Larson (2004). "MITes: Wireless portable sensors for studying behavior." Proceedings of Extended Abstracts Ubicomp 2004: Ubiquitous Computing. Student lead author
We present MITes (MIT Environmental used Sensors): in two ways: a portable kit of ubiquitous wireless sensing devices for realtime data collection of human activities in natural settings. The sensors designed to be (1) determining people's interaction with objects in the environment, and (2) measuring acceleration on different parts of the body. The sensors have been designed to permit low-cost research studies where data is acquired simultaneously from hundreds of objects in an environment and multiple parts of the body.
Intille, S. S., K. Larson and E. M. Tapia (2003). "Designing and evaluating technology for independent aging in the home." Proceedings of the International Conference on Aging, Disability and Independence.
We are developing technology and design strategies to support aging in place. To design and evaluate our solutions, we have created new tools that can be used to study behavior in actual homes. Our goal is to evaluate solutions in context, taking into account the complexity of real-world behavior that is often missing from laboratory environments. Two of our tools are described here: a portable kit of tape-on sensors for studying behavior in existing homes, and the PlaceLab, a residential observational facility with ubiquitous sensing technology. In both cases, the tools are used to study behavior outside of a typical laboratory in a natural home setting. We propose one scenario that illustrates how the portable sensors can be used to create a new device to support aging in place. We then discuss how the portable sensor kit and the PlaceLab can be used to study the technical and social challenges that must be overcome to realize the scenario.
Intille, S. S., V. Lee and C. Pinhanez (2003). "Ubiquitous computing in the living room: Concept sketches and an implementation of a persistent user interface." Adjunct Proceedings of the Ubicomp 2003 Video Program: 265-266.
This video shows some concept sketches of applications that might be created for a living room with ubiquitous display and laser pointer interaction technology. A fully-functioning prototype of a persistent interface is also described: a language-learning tool.
Intille, S. S., E. Munguia Tapia, J. Rondoni, J. Beaudin, C. Kukla, S. Agarwal, L. Bao and K. Larson (2003). "Tools for studying behavior and technology in natural settings." Proceedings of UbiComp 2003: Ubiquitous Computing LNCS 2864: 157-174.
Three tools for acquiring data about people, their behavior, and their use of technology in natural settings are described: (1) a context-aware experience sampling tool, (2) a ubiquitous sensing system that detects environmental changes, and (3) an image-based experience sampling system. We discuss how these tools provide researchers with a flexible toolkit for collecting data on activity in homes and workplaces, particularly when used in combination. We outline several ongoing studies to illustrate the versatility of these tools. Two of the tools are currently available to other researchers to use.
Intille, S. S., J. Rondoni, C. Kukla, I. Anacona and L. Bao (2003). "A context-aware experience sampling tool." Proceedings of CHI '03 Extended Abstracts on Human Factors in Computing Systems: 972-973.
A new software tool for user-interface development and assessment of ubiquitous computing applications is available for CHI researchers. The software permits researchers to use common PDA mobile computing devices for experience sampling studies. The basic tool offers options not currently available in any other open-source sampling package. However, the tool also has new functionality: context-aware experience sampling. This feature permits researchers to acquire feedback from users in particular situations that are detected by sensors connected to a mobile computing device.
Munguia Tapia, E., S. S. Intille, J. Rebula and S. Stoddard (2003). "Concept and partial prototype video: Ubiquitous video communication with the perception of eye contact." Proceedings of the Ubicomp 2003 Video Program. Student lead author
This concept and partial prototype video introduces a strategy for creating a video conferencing system for future ubiquitous computing environments that can guarantee two remote conversants the ability to establish eye contact. Unlike prior work, eye contact can be achieved even as people move about their respective environments engaging in everyday tasks.
Intille, S. S. (2002). "Change blind information display for ubiquitous computing environments." Proceedings of the Ubicomp 2002: Ubiquitous Computing LNCS 2498: 91-106.
Occupants of future computing environments with ubiquitous display devices may feel that they are in a space where they are surrounded with continuously changing digital information. One solution is to create a reasoning module that accepts requests to display information from multiple applications and controls how the information is presented to minimize visual disruptions to users. Such a system might use information about what activity is occurring in the space to exploit a powerful phenomena of the human visual system: change blindness.
Intille, S. S. (2002). "Designing a home of the future." IEEE Pervasive Computing 1(2): 80-86.
An interdisciplinary team is developing technologies and design strategies that use context-aware sensing to empower people by presenting information at precisely the right time and place. The team is designing a living laboratory to study technology that motivates behavior change in context.
Intille, S. S., C. Kukla and X. Ma (2002). "Eliciting user preferences using image-based experience sampling and reflection." Proceedings of the CHI '02 Extended Abstracts on Human Factors in Computing Systems: 738-739.
Determining requirements for any design project involves identifying and ranking user needs and preferences. User needs are typically elicited via personal or focus group interviews, site visits, and photographic and video analysis. Often, however, users know more than they say in a single or even several interviews \[1\]. We propose a methodology for assisting a user who is interested in learning about his or her own preferences using a process we call image-based experience sampling and reflection. We describe the methodology using a storyboard example from the domain of architectural redesign of home environments.
Intille, S. S. and A. F. Bobick (2001). "Recognizing planned, multi-person action." Computer Vision and Image Understanding (1077-3142) 81(3): 414-445.
Multiperson action recognition requires models of structured interaction between people and objects in the world. This paper demonstrates how highly structured, multiperson action can be recognized from noisy perceptual data using visually grounded goal-based primitives and low-order temporal relationships that are integrated in a probabilistic framework. The representation, which is motivated by work in model-based object recognition and probabilistic plan recognition, makes four principal assumptions: (1) the goals of individual agents are natural atomic representational units for specifying the temporal relationships between agents engaged in group activities, (2) a high-level description of temporal structure of the action using a small set of low-order temporal and logical constraints is adequate for representing the relationships between the agent goals for highly structured, multiagent action recognition, (3) Bayesian networks provide a suitable mechanism for integrating multiple sources of uncertain visual perceptual feature evidence, and (4) an automatically generated Bayesian network can be used to combine uncertain temporal information and compute the likelihood that a set of object trajectory data is a particular multiagent action. The recognition method is tested using a database of American football play descriptions and manually acquired but noisy player trajectories. The strengths and limitations of the system are discussed and compared with other multiagent recognition algorithms.
Pinhanez, C. S., J. W. Davis, S. S. Intille, M. Johnson, A. Wilson, A. F. Bobick and B. Blumberg (2000). "Physically interactive story environments." IBM Systems Journal 39(3/4): 438-455.
Most interactive stories, such as hypertext narratives and interactive movies achieve an interactive "feel" by allowing the user to choose among multiple story paths. In this paper we discuss physically interactive environments with narrative structure in which the ability to choose among multiple story lines is replaced with having users, first, interact with the story characters in small, local "windows" of the narrative and, second, actively engage their bodies in movement, In particular, we found that compelling interactive narrative story systems can be perceived as highly responsive, engaging, and interactive even when the overall story has a single-path structure, in what we call a "less-choice, more-responsiveness" approach to the design of story-based interactive environments. We have also observed that unencumbering, rich sensor technology can facilitate user immersion in the experience as the story progresses\--users can act as they typically would without worrying about manipulating a computer interface. To support these arguments, the paper describes the physical setup, the interactive story, the technology, and the user experience of four projects developed at the MIT Media Laboratory: KidsRoom, It/I, Personal Aerobics Trainer, and Swamped!
Bobick, A. F. and S. S. Intille (1999). "Large occlusion stereo." International Journal of Computer Vision 33(3): 181-200.
A method for solving the stereo matching problem in the presence of large occlusion is presented. A data structure---the disparity space image---is defined to facilitate the description of the effects of occlusion on the stereo matching process and in particular on dynamic programming (DP) solutions that find matches and occlusions simultaneously. We significantly improve upon existing DP stereo matching methods by showing that while some cost must be assigned to unmatched pixels, sensitivity to occlusion-cost and algorithmic complexity can be significantly reduced when highly-reliable matches, or ground control points, are incorporated into the matching process. The use of ground control points eliminates both the need for biasing the process towards a smooth solution and the task of selecting critical prior probabilities describing image formation. Finally, we describe how the detection of intensity edges can be used to bias the recovered solution such that occlusion boundaries will tend to be proposed along such edges, reflecting the observation that occlusion boundaries usually cause intensity discontinuities.
Bobick, A. F., S. S. Intille, J. W. Davis, F. Baird, L. W. Campbell, Y. Ivanov, C. S. Pinhanez, A. Schütte and A. Wilson (1999). "The KidsRoom: A perceptually-based interactive and immersive story environment." PRESENCE: Teleoperators and Virtual Environments 8(4): 367-391.
The KidsRoom is a perceptually-based, interactive, narrative playspace for children. Images, music, narration, light, and sound effects are used to transform a normal child's bedroom into a fantasy land where children are guided through a reactive adventure story. The fully automated system was designed with the following goals: (1) to keep the focus of user action and interaction in the physical and not virtual space; (2) to permit multiple, collaborating people to simultaneously engage in an interactive experience combining both real and virtual objects; (3) to use computer-vision algorithms to identify activity in the space without requiring the participants to wear any special clothing or devices; (4) to use narrative to constrain the perceptual recognition, and to use perceptual recognition to allow participants to drive the narrative; and (5) to create a truly immersive and interactive room environment. We believe the KidsRoom is the first multi-person, fully-automated, interactive, narrative environment ever constructed using non-encumbering sensors. This paper describes the KidsRoom, the technology that makes it work, and the issues that were raised during the system's development.1 A demonstration of the project, which complements the material presented here and includes videos, images, and sounds from each part of the story
Intille, S. S. and A. F. Bobick (1999). "A framework for recognizing multi-agent action from visual evidence." Proceedings of the Sixteenth National Conference on Artificial Intelligence: 518-525.
A probabilistic framework for representing and visually recognizing complex multi-agent action is presented. Motivated by work in model-based object recognition and designed for the recognition of action from visual evidence, the representation has three components: (1) temporal structure descriptions representing the temporal relationships between agent goals, (2) belief networks for probabilistically representing and recognizing individual agentgoals from visual evidence,and (3) belief networks automatically generated from the temporal structure descriptions that support the recognition of the complex action. We describe our current work on recognizing American football plays from noisy trajectory data.
Intille, S. S. and A. F. Bobick (1999). "Visual recognition of multi-agent action using binary temporal relations." Proceedings of the 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition 1: 56-62.
A probabilistic framework for representing and visually recognizing complex multi-agent action is presented. Motivated by work in model-based object recognition and designed for the recognition of action from visual evidence, the representation has three components: (1) temporal structure descriptions representing the temporal relationships between agent goals, (2) belief networks for probabilistically representing and recognizing individual agent goals from visual evidence, and (3) belief networks automatically generated from the temporal structure descriptions that support the recognition of the complex action. We describe our current work on recognizing American football plays from noisy trajectory data
Bobick, A. F., S. S. Intille, J. W. Davis, F. Baird, L. W. Campbell, Y. Ivanov, C. S. Pinhanez, A. Schütte and A. Wilson (1998). "Design decisions for interactive environments: Evaluating the KidsRoom." Proceedings of the AAAI Spring Symposium on Intelligent Environments Technical Report SS-98-02: 7-16.
We believe the KidsRoom is the first multi-person,fully-automated, interactive, narrative environment ever constructed using non-encumbering sensors. The perceptual system that drives the KidsRoom is outlined elsewhere(Bobick et al. 1996). This paper describes our design goals, successes, and failures including several general observations that may be of interest to other designers of perceptually-based interactive environments
Intille, S. S., J. Davis and A. Bobick (1997). "Real-time closed-world tracking." Proceedings of IEEE Conference on Computer Vision and Pattern Recognition: 697-703.
A real-time tracking algorithm that uses contextual information is described. The method is capable of simultaneously tracking multiple, non-rigid objects when erratic movement and object collisions are common. A closedworld assumption is used to adaptively select and weight image features used for correspondence. Results of algorithm testing and the limitationsof the method are discussed. The algorithm has been used to track children in an interactive, narrative playspace.
Intille, S. S. and A. F. Bobick (1995). "Closed-world tracking." Proceedings of the Fifth International Conference on Computer Vision: 672-678.
A new approach to tracking weakly modeled objects in a semantically rich domain is presented. We dejine a closed-world as a space-time region of an image sequence in which the complete taxonomy of objects is known, and in which each pixel should be explained as belonging to one of those objects. Given contextual object information, context-specific features can be dynamically selected as the basis for tmcking. A context-specijic feature is one that has been chosen based upon the context to maximize the chance of successful tmcking between fmmes. Our work is motivated by the goal of video annotation the semi-automutic genemtion of symbolic descriptions of action taking place in a contentually-rich dynamic scene. We describe how contextual knowledge in the 'yootball domain '' can be applied to closed-world football player tracking and present the details of our implementation. We include tracking results based on hundreds of images that demonstrate the wide mnge of tmcking situations the algorithm will successfully handle as well as a fa0 examples of where the algorithm fails.
Intille, S. S. and A. F. Bobick (1994). "Disparity-space images and large occlusion stereo." Proceedings of the Third European Conference on Computer Vision: 179-186.
A new method for solving the stereo matching problem in the presence of large occlusion is presented. A data structure \-- the disparity space image \-- is defined in which we explicitly model the effects of occlusion regions on the stereo solution. We develop a dynamic programming algorithm that finds matches and occlusions simultaneously. We show that while some cost must be assigned to unmatched pixels, our algorithm's occlusion-cost sensitivity and algorithmic complexity can be significantly reduced when highly-reliable matches, or ground control points, are incorporated into the matching process. The use of ground control points eliminates both the need for biasing the process towards a smooth solution and the task of selecting critical prior probabilities describing image formation.
Intille, S. S. and A. F. Bobick (1994). "Incorporating intensity edges in the recovery of occlusion regions." Proceedings of the 12th International Conference on Pattern Recognition 1: 674-677.
A method for incorporating intensity edge information into the recovery of occlusion regions using a pixel-based stereo algorithm is presented. We review the construction of disparity-space images and their use to solve the stereo occlusion problem. We show the relationship between intensity edges and the disparity space images, and extend our stereo technique to use information about intensity discontinuities at occlusion edges. The combination of ground control points \[6\] and edge information yields excellent occlusion regions.
