Context Matters: Understanding the Effect of Usage Contexts on Users’ Modality Selection in Multimodal Systems

Users often select different modalities in multimodal systems based on context and function. This study aimed to investigate the impact of context on users’ modality selection in a multimodal system. A modality selection task was presented through the developed multimodal system, where the task’s difficulty was controlled based on the number of syllables and touches. The experiments were examined in four usage contexts (baseline, watching, reading, and driving), and the selected modality, physical and mental effort, and satisfaction were measured. A regression model for predicting modality selection was established using binomial logistic regression. The results showed that voice usage increased in the order of baseline, watching, reading, and driving contexts. It was revealed that these changes were attributable to differences in physical and mental interaction efforts of modalities according to contexts. Our finding provides valuable insights into users’ modality selection in different contexts, which could inform the design of more efficient multimodal systems.