multimodal data across the physical and the virtual