Email: 

rabelea(at)b-tu.de

Research topic:

Empathic virtual agents in virtual reality

Abstract:

Over the past decades, interaction between humans and machines has become increasingly seamless. With the development of large-language models (LLM), such as ChatGPT, human-to-human-like dialogue has become a reality. However, factual information, i.e. the content of speech, is only one cornerstone of human communication. People forge an intuitive, empathic understanding of their interlocutors, based on cues like gestures or facial expressions. This kind of understanding is still lacking in recent human-computer interaction (HCI) systems. The core of my research is to bridge this gap by integrating neurophysiological measures into an LLM. This should allow the machine to ‘understand’ more than just the speech content in an HCI scenario, mimicking the intuitive comprehension that humans have of one another. The LLM will be embedded in a virtual agent, an entity with which the participant interacts, inside a virtual reality (VR) scenario, thereby creating a fully immersive HCI experience. It will be investigated which neurophysiological input streams, e.g. heart rate variability or brain signals, best improve the interaction between human and virtual agent. In a user-centered approach, subjective questionnaires will help us to understand in which scenario the participant feels most understood or supported and how trust into the system is affected by these additional input streams.  

Short bio:

During her Bachelor of Psychology at OvGU Magdeburg, Lea Rabe found her passion to understand the human brain. Already back then she investigated the processing of emotion in the brain in an EEG study for her Bachelor thesis. During her Master’s in Human Factors at TU Berlin, an interdisciplinary study program for engineers and psychologists, she specialized in biological psychology, statistics and user-centered design. Lea Rabe worked in a study on measuring mental workload with neurophysiological signals in air traffic controllers by the Federal Institute for Occupational Safety and Health (BAuA) and the German Aerospace Center (DLR). Following that she wanted to keep on investigating the brain under more realistic conditions, which was the focus of Prof. Klaus Gramann’s Mobile Brain-Body Imaging (MoBI) laboratory at TU Berlin. There she eventually started her master thesis in a study by Dr. Marius Klug investigating brain potentials in a full-body movement paradigm in virtual reality. Lea Rabe is now working on expanding that research in her PhD at BTU Cottbus-Senftenberg.

Research topic:

Multimodal Classification of Users' Mental States Based on Physiological Data inVirtual Environments

Abstract:

AI systems currently rely on explicit feedback or behavior to understand users. What if an AI model could use physiological data, such as EEG, eye gaze, heart rate, or facial expression, to recognize a user's mental state and adapt accordingly? This research aims to explore the feasibility of such an approach, which could create a more human-centric AI and enhance the human-computer interaction experience. The project involves designing and building suitable VR-based experimental paradigms to collect and analyze multimodal physiological and behavioral data. Initial offline analysis will be performed to establish a proof of concept, which will then be extended to real-time applications. The ultimate goal is to apply these findings to practical settings, such as neurofeedback-based VR games, to improve the interactivity and responsiveness of AI systems in virtual environments.

Short bio:

Yanzhao Pan completed a Bachelor of Engineering in Automotive Engineering from 2014 to 2018. After that, he pursued a Master of Science in Human Factors at Technische Universität Berlin from 2020 to 2022. During his master's study, he delved into brain-computer interfaces, focusing on their integration with eye tracking and hand motion in virtual reality for his master's thesis. Following this, Yanzhao gained practical experience as an intern at Zanderlabs in 2022, mainly working with EEG data analysis and PPG-based heart rate analysis. In May 2023, he began his PhD at Brandenburgische Technische Universität Cottbus-Senftenberg, researching the multimodal classification of users' mental states based on physiological data in virtual environments.