Fan, Y. (CSE) – Building Human-Centered Multimodal AI Agents

As multimodal artificial intelligence systems become increasingly embedded in everyday technology, there is a growing need to design human-centered AI agents that support and amplify human capabilities rather than replace them. This dissertation investigates how to build human-centered multimodal AI agents, framing human-centeredness as an agent-level objective that requires both accessible, assistive interaction and reliable, trustworthy behavior across physical and digital environments. This dissertation explores two complementary dimensions of human-centered agent design. The first focuses on enhancing accessibility through conversational and interactive agents that assist users in everyday tasks. We study both embodied and digital settings in which agents reduce physical and cognitive burdens via natural language interaction, including hands-free drone control, navigation assistance in unfamiliar environments, and interactive access to complex graphical user interfaces. The second dimension focuses on strengthening agent capability to improve reliability and trust. We investigate how agents can acquire environment-specific knowledge through autonomous exploration and how they can reason about visual information in a grounded and transparent manner, drawing inspiration from human learning and reasoning behaviors.

Event Host: Yue Fan, Ph.D. Candidate, Computer Science and Engineering

Advisor: Xin Eric Wang

Zoom- https://ucsc.zoom.us/j/99619642071?pwd=dwWOlkJxjbamgpB4IbRxYDXbngqXOE.1

Passcode- 467959