ECE Special Seminar: “Bilevel Optimization for Agentic AI: From Theory to Retrieval-augmented Agentic Reinforcement Learning”

Presented by: Quan Xiao, PhD Candidate @ Cornell University

Description: “Agentic AI systems combine large language models with system-level design choices such as post-training data selection, retrieval, tool use, and memory management in augmented inference. Training these systems involves multiple coupled objectives beyond task success, including safety, retrieval quality, and adaptation efficiency. Sequential tuning can cause catastrophic forgetting, while direct objective mixing training can introduce unit mismatch and obscure credit assignments across components.

In this talk, I will present bilevel optimization as a framework for jointly learning LLM parameters and agentic-system design variables. I will introduce an efficient first-order penalty formulation that replaces lower-level optimality with a merit constraint, avoiding Hessian inverses and differentiation through full inner trajectories. By turning the bilevel problem into a penalized objective with analyzable structure, this formulation enables landscape analysis that leads to global convergence guarantees.

I will conclude with an application to bilevel retrieval optimization for tool-using agents, where retrieval enables LLMs to access up-to-date information and manage external memory. In this setting, landscape analysis also guides the design of memory-efficient variants of the penalty method tailored to the application. Experiments on multi-hop and medical question-answering tasks demonstrate the effectiveness of jointly optimizing retrieval and the LLM. Using only a 3B LLM backbone, the proposed approach outperforms direct inference with GPT-5.6 Luna, as well as methods based solely on agentic reinforcement learning or retrieval optimization.”

Bio: Quan Xiao (https://jenniferquanxiao.github.io) is a Ph.D. candidate in the Department of Electrical and Computer Engineering at Cornell University. She received her M.S. in Electrical Engineering from Rensselaer Polytechnic Institute in 2023, and her B.S. in statistics from the University of Science and Technology of China in 2020. Her research focuses on the theoretical foundations of bilevel and multi-objective optimization, as well as their applications to generative, agentic AI and hardware-algorithm co-design. Her work has appeared in leading machine learning and signal processing venues, including NeurIPS, ICML, and IEEE Transactions on Signal Processing, with papers selected for oral and spotlight presentations.

Quan has received several awards, including the IEEE Signal Processing Society Scholarship in 2024 and the Belsky Award for Computational Sciences and Engineering in Rensselaer Polytechnic Institute in 2023. Her research has also received supports from industries, including Google and Lambda Research.

Hosted by: Professor Yu Zhang, Electrical & Computer Engineering Department

When: Friday, October 30th from 10:30AM to 12:00PM

Location: Engineering 2, Room 506