Zhu, R. (ECE) – From Neuromorphic Principles to Efficient Neural Language Architectures

This dissertation investigates how neuromorphic and brain-inspired principles can guide the design of efficient neural language architectures. It addresses two central limitations of modern Transformer-based language models: memory growth with context length and high computational cost from dense matrix multiplication. Through studies of spiking neural networks, linear-recurrent language models, hybrid attention architectures, MatMul-free models, and looped language models, the dissertation develops practical approaches for bounded-memory and bounded-compute language modeling. The central conclusion is that recurrent state, temporal decay, sparse computation, and parameter reuse can provide useful design principles for scalable language models, even when they are abstracted beyond literal biological spiking.

Event Host: Ridger Zhu, Ph.D. Candidate, Electrical & Computer Engineering 

Advisor: Jason Eshraghian

Zoom: https://ucsc.zoom.us/j/96672322005?pwd=3MSitgbm5WboIENbf1hKpxwXnt9VXh.1