Scott, J. (CSE) – Mechanistic Specialization Does Not Guarantee Performance: Evidence from Dual AttentionTransformers

Dual Attention Transformers (DATs) extend decoder-only Transformers with a dedicated relational-attention stream, making them a natural architecture for abstract identity rules such asABA and ABB. Surprisingly, we find that comparably sized GPT-2 models outperform DATs on these tasks. We investigate this gap with two complementary mechanistic analyses. First, causal mediation analysis shows that DATs exhibit stronger evidence of hypothesized symbolic mechanisms: symbol abstraction, symbol induction, and retrieval, than GPT-2. Second, a routing analysis shows why this specialization does not translate into better behavior: DATs make more wrong-copy errors, can attend to the correct source token while still predicting the wrong token, and show weak direct contribution from relational attention to the correct-versus-wrong outputmargin. Ablating positive-routing heads hurts performance, while amplifying those headsimproves DAT more than matched controls. These results show that explicit relational attentioncan shape internal organization without guaranteeing task success. For identity-rule tasks, performance depends not only on whether relational information is represented, but whether it is routed to the final output position in a form that affects the next-token prediction. Because pretrained DAT and GPT-2 models differ in training data, tokenizer, and other implementation details, these findings should be interpreted as evidence about the mechanisms used by existing models rather than as a definitive architectural comparison. Follow-up experiments will address these confounders through controlled training comparisons that match data, scale, and evaluation conditions across architectures.

Event Host: Jonathan Scott, Ph.D. Student, Computer Science & Engineering

Advisor: Leilani Gilpin

Zoom: https://ucsc.zoom.us/j/95404396322?pwd=0e0AegKSxhcFDDKrn08muHcqfHs6WW.1

Passcode: 985103