Castro, S. (CSE) – Agentic AI for Security: Adversarial Foundations for Autonomous Cyber Operations

Autonomous Cyber Operations (ACO) agents promise effective security automation with minimal human intervention, yet their deployment raises three interconnected challenges: agents must be realistic (reproducing diverse attacker sophistication), secure (preventing autonomy from becoming an attack surface), and feasible (safely replicating human behavior at full autonomy).

We argue that these three properties are requirements for ACO agents. Existing approaches do not address them together and lack diverse adversarial coverage, formal threat models for attacks against the agents themselves, and systematic evaluation of multi-agent topologies.

We advance all three ACO properties: (1) For realism, we establish adversarial foundations by discovering Windows OS vulnerabilities and releasing two exploits reliable across XP through 11. (2) For security, we formalize ACO meta-attacks and meta-defenses, propose the first invariant-based Meta-IDS detecting both sensor and actuator meta-attacks, and introduce the first hybrid LLM–RL ACO integration for defense with a novel inter-agent communication protocol. (3) For feasibility, we present MaLO, the first dynamic-topology multi-agent ACO system, achieving a 78.6\% success rate across a new 42-task security benchmark and solving operations up to 40× faster than human experts. We further propose the Security Operation Complexity Index (SOCX) classification and the T×V×O taxonomy as the first objective-driven evaluation methodology for coding-agent attacks.

Together, these contributions demonstrate that ACO agents can match real-world adversarial sophistication, resist meta-attacks, and outperform human operators on complex security tasks. Open challenges remain in adaptive adversaries, LLM–RL co-training, dynamic topology selection, and deployment beyond simulated environments.

Event Host:  Sebastián R. Castro, PhD Candidate, Computer Science & Engineering

Advisor: Alvaro A. Cárdenas

Zoom: https://ucsc.zoom.us/j/2267557290?pwd=S0dNTTV3emZGUzlqV3dLbTg3a0NFUT09&omn=92791061627

Passcode: G20c06