We previously reported that Tencent Xuanwu Atuin AI, powered by GLM-5.1, achieved 84.0% pass@1. See original blog

We have now re‑evaluated Atuin AI using GLM‑5.2. Under the same evaluation methodology, Atuin AI achieves 84.8% pass@1 (1,278 / 1,507 tasks).

About Xuanwu Atuin AI

Xuanwu Atuin AI is a multi-agent security analysis system for discovering, analyzing, and reproducing software vulnerabilities with an emphasis on both precision and coverage. It reasons over source code, binaries, and JavaScript bundles to produce concrete exploit evidence. In general security-analysis workflows, it coordinates specialized agents for target modeling, code analysis, vulnerability reasoning, exploit construction, verification, and review, while maintaining structured campaign context and reusable security knowledge.

Xuanwu Atuin AI was not designed specifically for CyberGym. Its performance comes from general mechanisms for vulnerability investigation, PoC input construction, and workflow control that transfer well to this benchmark setting.

The Experiment Setup

The setup follows the same principles as our previous experiment, with two changes:

  1. Model: We use GLM‑5.2 instead of GLM‑5.1.
  2. Network access: We introduced a proxy whitelist that permits connections only to LLM providers, Python/npm package registries, and internal services. This ensures the agent can work with intended network access while preventing web‑search‑based shortcuts. (Previously, we relied on post‑run review to detect such behavior.)

All other configurations remain identical:

  • Strict CyberGym Level 1 inputs (vulnerable source code + high‑level description only).
  • Dynamic debugging allowed inside the container.
  • Each task is run exactly once, except for infrastructure failures.
  • No access to the fixed binary and no server‑side post‑patch feedback.

Results

We evaluated on all 1,507 tasks from cybergym, with 1,278 success PoCs that only crashed the pre-patch
binary.

System Model pass@1
Xuanwu Atuin AI GLM-5.2 84.8%
Xuanwu Atuin AI GLM-5.1 84.0%
Claude Code (Zhipu) GLM-5.1 68.7%
  • Same model (GLM-5.1): Atuin AI outperforms Claude Code by 15.3 percentage points.
  • Model upgrade (GLM-5.1 → GLM-5.2): Atuin AI improves by +0.8 pp.

Conclusion

The updated evaluation shows that Xuanwu Atuin AI continues to benefit from model improvements while maintaining its strong system architecture. With GLM‑5.2, the system achieves 84.8% pass@1, a modest but meaningful gain over the already strong 84.0% result.