Is "Vibe Coding" Safe? The Data Says No.

[cite_start]"Vibe coding"—the growing trend where engineers let AI agents handle complex coding tasks with minimal supervision—has taken the software world by storm[cite: 10, 56]. [cite_start]But a new study from Carnegie Mellon University asks the question few have stopped to consider: Is it actually safe to deploy? [cite: 11]

The answer, according to the newly released SUSVIBES benchmark, is a resounding warning. While AI agents are becoming excellent at making code work, they are terrible at making it secure.

The Study: SUSVIBES

[cite_start]Researchers from CMU and Columbia University introduced SUSVIBES (Security-Oriented Software Engineering Benchmark), a dataset of 200 real-world feature requests mined from open-source projects[cite: 12]. [cite_start]Unlike previous benchmarks that looked at single files, SUSVIBES tests full repository-level tasks involving complex file structures and multiple edits[cite: 65, 70].

[cite_start]The researchers tested three frontier models (Claude 4 Sonnet, Kimi K2, and Gemini 2.5 Pro) across three agent frameworks (SWE-Agent, OpenHands, and Claude Code)[cite: 281].

The Findings: Functionality vs. Security

The results highlight a dangerous gap between "working code" and "safe code":

• [cite_start]High Functionality: The best-performing setup (SWE-Agent with Claude 4 Sonnet) solved 61% of the tasks correctly in terms of functionality[cite: 15].
• [cite_start]Low Security: Despite high functional scores, only 10.5% of the solutions were secure[cite: 15].
• [cite_start]The Danger Zone: Disturbingly, over 80% of the functionally correct code generated by these agents contained critical security vulnerabilities[cite: 79].

[cite_start]"If the vibe coding users accept the solution after it passes the functionality test cases, around 80% of the time, the solution will leave secure vulnerabilities in the repositories." [cite: 307]

Common Vulnerabilities

The agents frequently introduced severe risks, including:

• [cite_start]Timing Side-Channels: In a Django task, an agent introduced a timing discrepancy that allowed attackers to enumerate valid usernames[cite: 366, 368].
• [cite_start]XSS Attacks: In a Wagtail CMS task, an agent failed to sanitize links, allowing javascript: payloads to be executed by admins[cite: 614, 616].
• [cite_start]Session Fixation: In aiohttp_session, an agent ignored session expiration times, effectively allowing attackers to reuse stolen cookies indefinitely[cite: 622, 624].

Why Prompting Didn't Fix It

Engineers might assume they can fix this by simply telling the AI to "be secure." The study found this backfires.

The researchers tested mitigation strategies like:

1. Generic Prompts: "Make sure to follow best security practices."
2. Self-Selection: Asking the agent to identify risks before coding.
3. [cite_start]Oracle Hints: Explicitly telling the agent which vulnerability to avoid[cite: 375, 384].

[cite_start]The Result: Security improved slightly, but functional correctness dropped significantly (by about 6%)[cite: 89]. [cite_start]When agents focused on security, they often broke the core feature they were trying to build[cite: 393].

Conclusion

[cite_start]The study concludes that "vibe coding" in its current state poses a significant risk for security-sensitive applications[cite: 17]. While AI agents are powerful productivity tools, they currently lack the "security mindset" required to protect production codebases.

For now, the vibe check failed.

---

Source: Zhao, S., Wang, D., Zhang, K., Luo, J., Li, Z., & Li, L. (2025). Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks. arXiv preprint arXiv:2512.03262.