R073 · AI-built software and its verification › Security scanning for AI-written code

Twelve open-source repos for securing AI-written code

Read the original

appsecai-built-softwarecode-reviewagent-skillsmodel-supply-chain

The claim

Coding agents now produce code faster than humans can review it, so security checks have to be automated and layered. Beliaev lists 12 open-source repos (vendors include OpenAI, Anthropic, NVIDIA, Microsoft, Cloudflare and Cisco) that together cover code, dependencies, secrets, running services, and the agents and skills themselves.

The list is a listicle, not an evaluation: no head-to-head results, no false-positive rates. Its value is the taxonomy, and the fact that half the entries are now LLM agents rather than rule engines.

The twelve, by layer

LayerToolKindWhat it does
Source, at PR timeSemgrepRules (SAST)Pattern rules for insecure code in diffs
Source, at PR timeClaude Code Security Review (Anthropic)LLM GitHub ActionDiff-aware review of each PR, comments findings, false-positive filter
Source, whole repoSecurity Audit Skill (Cloudflare)Agent skill, multi-agentSix-phase audit with isolated hunters and adversarial verifiers
Source, whole repoCodex Security (OpenAI)Agent CLI, SDK, ActionScan, validate, patch, verify fixes; SARIF export, tracker sync
Running appStrixAutonomous pentest agentsRuns the app in a Docker sandbox, proves each bug with a working exploit
Running appNuclei (ProjectDiscovery)Template scannerThousands of known-vulnerability templates against live hosts
Dependencies and infraTrivy (Aqua)ScannerCVEs and misconfigurations in images, Kubernetes, Terraform
SecretsTruffleHogScannerSecrets in git history, with live verification of which still work
Files and uploadsClamAV (Cisco)AntivirusMalware in servers and user uploads
Agent skillsSkillSpector (NVIDIA)Static plus optional LLMScans skills for prompt injection, exfiltration, excessive agency before install
LLM appsPyRIT (Microsoft)Red-team frameworkAutomated jailbreak and prompt-injection attacks on your LLM app
Release artifactsBLA, Blind LLM AgentsObfuscation, MCP and CLIMakes bundles and binaries costly for AI reverse-engineering agents

What the agent-native tools share

The four LLM-based scanners share a converging design.

Agent and skill supply chain

Two entries treat the agent itself as attack surface. SkillSpector scans skills before installation: 71 patterns across 17 categories (prompt injection, data exfiltration, privilege escalation, memory poisoning, tool misuse and more), a fast static stage, an optional LLM stage, a 0 to 100 risk score, and baselines so re-scans show only new findings. Its README cites a research dataset in which 26.1% of 31,132 skills had vulnerabilities and 5.2% looked malicious. It is part of NVIDIA's Verified Skills pipeline, which scans, evaluates and signs skills before publication. PyRIT attacks the deployed LLM app from outside.

Anthropic's own action carries a warning worth repeating: it is not hardened against prompt injection and should only review trusted PRs. Any LLM reviewer that reads attacker-controlled diffs with write tokens is itself a target.

Gaps and caveats

Review checklist

  1. Is there a deterministic floor on every PR: SAST rules, dependency and IaC scanning, secret scanning (history plus pre-commit)?

  2. Does an LLM reviewer run only on trusted PRs, with read-only tokens, given that it can be prompt-injected?

  3. Is every LLM finding checked by a separate agent or by an exploit before it is filed?

  4. Is coverage recorded, so a clean report can be told apart from an unexamined area?

  5. Are repeat or deep scans scheduled for high-value code, since one run finds about half?

  6. Are agent skills, MCP servers and plugins scanned before install and pinned after?

  7. Is there a written authorization model that a scanner or reviewer can check BOLA-type bugs against?

  8. Is target code executed only in a sandbox with no external network?

Sources

Connections

See R073 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.