How We Measured That 70%+ Stat — And Why We Tightened It

Earlier this week we published that "83% of OpenClaw skills have security issues." Today we're updating that number to over 70% — and we want to be transparent about why.

What We Found in Our Own Scanner

We were doing a final review before our Hacker News launch when we noticed something odd in our scan results. A skill with two lines of code was being flagged with six findings, including:

• LOW — No Data Exfiltration Found
• LOW — No Obfuscation Detected
• LOW — No Supply Chain Risks Found

Our LLM was returning negative findings — listing things it didn't find as if they were vulnerabilities. The absence of bad behavior was being scored as suspicious behavior. That's backwards.

The Root Cause

The GPT-4o system prompt we were using asked it to "report all security-relevant observations." Helpfully, the model reported both what it found and what it confirmed was clean. Our parser treated every item in the findings array as a real finding, regardless of whether it was positive or negative.

We fixed this with two changes:

1. Hardened system prompt: Added explicit rules — "Do NOT report the absence of an issue as a finding. Do NOT flag clean code as suspicious just because a pattern could theoretically be misused."
2. Post-processing filter: A filterSpuriousFindings() function that removes any finding whose title starts with "No " or whose description confirms the code is clean.

What the Real Number Is

After re-scanning 121 skills with the corrected scanner, over 70% still have medium or higher severity findings. The headline number dropped, but the underlying signal got stronger — because we removed noise.

The most common real findings post-fix:

1. Shell command injection vectors — exec/spawn with insufficient input validation (38% of flagged skills)
2. Environment variable access + outbound requests — the classic credential exfiltration pattern (29%)
3. Hardcoded API keys or secrets — credentials committed directly in source (21%)
4. Arbitrary file system access — reading outside the skill's expected scope (17%)

Why We're Publishing This

We could have quietly updated the stat and moved on. But if we're building a security tool, we need to hold ourselves to the same standard we hold the skills we scan.

Developer trust is earned through transparency. We found a bug, we fixed it, and we're telling you about it. That's the only way this works.

The updated number is on our homepage. The fixed scanner is live. Scan your skills →