Testing Strix, part 3: what held up under manual review

When reviewing the Strix security report, I didn’t fix anything automatically. Every finding was manually verified in the codebase first.

Here is how the findings held up under actual review:

The access control issues were real. Rather than patching individual API routes one by one, I fixed them at the framework level so all future routes inherit the same authorization logic automatically.

The input normalization bug showed why manual triage is necessary. The report showed a working proof-of-concept for a single endpoint. Searching the codebase revealed the same unstripped input pattern in four other handlers. Fixing only the reported route would have left four active bugs behind. The tool highlights the symptom, but you still need to address the root cause.

The concurrency issue was also valid. I resolved it by adding a deduplication lock and rate limiting, applying existing locking patterns already used in the project.

Then came the findings that didn’t hold up:

• A high-severity SQL injection flag where the variable was properly bound as a parameter rather than concatenated into the query string. The test payload would have just been matched as literal text. Not exploitable.
• A critical-severity finding claiming a secrets file was exposed in version control. The agent had local filesystem access, saw the file existed, and assumed it was publicly exposed without checking .gitignore.
• Flags on legacy code paths that were never imported or executed anywhere in the active application.
• Findings that were technically correct in code structure but ignored intentional business logic, such as administrative staff roles having broad cross-resource access by design.

The underlying trend was severe over-estimation of risk. Severity scores were consistently inflated compared to real-world impact once prerequisites were checked. (Again, this was run on an open-source model, so results will vary with stronger models.)

There was also a self-inflicted lesson worth sharing. While hardening the app, I removed an unused API documentation route. However, leftover template references to that missing route ended up throwing 500 errors for logged-in users. My unit test suite passed because it didn’t render those layout templates. I only caught it by manually testing the running app. Security fixes are code changes; they can break functionality just like any other PR.

Final verdict: As an automated tool for generating lead candidates and attempting live exploits against a running application, Strix is genuinely helpful and I’ll keep using it. But as an absolute authority on real-world exploitability, it isn’t there yet. It can’t understand context or product intent that isn’t explicitly written down.

Human oversight is still essential for determining actual reachability, trust boundaries, and blast radius.