Testing Strix, part 2: black-box vs white-box scans

I ran Strix twice against a project I’ve been building, taking a few hours total to complete the scans and work through the fixes.

The first run was black-box against a live running instance, mimicking an external attacker. The second was white-box against the source code repository, giving the agent full access to read every file.

The two modes yielded completely different finding profiles, which ended up being the most insightful part of the experiment.

The live-target run was significantly stronger. It uncovered legitimate broken access control issues, both at the object level and function level, where the API surface had quietly drifted out of sync with the auth layer in front of it. That kind of auth drift happens all the time in fast-moving projects, and because it’s tedious to check manually, it often slips through.

It also caught something a static analysis tool would almost certainly miss: a state-changing endpoint lacking concurrency control. The agent fired off multiple simultaneous requests and confirmed that every single request was accepted and spawned its own job. It wasn’t guessing based on code syntax; it proved the behavior by executing it.

It also spotted a subtle input normalization issue. An ID parameter with surrounding whitespace was accepted but passed directly into an exact-match query, causing valid lookups to silently fall through to a degraded default mode.

A quick note on setup before covering the white-box run: I ran these tests using an open-source model rather than one of the top-tier proprietary frontier models. A good portion of the hallucinations and incorrect logic I saw later was likely tied to model capability rather than the framework itself.

With that context, the white-box scan over the source code was noticeably weaker. Given full file access, the agent defaulted back to standard pattern matching: flagging secrets handling, dynamic query construction, potential SSRF points, and spreadsheet formula injection on uploads. While those are valid risk categories, manual inspection showed that most were already mitigated or gated behind strict prerequisites.

The biggest takeaway across both runs? The live target agent was surprisingly aggressive in executing attacks, but its confidence level stayed exactly the same whether its finding was completely right or totally wrong.