// red-team qa / falsification / reproducible findings / defensible verdicts
Red-team QA for autonomous engineering. Most QA asks whether the software works. HowlProof tries to prove that it doesn't, and only reports a pass when the evidence can carry one.
PROVEN
Criteria satisfied, nothing unresolved above the threshold, and enough checks actually ran.
CONDITIONALLY_PROVEN
Passed, with every exclusion and limitation named in the result.
REQUIRES_HUMAN
The evidence cannot safely support an autonomous decision.
REJECT
Evidence demonstrates a violation. The artifact does not advance.
INSUFFICIENT_EVIDENCE
The evaluation that ran could not decide, so nothing is claimed.
HowlBoard states the problem in its own domain model: a claimed result is one an agent asserted and nothing independently confirmed. The ecosystem could turn intent into code, route it, govern it and render it. Nothing in it was a dedicated adversary whose job is to try to prove the result wrong.
HowlProof is that component. It challenges an artifact, tries to falsify it, captures reproducible evidence, and issues a verdict that says exactly what was and was not established.
A check that could not run is not a check that succeeded. HowlProof distinguishes six outcomes, and four of them are not passes. Every one that is not a pass records a reason, and every check records what it does not establish.
VERIFIED — it ran and the artifact behavedFAILED — it ran and the artifact did notSKIPPED — deliberately not run, with a reasonUNAVAILABLE — the tooling was absent, with a reasonNOT_APPLICABLE — no such surface, with a reasonERROR — the evaluator failed, which is evidence about the evaluator| Traditional QA | HowlProof |
|---|---|
| Does this pass? | How can this fail, and can I prove the failure? |
| A green run means done | A green run means the checks that ran, ran |
| A skipped check is invisible | A skipped check is a recorded limitation that weakens the verdict |
| Fixes what it finds | Reports what it finds, and re-verifies the repair somebody else made |
Each adversary owns a family of checks. An evaluation selects them by profile, and any that finds no matching surface reports NOT_APPLICABLE rather than passing vacuously.
What input, state or sequence makes this behave incorrectly?
Where does the trust boundary not hold, and what can reach across it?
What happens when a dependency, a restart or the stored state goes wrong?
SIMULATED because the fault is injectedCan untrusted text become instruction, or malformed output become a trusted value?
NOT_APPLICABLE unless the artifact declares an AI surfaceWhat would a careful person get wrong, and what does the documentation promise that the code does not do?
Security evaluation is defensive and bounded. Requests are refused unless the destination is this machine, external network access is opt-in, and no exploitation tooling ships with the package. Every check, and what each one does not establish
verify-fix refuses when the artifact is unchanged, and re-runs the evaluator when it is not.
A finding moves FOUND → REPRODUCED → REMEDIATION_REQUESTED → RETESTED → VERIFIED_FIXED, and only a re-execution moves it to the last state. verify-fix compares the artifact's tree digest against the state the finding was raised at and refuses to answer when they match, because an unchanged artifact cannot have been repaired. When the evaluator that raised the finding cannot run against the changed artifact, the finding's absence is reported as inconclusive rather than as a fix.
CONFIRMED confidence is reserved for defects that were directly observed, and the model refuses to construct one without a recorded reproduction. A finding on its own can only reject an artifact when it is both at or above the blocking severity and CONFIRMED; a severe finding that was not reproduced routes to REQUIRES_HUMAN instead. A violated acceptance criterion rejects on its own terms, whether or not any finding was raised.
| Verdict | Exit | What it means |
|---|---|---|
| PROVEN | 0 | Every acceptance criterion was satisfied, no unresolved finding sits at or above the blocking severity, and enough checks produced a conclusive result to justify saying so. |
| CONDITIONALLY_PROVEN | 10 | Passed subject to documented exclusions, environment limitations or unresolved non-blocking findings. Each one is named in the result rather than folded into the pass. |
| REQUIRES_HUMAN | 20 | A severe finding was not reproduced, an evaluator asked for judgment, or an exclusion has expired. The evidence cannot safely support an autonomous decision. |
| REJECT | 30 | A blocking criterion was violated, or a reproduced finding sits at or above the blocking severity. |
| INSUFFICIENT_EVIDENCE | 40 | A criterion rested on a check that never ran, coverage fell below the declared floor, the artifact declares no criteria at all, or the artifact changed during its own evaluation. |
INSUFFICIENT_EVIDENCE is the point of the whole design. An evaluation where required tooling was missing has not shown the artifact is bad. It has also not shown it is good, and saying so is the only honest option available.
Exit codes are distinct per verdict so a pipeline can branch without parsing output, alongside 2 for a usage error, 3 for an internal fault and 4 when the artifact changed during its own evaluation. --fail-under collapses them for a pipeline that only wants pass or fail.
This is the committed dogfood evaluation of HowlBoard, run on 11 September 2026 against commit 56246a9b. The bundle it produced is in the repository and can be re-verified with howlproof inspect. Nothing on this page is an invented figure.
HOWLPROOF RESULT
Artifact: howlboard
Commit: 56246a9b47b0fe0b829756069fa70764a42c7213
Run: hp-20260911-133826-ddbaef4e94e2
Verdict: REJECT [blocking_evidence]
Acceptance criteria: 6 / 7 satisfied
Checks: 12 verified, 4 failed, 0 skipped, 1 unavailable, 3 n/a, 0 errored
security 7 finding(s)
operator 2 finding(s)
Blocking findings:
HP-SEC-0007 HIGH Stored data executes as script in the interface: mission_id_js_string
Why:
- AC-01 violated: 2 security findings at or above HIGH, limit 0
- HP-SEC-0007 (HIGH, reproduction confirmed): Stored data executes as script in the interface: mission_id_js_string
Evidence: dogfood/howlboard/hp-20260911-133826-ddbaef4e94e2
HowlBoard's architecture document claims that authority is computed on every read and that a mission cannot enter EXECUTING without live delegated authority. Both claims were probed directly and both held: a transition attempted with no approval, and one attempted with an approval minted already expired, were each refused with 403 AUTHORITY_DENIED.
The same document states that escaping "is applied to every interpolated value". It is. The finding is that the escaper is adequate for HTML text and inadequate for the context one of those values lands in.
HowlBoard's escape function handles &, <, > and " but not the single quote, while one of its call sites places the escaped value inside a single-quoted JavaScript string in an event-handler attribute. Escaping the double quote prevents breaking out of the attribute; it does not prevent breaking out of the string.
Static analysis raised that at HIGH confidence. A real browser then loaded HowlBoard's own compiled interface against a locally started instance, with an authored identifier placed in the fixture its seed endpoint reads, and the value executed. That is what moved the finding to CONFIRMED, and a confirmed finding at HIGH is what produced the rejection.
The result document keeps REAL, SIMULATED, SKIPPED, UNAVAILABLE, NOT_APPLICABLE and ERROR apart rather than reporting a count of "passed". Of this run's twenty checks, fifteen observed the real artifact, one was simulated (persistent state was corrupted deliberately, and the artifact refused to start with a diagnostic, which is the correct behaviour), one was unavailable because HowlBoard is not a Python package, and three did not apply.
proof/hp-20260911-133826-ddbaef4e94e2/
manifest.json authority ADVISORY, implementation hash, file hashes over every file
target.json repo, commit, branch, dirty, tree digest before and after
environment.json every tool probed, and why an absent one is absent
checks.jsonl each check with its checker version, status, mode and limitation
findings/HP-SEC-0007.json
reproductions/HP-SEC-0007/steps.json
evidence/web.dom_injection/mission_id_js_string.json
evidence/web.responsive/index-375.png
result.json verdict, deciding rule, criteria table
result.md the same, for a human
handoff.json advisory, for HowlPlane and HowlRelayInspect the committed bundle · Read the full dogfood record, including what was not established
HowlProof does not fix the artifact it evaluates.
A component that can both condemn and repair has an incentive to condemn what it can fix and overlook what it cannot, and its verdicts stop being independent evidence. The separation is the product, so it is enforced in code rather than promised in documentation.
4.The test suite registers an evaluator that deliberately writes into its target and asserts the evaluation is invalidated rather than reported as a pass. If the guard regresses, that test fails.
Integrations are files, written in schemas the ecosystem already reads, so no sibling component changes to consume them. Every handoff carries authority: ADVISORY: HowlProof issues verdicts, HowlPlane decides what to do about them, and HowlChangeOps owns promotion and rollback.
| Handoff | Emits | Read by |
|---|---|---|
--for plane | ai.review_finding/v1 | HowlPlane's reconcile --findings-file |
--for board | ai.evidence_entry/v1 | HowlBoard, through HowlPlane's evidence ledger |
--for relay | Markdown under recognised headings | HowlRelay's continuity collector |
--for verification-plan | ai.verification_plan/v1 | HowlPlane's verification artifacts |
--for changeops | A human-readable advisory | Nothing. HowlChangeOps has no ingest path for an external verdict, and the file says so |
They are adjacent and easily confused. HowlProof asks whether an artifact can be demonstrated untrustworthy and produces evidence. HowlChangeOps asks whether a change is authorised to happen and can be undone, and owns the release boundary.
A ChangeOps approval over an unverified artifact is a governed mistake. A HowlProof verdict with no ChangeOps gate is an ungoverned success. Both are needed and neither substitutes for the other.
A future HowlGuard would ask whether the ecosystem's security posture has changed in a way that requires new evaluation. That is continuous and ecosystem-scoped; HowlProof is episodic and artifact-scoped.
They are deliberately not the same component: an evaluator that maintains a standing opinion about its subject stops evaluating it freshly. Nothing of HowlGuard is implemented here.
Full boundary documentation · Explore the Howl ecosystem · HowlProof is not included in the default Howl installer manifest.
git clone https://github.com/howlcipher/howlproof.git
cd howlproof
python3 -m venv .venv && source .venv/bin/activate
pip install -e .
howlproof doctor
howlproof evaluate tests/fixtures/vulnerable_app --evidence-root /tmp/proof
howlproof findings /tmp/proof
howlproof report /tmp/proof
tests/fixtures/vulnerable_app is a deliberately defective artifact shipped with the package. The evaluation exits 30 and raises twelve findings: a committed credential, four workflow weaknesses, an escaper that is wrong for the context it is used in, an HTML sink with no escaping at all, a documented sync that has drifted, broken links, missing page metadata, rendering defects at a declared width, and its own unqualified documentation claims. Its neighbour tests/fixtures/clean_app is the same shapes built correctly, raises none of those, and is not rejected.
Milestone one. Experimental, not production-ready. The verdict model is argued for rather than measured: there is no labelled corpus yet, so no claim about detection rate appears anywhere in this project. Roadmap · The contract · Limitations