Prove It
Establish a narrow claim: the same test catches the specified defective behavior before a repair and passes after it. A passing suite alone does not establish this.
Choose the witness
Inspect the fix and the relevant test. Identify the expected input/output or state transition, and select exactly one pytest node. Use two committed local Git refs; the after ref must contain both the fix and the test. Uncommitted edits are not included. Do not commit someone else's work just to run this tool.
The defect assertion needs a specific message, such as:
assert expired(now=10, deadline=10), "PROVEIT:expiry-boundary"
The message must be on an assertion in the selected test function. Do not put it on unrelated assertions, in a fixture, in application code, or in a print statement. Inspect that this assertion actually observes the behavior described by the bug. If the existing test needs a message, add it only within the authorized edit scope and include that test in the after revision. Otherwise explain the missing input.
Run the comparison
Run the bundled script using its absolute installed path. The selected interpreter needs pytest and the project's dependencies already installed. No dependency installation, network access, Git checkout, reset, stash, or commit is performed.
python <skill-directory>/scripts/proveit.py --repo <repository> --before <old-ref> --after <fixed-ref> --test tests/test_expiry.py::test_at_deadline --expect PROVEIT:expiry-boundary --output <new-directory-outside-repository>
The runner overlays the after revision's selected test file on both snapshots,
uses a fresh copy per attempt, and defaults to three runs per revision with a
10-second timeout per run. Use --python, --repeat 2..5, or --timeout only as
the task needs. Select a full parameter ID if the test is parametrized.
Snapshots isolate files, not execution. Run only code whose execution is already authorized in this environment. Tests retain the user's filesystem, network, and credential access. Do not interpret a temporary directory as a security sandbox. Never clean up unrelated worktrees or directories. The runner removes only its own temporary snapshots and leaves the evidence directory for review.
Review the evidence
Read report.md, report.json, selected-test.py, fix.diff, and relevant logs.
Check the observed assertion, selected node, phase records, and changed paths.
| Result | Interpretation | Exit |
|---|---|---|
supported | Every old run hits the declared assertion; every new run passes. | 0 |
not_supported | The old test also passes, or the repaired revision still hits that assertion. | 1 |
inconclusive | Environment, unrelated failure, timeout, skip, incomplete or unstable execution. | 2 |
An unrelated call failure is inconclusive even when the new revision passes. Collection, dependency, syntax, setup, and teardown failures are never a witness. Do not relabel them as a caught bug. Resolve the observed cause within scope and rerun into a new evidence directory; do not repeat an unchanged failed setup.
Before accepting supported, inspect revision-specific fixtures/configuration
and confirm they do not explain the change. Only the chosen test file is overlaid.
Installed dependencies are shared, third-party pytest plugin autoload is disabled,
and project addopts is cleared. Projects requiring external plugins or builds
may need a separately prepared environment; report unsupported execution plainly.
Report the test, refs, outcome, evidence location, and material limitations. Say that this witness caught the specified behavior in these runs. Do not claim a proof of overall correctness, absence of flakiness, general test quality, or measured productivity improvement. Do not publish reports containing local paths or logs without reviewing them and obtaining the applicable publication authorization.