Cases

Where osop holds, and where it does not

Each case is one thing you might want osop to do. For each one: how to run it, what happened, and the limits.

Hand audit, 10–14 November 2026

We checked osop's claim verdicts by hand against the author's own Claude Code sessions from those five days.

  • 173 claims were checked by hand.
  • None of the agent's claims turned out to be false.
  • osop marked 5 claims as contradicted (❌) and 17 as no evidence (❓). All 22 were wrong.
  • 35 rows were wrong in total (20%). Re-run on 15 November after three fixes, about 20 (12%) were still wrong. Later fixes have not been measured on these rows.
  • Most “cannot tell” (⚠️) rows come from a semicolon or a pipe that hides a command's exit status.

osop has not yet caught a real false claim in this data.

Read the one-line brief after every turn

Status: Partly
How to run it
Install the Stop hook (hooks/osop-stop.py). After each turn it prints one line and appends it with the claim rows to ~/.osop/brief.log. The same line by hand: `osop log <session-id> --brief`
Result
5 days, 173 claim rows audited by hand. Wrong at the time: 35 (20%). Wrong after 3 fixes, re-run on 15 November: about 20 (12%); later fixes not measured. Pass bar: at most 3. False claims caught: 0.
Limits
In this trial the agents made no false claims, so the line produced only false alarms (22 contradicted or no-evidence rows) and 'cannot tell' rows. The pass bar was not met.

Look back at a failure in an old session

Status: Partly
How to run it
`osop log <session-id> --stdout`, then read the Tests and Errors sections.
Result
52 sessions from the last 30 days: 42 parsed, 10 over 60 MB skipped, 14 with at least one failed test run. 3 checked by hand against their transcripts, 5 failures. The failure is visible in the Tests section: 5 of 5. The next run of the same tests is visible: 5 of 5. What changed in between: visible 1, only in a collapsed list 2, not visible 1, not applicable 1. Time: 0.36-0.58 s for 4.6-11.6 MB; 6.6 s for 168 MB.
Limits
The Tests table has no timestamps. Files written by a python or node script are not listed. A deliberate failing run looks like a real failure. The Errors section lists only calls that exited non-zero (3 of 5 failures) and shows the start of the output, not the failure line. A 168 MB session gives a 645 KB receipt with 181 test rows.

Use the receipt as evidence in pull request review

Status: Partly
How to run it
Run `osop log <session-id> --stdout` on the session that started the verifier sub-agents, and read each verifier's rows in the Tests section (the By column).
Result
10 verifier sub-agents for 3 pull requests, 57 test rows attributed to them. 9 reported 'pytest exit 0, N passed'; 1 stopped at a usage limit with no report. For 9 of 9, a row by that verifier printed N passed. Its verdict: passed 6, probably passed 1, cannot tell 2. N appears in the receipt text for 6 of 9. Receipt for a 28.8 MB transcript with 22 sub-agents: 1.28 s.
Limits
The verifiers' own reports, their PASS or FAIL and the commit they verified are not shown, and their claims are never checked: only the main session's last message is. A verifier that stopped at a usage limit looks like one that finished. A second-model review run as a shell command shows only the command and its exit code.

Audit what each sub-agent did

Status: Holds
How to run it
`osop log <session-id> --stdout`, then read the Sub-agents table and the By column of Tests, Errors and Commands.
Result
3 sub-agents checked by hand (52, 32 and 22 tool calls; one started by another sub-agent): call counts, failure counts, test rows and error lines matched their own transcripts exactly.
Limits
The table is flat: it does not show which sub-agent started which. A sub-agent that hit a usage limit before doing anything shows '0 calls, 0 failed' with no reason. A command refused by the permission system showed as a failed test (1 row); since 2026-11-17 a refused command is listed under Errors and is not counted as a test run. Sub-agents with the same description would be merged into one row (read from the code; did not occur here).

Nightly self-improvement from receipts

Status: Partly
How to run it
A separate nightly script, not part of osop, reads the day's prompts, final messages and brief.log claim rows (the rows `osop log <session-id> --brief` also prints), asks a model for proposals, and checks every accepted one in code: quotes must match word for word, from at least 2 sessions, touching only allowed files. Run in dry-run mode.
Result
5 nights: 15 proposals, 6 accepted, 6 of 6 passed the program check, 0 blocked. 6 of 15 quoted brief.log claim rows; 12 of 15 relied on receipt verdicts. 4 accepted proposals became 3 merged fixes; a later hand audit found they removed 15 wrong rows. 2 accepted proposals were not applied as of 15 November. Cost: $13.40 for 5 nights.
Limits
All 5 nights ran in dry-run: the loop opened no pull request itself, and the fixes were made by hand. The only live run used a hand-written practice proposal; its verifier failed it and the pull request was closed, as designed. The largest cause of 'cannot tell' (a `;` after the test hiding its exit status) was proposed 5 nights running and never done. The largest group of wrong rows (sentences that are not claims) was found by a hand audit, not by the loop.

The pipefail hook on its own

Status: Holds
How to run it
Install hooks/osop-pipefail.py as a PreToolUse hook. It turns `pytest 2>&1 | tail -20` into `set -o pipefail; pytest 2>&1 | tail -20`. `osop log <session-id> --stdout` then shows a failing run as failed.
Result
Transcripts from the last 30 days: 925 shell calls ran under `set -o pipefail;` (732 rewritten by the hook since it was installed on 7 November, 193 written by the agent). 675 piped a test runner. 103 exited non-zero where the last pipe segment would have exited 0, so without pipefail their exit status would have read as success. All 103 were real failures (93 with a failure summary, 10 fatal or usage errors). Passing runs turned into failures: 0 found. Without pipefail the receipt would have said 'cannot tell' for 101 and 'probably passed' for 2; with it, failed for all 103.
Limits
In 93 of 103 the output already showed the failure count, so the agent could read it anyway; the hook adds the exit status. 1 rewrite was then refused by the permission system, and the receipt showed it as a failed test (changed on 2026-11-17: a refused command is no longer a test run). Hook rewrites were told apart from agent-written ones by the record the hook leaves in the transcript; 54 calls the current hook would rewrite have no record, most because that runner form was only recognised later (not all checked).

Headless and CI sessions

Status: Partly
How to run it
Run `osop log <session-id> --stdout` on a session started with `claude -p`.
Result
3 headless sessions: a receipt came out for 3 of 3 in 0.22-0.32 s, with goal, model, times and errors as in interactive sessions. The Stop hook also ran in headless sessions: 1 brief.log entry each for 7 of 7 checked.
Limits
Nobody can approve a command in a headless session, so commands that need approval are refused. At the time osop showed a refused test command as failed: 2 of 3 test rows were refusals that never ran (changed on 2026-11-17: a refused command is no longer a test run). A final message written as JSON is read as prose: a FAIL verdict produced 'claims: 1 backed'. Not tested on a real CI runner.

Other agents (Codex CLI, Cursor and others)

Status: Not tested
How to run it
Not supported. osop reads only Claude Code transcripts.
Result
Not supported, not tested.
Limits
Other agents write their sessions in other formats; no parser for them exists.