Read the one-line brief after every turn
Status: Partly- How to run it
Install the Stop hook (hooks/osop-stop.py). After each turn it prints one line and appends it with the claim rows to ~/.osop/brief.log. The same line by hand: `osop log <session-id> --brief`
- Result
- 5 days, 173 claim rows audited by hand. Wrong at the time: 35 (20%). Wrong after 3 fixes, re-run on 15 November: about 20 (12%); later fixes not measured. Pass bar: at most 3. False claims caught: 0.
- Limits
- In this trial the agents made no false claims, so the line produced only false alarms (22 contradicted or no-evidence rows) and 'cannot tell' rows. The pass bar was not met.
Look back at a failure in an old session
Status: Partly- How to run it
`osop log <session-id> --stdout`, then read the Tests and Errors sections.
- Result
- 52 sessions from the last 30 days: 42 parsed, 10 over 60 MB skipped, 14 with at least one failed test run. 3 checked by hand against their transcripts, 5 failures. The failure is visible in the Tests section: 5 of 5. The next run of the same tests is visible: 5 of 5. What changed in between: visible 1, only in a collapsed list 2, not visible 1, not applicable 1. Time: 0.36-0.58 s for 4.6-11.6 MB; 6.6 s for 168 MB.
- Limits
- The Tests table has no timestamps. Files written by a python or node script are not listed. A deliberate failing run looks like a real failure. The Errors section lists only calls that exited non-zero (3 of 5 failures) and shows the start of the output, not the failure line. A 168 MB session gives a 645 KB receipt with 181 test rows.
Use the receipt as evidence in pull request review
Status: Partly- How to run it
Run `osop log <session-id> --stdout` on the session that started the verifier sub-agents, and read each verifier's rows in the Tests section (the By column).
- Result
- 10 verifier sub-agents for 3 pull requests, 57 test rows attributed to them. 9 reported 'pytest exit 0, N passed'; 1 stopped at a usage limit with no report. For 9 of 9, a row by that verifier printed N passed. Its verdict: passed 6, probably passed 1, cannot tell 2. N appears in the receipt text for 6 of 9. Receipt for a 28.8 MB transcript with 22 sub-agents: 1.28 s.
- Limits
- The verifiers' own reports, their PASS or FAIL and the commit they verified are not shown, and their claims are never checked: only the main session's last message is. A verifier that stopped at a usage limit looks like one that finished. A second-model review run as a shell command shows only the command and its exit code.
Audit what each sub-agent did
Status: Holds- How to run it
`osop log <session-id> --stdout`, then read the Sub-agents table and the By column of Tests, Errors and Commands.
- Result
- 3 sub-agents checked by hand (52, 32 and 22 tool calls; one started by another sub-agent): call counts, failure counts, test rows and error lines matched their own transcripts exactly.
- Limits
- The table is flat: it does not show which sub-agent started which. A sub-agent that hit a usage limit before doing anything shows '0 calls, 0 failed' with no reason. A command refused by the permission system showed as a failed test (1 row); since 2026-11-17 a refused command is listed under Errors and is not counted as a test run. Sub-agents with the same description would be merged into one row (read from the code; did not occur here).
Nightly self-improvement from receipts
Status: Partly- How to run it
A separate nightly script, not part of osop, reads the day's prompts, final messages and brief.log claim rows (the rows `osop log <session-id> --brief` also prints), asks a model for proposals, and checks every accepted one in code: quotes must match word for word, from at least 2 sessions, touching only allowed files. Run in dry-run mode.
- Result
- 5 nights: 15 proposals, 6 accepted, 6 of 6 passed the program check, 0 blocked. 6 of 15 quoted brief.log claim rows; 12 of 15 relied on receipt verdicts. 4 accepted proposals became 3 merged fixes; a later hand audit found they removed 15 wrong rows. 2 accepted proposals were not applied as of 15 November. Cost: $13.40 for 5 nights.
- Limits
- All 5 nights ran in dry-run: the loop opened no pull request itself, and the fixes were made by hand. The only live run used a hand-written practice proposal; its verifier failed it and the pull request was closed, as designed. The largest cause of 'cannot tell' (a `;` after the test hiding its exit status) was proposed 5 nights running and never done. The largest group of wrong rows (sentences that are not claims) was found by a hand audit, not by the loop.
The pipefail hook on its own
Status: Holds- How to run it
Install hooks/osop-pipefail.py as a PreToolUse hook. It turns `pytest 2>&1 | tail -20` into `set -o pipefail; pytest 2>&1 | tail -20`. `osop log <session-id> --stdout` then shows a failing run as failed.
- Result
- Transcripts from the last 30 days: 925 shell calls ran under `set -o pipefail;` (732 rewritten by the hook since it was installed on 7 November, 193 written by the agent). 675 piped a test runner. 103 exited non-zero where the last pipe segment would have exited 0, so without pipefail their exit status would have read as success. All 103 were real failures (93 with a failure summary, 10 fatal or usage errors). Passing runs turned into failures: 0 found. Without pipefail the receipt would have said 'cannot tell' for 101 and 'probably passed' for 2; with it, failed for all 103.
- Limits
- In 93 of 103 the output already showed the failure count, so the agent could read it anyway; the hook adds the exit status. 1 rewrite was then refused by the permission system, and the receipt showed it as a failed test (changed on 2026-11-17: a refused command is no longer a test run). Hook rewrites were told apart from agent-written ones by the record the hook leaves in the transcript; 54 calls the current hook would rewrite have no record, most because that runner form was only recognised later (not all checked).
Headless and CI sessions
Status: Partly- How to run it
Run `osop log <session-id> --stdout` on a session started with `claude -p`.
- Result
- 3 headless sessions: a receipt came out for 3 of 3 in 0.22-0.32 s, with goal, model, times and errors as in interactive sessions. The Stop hook also ran in headless sessions: 1 brief.log entry each for 7 of 7 checked.
- Limits
- Nobody can approve a command in a headless session, so commands that need approval are refused. At the time osop showed a refused test command as failed: 2 of 3 test rows were refusals that never ran (changed on 2026-11-17: a refused command is no longer a test run). A final message written as JSON is read as prose: a FAIL verdict produced 'claims: 1 backed'. Not tested on a real CI runner.
Other agents (Codex CLI, Cursor and others)
Status: Not tested- How to run it
Not supported. osop reads only Claude Code transcripts.
- Result
- Not supported, not tested.
- Limits
- Other agents write their sessions in other formats; no parser for them exists.