R05 Pilot Postmortem
Date: August 28, 2026 · Scope: the full R05 pipeline run, August 25–28, 2026 Purpose: revise PROTOCOL from observed failures, not hypothetical ones. Changes adopted here become PROTOCOL v1.2.
1. What the pilot proved
The full pipeline executed end-to-end in one cycle: brief (v1.0 → v1.1 via editorial review) → archive audit (5 legacy docs, 14 claims queued for re-test) → two independent deep-research runs → adversarial cross-examination → live citation verification → 33-claim graded ledger → canonical synthesis → plain-language derivatives → content extraction. Danny signed off ledger and synthesis at v1.0.
The architecture caught real errors that no single component could have caught about itself. That was the design bet, and it paid off on the first run.
2. What worked (keep, unchanged)
- Dual independent researchers. The decisive win: both models cited the same 2023 Molecules study with different author lists — a contradiction only visible because two independent passes existed. Consensus between AIs correctly triggered investigation, not automatic trust.
- Mandatory live citation verification. Changed the evidentiary picture materially: ChatGPT 8/8 confirmed; Gemini 3 refuted/unlocatable + 5 unsourced. Without this stage, a confabulated author list, a nonexistent paper (“Mettu et al. 2023”), and invented quantitative ranges would have entered the canonical corpus looking perfectly scholarly.
- The candidate-claims convergence test (added in brief v1.1). ~12–14 of the two claim sets overlapped conceptually — a useful signal of which findings are load-bearing, and the divergence pattern itself was diagnostic.
- Archive audit before external research. Produced the then-vs-now spine of the synthesis, the strongest build-in-public material, and caught a factual error in the legacy docs (“purple chlorophyll”) before any external source was consulted.
- Three-axis grading with
unknownas a first-class status. Five claims are honest data gaps; they turned out to be among the most distinctive content (“any confident X% is made up”). - Editorial review of the brief itself (ChatGPT’s five v1.1 additions). The methodology-hierarchy, functional-unit, and lifecycle-boundary requirements visibly disciplined both research outputs.
- Plain-language derivatives flowed almost effortlessly from the ledger — evidence the claim-ID architecture does what it was designed to do.
3. Failures observed (each drives a v1.2 change)
| # | Failure | Detail | Severity |
|---|---|---|---|
| F1 | Confabulated citations that look scholarly | Gemini: spliced author list on a real paper; a nonexistent J. Cleaner Production article; a misattributed lighting review | High — the core threat this system exists to stop |
| F2 | Corrections can also be confabulated | Gemini’s replacement lighting citation (“Kusuma/Westmoreland/Bugbee 2020, PLOS ONE e0230755”) also failed verification | High — trust in the correction round was the near-miss |
| F3 | Unsourced precision | Four production-share percentages, subtype carbon ranges, water figures, efficacy figures — all specific, all untraceable; later retracted as “synthetic extrapolations” | High |
| F4 | Overcorrection in myth-busting | Confident “True/False” verdicts in the opposite direction of myths (“light-dep sweet spot: True”; “outdoor passes testing at high rates”) where evidence supports only context-dependent/unknown | Medium — violates the non-tribal voice from the anti-myth side |
| F5 | Brief-version asymmetry (near-miss) | Gemini launched on v1.0 before v1.1 was finalized; resolved by luck (it re-ran on v1.1) | Low — cost nothing this time |
| F6 | Tooling: binary uploads through chat corrupt | Hand-carried base64 docx to Drive arrived broken; caught by read-back verification | Low — plumbing, not research; fixed via Google Doc conversion |
| F7 | Mirror drift | Local canonical files ran ahead of Drive snapshots several times mid-cycle | Low — acceptable per git-is-truth rule, but needs a stated refresh cadence |
4. Changes adopted → PROTOCOL v1.2
- Resolvable-identifier rule (from F1, F3): every citation supporting a ledger-eligible claim must carry a DOI, stable URL, or independently retrievable identifier. Unresolvable citation ⇒ the claim is treated as unsourced until independently verified. (Added to §2 step 3 researcher requirements and standing brief language.)
- Two-check verification (from F1; ChatGPT’s proposal): verification records two separate results per load-bearing source — Identity verified (paper exists as cited: authors/title/venue/DOI) and Claim-support verified (paper actually says what the researcher said it says: Yes/Partial/No). A real paper cited for something it doesn’t contain is the sneakier failure, and DOI-checking alone won’t catch it.
- Verify-the-corrections rule (from F2): researcher corrections, retractions, and replacement citations receive the same verification as original claims. A correction round is not a trusted channel.
- Model-agnostic principle (from F1, explicitly guarded): R05’s scorecard (ChatGPT 8/8; Gemini 0-for-2 rounds) does NOT establish a reliable/unreliable hierarchy. Roles stay symmetric; the next report could invert the result. The protocol trusts the process, not a model.
- Calibration instruction for myth-checking (from F4): standing brief language — “When fact-checking public claims, ‘the myth is unsupported’ and ‘the opposite is true’ are different findings. Do not overcorrect; unknown is an acceptable verdict.”
- Version lock before launch (from F5): the brief is frozen and version-stamped before either researcher launches; both runs must cite the same brief version, and the critique must record it.
- Search-stopping rationale required (from ChatGPT’s practice): each research pass ends with a short statement of why searching stopped (diminishing returns per evidence lane). Makes coverage auditable.
- Plain-language derivatives standardized (from what worked): each report produces, after sign-off, a plain-English ledger companion and a layperson synthesis, with citation links drawn from SOURCE_LIBRARY’s URL table. These double as the first content products.
- Source-library URL table (from the citation-links request): SOURCE_LIBRARY.md maintains an S-ID → URL table as the single place links live; downstream documents pull from it.
- Mirror mechanics (from F6, F7): binary files are never pushed to Drive via inline base64 — readable versions go up as converted Google Docs, true .docx via direct file delivery + git
exports/. Drive mirror refreshes at stage completions; mid-stage drift is acceptable (git is truth).
5. Metrics (R05 baseline for future comparison)
- Reports: 2 researchers × 1 run each · Ledger: 33 claims (12 established / 12 probable / 2 traditional-practice / 5 unknown, plus 2 dual-status) · Sources: 25 S-IDs registered, 4 refuted/do-not-cite logged
- Verification: 13 sources web-checked; 8/8 ChatGPT load-bearing confirmed; 3 Gemini citations refuted/unlocatable; 1 second-round confabulation caught
- Convergence: ~12–14 of 20–30 candidate claims overlapped conceptually
- Archive: 5 legacy docs audited; 14 claims re-tested; 1 confirmed near-verbatim (four-garden taxonomy), 2 retired/revised (“medical quality,” “5×”), 1 legacy factual error caught pre-research
- Wall-clock: brief → signed v1.0 + content extraction in ~3 days of sessions
6. Open items carried forward
- Dillis et al. full-text check on measured per-plant water rates (upgrades C014 context if confirmed)
- Obtain the actual RII PowerScore benchmark report (would source the g/kWh efficiency figures)
- Watch for subtype-specific greenhouse LCAs (fills C009) and any blinded matched-genetics quality study (C020–C022)
- Gemini’s remaining unsourced aggregates (LeafLink premium quantification) — source or drop permanently
- Drive copies of ledger/synthesis carry v0.1 headers until next mirror refresh (git is canonical)
7. Verdict
The pilot did its job twice over: it produced a genuinely strong R05 corpus, and it stress-tested the machinery hard enough to surface a failure taxonomy worth encoding. PROTOCOL v1.2 is built from seven observed failures and seven observed successes — zero hypotheticals. R01 launches on the improved machinery.