MED01-P2 — §M10a cross-audit dispositions (2026-10-13, Claude)
Raw audits: ChatGPT · Gemini. Verifier checks of both: secondary-audit-P2-verification.md.
Update 2026-10-14 — independent check of the ChatGPT-origin components COMPLETE: Danny’s source-level spot check (option b) passed 8 of 8, with one refinement applied as correction #14 (spot-check-P2-danny.md). Corrections #5–#13 were applied on 2026-10-13. Remaining, carried as held items (as at the P1 lock): the Whiting erratum text and Table 3 (JAMA bot-blocked; Danny to paste), the content of five other sources’ errata, and the convergence items in the ledger’s held list. §M10a can close at P2 LOCK with these carried.
Status (2026-10-13): §M10a NOT closed. ChatGPT’s audit is accepted, and its findings are mostly confirmed at source level. Gemini’s audit is not accepted as an independent audit (fabricated quotations and downstream passages; verification §A). P2 cannot lock until (1) the corrections below are applied as logged corrections, (2) the pending verifier reads are done or explicitly carried, and (3) the ChatGPT-origin components get a valid independent check.
Decisions for Danny
-
Replacement for the Gemini audit of the ChatGPT-origin components (14 claims plus the C065 and PC8 rulings). Options:
- (a) Rerun Gemini, with the five signed documents and the source list pasted as text and a three-batch prompt. Every quotation is diffed on receipt, and the audit is rejected again if the fabrication repeats.
- (b) Danny’s source-level spot check of at least 6 ChatGPT-origin components, as the §M10a escalation route allows. The verifier prepares one-page checklists with links.
- (c) Both.
Recommendation: (b), or (c) if time allows. Three consecutive fabricated Gemini verification filings argue against relying on (a) alone.
-
The JAMA pages for Whiting 2015 are bot-blocked for the verifier, but a person in a normal browser can open them. Please open:
- the 2015-08-25 erratum “Incorrect Values Reported” (doi 10.1001/jama.2015.9010);
- the 2016-04-12 erratum “Incorrect Effect Estimate” (doi 10.1001/jama.2016.3470);
- Table 3 of the article.
Paste their text here. That turns three convergence items into verified ones (C078 spasticity values, and the C077/C078 adverse-event, withdrawal and serious-AE odds ratios).
-
Approval to apply corrections #5–#13 below to the signed ledger, synthesis and plain-English pair as logged corrections.
ChatGPT findings
| # | Finding | Sev. | Disposition | Correction |
|---|---|---|---|---|
| B1-4 | C048 Wade 2004 primary mislabelled “composite” | minor | ACCEPT (confirmed) | #5: “primary (VAS of each patient’s most troublesome symptom) not significant; spasticity VAS improved” |
| B1-30 | C074 Pletcher “FEV1/FVC slightly higher” — wrong measure | major | ACCEPT (confirmed) | #6: “at low cumulative exposure, FEV1 and FVC were slightly higher; the association was nonlinear” |
| B1-23/40 | C065 “diabetes” → “Type II diabetes” | minor | ACCEPT (confirmed) | #7, in the ledger and both plain documents |
| B1-32 / self-check | C041 “fabricated” stronger than provable | none | KEEP the ruling. Gemini’s own correction-round reply admitted the figures were “estimated from background training approximations” after it had presented them as live index figures. Wording clarified: “presented as live counts; admitted estimated” | #8 (verification-log wording only) |
| B2-01 | C077 Wang 96.6% denominator | major | ACCEPT (confirmed) | #9: 31 studies (23 RCTs + 8 observational); 96.6% = 4,615 of 4,779 adverse events in the 23 RCTs; serious-event RR 1.04 (0.78–1.39) |
| B2-02 | C067 NCT01898520 published (Fairhurst 2020) | major | ACCEPT (confirmed). Verifier failure: the scheduled title search was never run | #10: remove it from the “no publication identified” examples; record the 2020 publication as retrospective (negative primary; 3 hallucination cases); re-screen all 23 by title, sponsor and design before stating any count, together with the 19 errored lookups. C067 count marked “under re-screen” until then |
| B2-03 | C064 Kerbrat: the IGAS regulatory record supports one death + four others | major | PARTLY ACCEPT. The ledger’s four-person wording stays, because it describes the NEJM report. The verifier’s ruling that Gemini’s counts were “not supported” is withdrawn as overbroad: they are not in the cited abstract but are consistent with the regulatory record (convergence; IGAS read pending, URL 404) | #11: C064 adds “the French regulatory inquiry (IGAS 2016) reported one death and four other participants with severe neurological disorders (convergence, verifier read pending)” |
| B2-04 | Whiting serious-AE OR 1.41 (1.04–1.92) not visible beside Wang | major | ACCEPT the harms-parity point. The figure is two-model convergence, JAMA-blocked | #12: C077 and the synthesis harms section add the attributed and dated Whiting AE / withdrawal / serious-AE ORs, labelled convergence until read |
| B3-01 | C078 Whiting Ashworth −0.36 superseded by erratum | major (most serious) | ACCEPT. The erratum’s existence is confirmed (PubMed ErratumIn ×4); the corrected values are convergence | #13: C078 replaces −0.36 with “the original abstract figure (−0.36; 7 trials) was later corrected by the journal (Aug 2015 erratum); corrected values reported as Ashworth −0.12 (−0.24 to 0.01; 5 trials, not significant) and patient-rated NRS −0.76 (−1.38 to −0.14; 3 trials) — verifier read pending”. Also note in C048 that the corrected Whiting split matches the clinician-vs-patient pattern. The verifier’s earlier ruling against Gemini’s −0.76 is re-stated: the number was real (patient-rated NRS); Gemini attached it to the wrong scale |
| B3-02 | Plain ledger C055 “214 children” — ages 1–30 | major | ACCEPT (confirmed) | in #5-batch: “214 children and young adults (aged 1–30)” |
| B3-03 | Meiri “about as well” — equivalence not shown | minor | ACCEPT | C053 and plain documents: “similar response rates (54% vs 58%); not designed to show equivalence” |
| B3-04 | Summary cross-references on the cancer-weight-loss bullet | minor | ACCEPT | citations attached to each bullet |
| B3-05 | Tramèr full text says “30 randomised controlled trials” | minor | ACCEPT (confirmed). The verifier’s critique remark against Gemini is withdrawn | C053 verification line, critique §4, verification log |
| whole-doc | “Every figure checked” needs a source-version qualification | — | ACCEPT | synthesis §13 and summary “How we did this”: “checked at the documented access level; publisher corrections were not systematically checked until 2026-10-13” |
HELD items after the audit
| Held item | Status |
|---|---|
| C071 Meier “8 IQ points” | Resolved (verifier read PMC full text): applies to the 23 persistent adolescent-onset users only. The ledger adds it with the subgroup |
| C062 UK exact day | Resolved: 16 June 2010 (emc SmPC §9) |
| C062 AEMPS | Resolved: 20 July 2010 (AEMPS ficha técnica §9) |
| C062 TGA | Convergence (26 Nov 2012); verifier read pending |
| C048 Novotna −0.84 | Convergence (secondary NCBI Bookshelf table); held |
| C048 CAMS correct-guess and capsules | Still held |
| C055 orphan designation | Convergence (2013-11-14, Dravet); verifier read pending |
| C055 Geffrey CYP2C19 | Mechanism not established by the study; held |
| C057 registry history | Convergence (Cochrane risk-of-bias note); held |
| C064 regulatory cohort | Convergence; IGAS read pending |
| C067 | Re-screen required (B2-02) |
| C069 programme legal bases | Netherlands 2003-09-01 = convergence; the rest held |
| C077 Whiting ORs | Two-model convergence; JAMA read pending (Danny) |
| C082 ACP/AMA content | Convergence on position text; amendment check pending |
| NEW — erratum content | Devinsky 2016, Bachhuber 2014, Van Gaal 2005, Pi-Sunyer 2006 and Wilson & Nicoll 2001 each have a published erratum. Check whether any ledgered figure changes |
Gemini audit
Not accepted (verification §A). Its two independently true verdicts (C065 FIRES; PC8 reasoning) are recorded as agreement only. A permanent caution row is added to SOURCE_LIBRARY: “Gemini §M10a cross-audit, MED01-P2: fabricated quotations and fabricated downstream-document passages; verdicts not usable without diff.”
Lessons (to PROTOCOL-MEDICAL at the MED01 postmortem)
- The integrity check must read ErratumIn / CommentsCorrections links, not only publication-type flags.
- A scheduled verification step (the title search) must run before any “no publication identified” example is published.
- Prefer the corrected publisher version over the PubMed abstract for any figure ledgered from a systematic review.
- Gemini cannot serve as a sole independent auditor without the diff gate. Consider a standing rule: every quotation is diffed automatically on receipt, and an audit that fails the gate is returned.