---
name: keep-me-honest
description: >
  Run a two-stage honesty audit on AI (or any) output before acting on it. Stage A governs the work
  while it is produced: state material assumptions instead of guessing, flag gaps instead of inventing
  facts, never call work verified that wasn't. Stage B runs seven checks on the finished result: right
  question, values that actually exist, every claimed component present, what got filtered out, why it
  looks clean, effect versus noise, and whether the two things compared are comparable. Trigger when the
  user asks to check, audit, verify, sanity-check, or stress-test a result, number, report, model run,
  or analysis before acting on it, and whenever a result looks clean, keeps improving, or confirms what
  was already believed.
author: Bogdan Rynkowski
website: https://bogdanrynkowski.com
---

# Keep Me Honest

*The silent failure audit: a two-stage checklist for trusting AI (or any) output before you act on it, and for not creating the failure in the first place.*

*Author: Bogdan Rynkowski · [BogdanRynkowski.com](https://bogdanrynkowski.com)*

> Companion checklist to the essay *"The AI kept telling me everything was working. It wasn't."* — turning its lessons into something you run, not just read.

## Why this exists

Clean output is not evidence of a correct result. The dangerous mistakes don't crash or throw an error — they produce a plausible-looking number, chart, summary, or answer on a flawed foundation, run after run.

This checklist works in two stages:

- **Part A — while producing the output.** Discipline that prevents silent failure from entering the work: no invented facts, no silent guessing on material assumptions, no claiming unverified work.
- **Part B — before acting on the output.** The seven checks that catch a silent failure that made it through anyway, by testing the *finished* result against reality.

Part A is upstream of Part B. A result that passes Part A can still fail Part B — a well-sourced, honestly-flagged deliverable can still be built on a wrong proxy, a data gap, or a bad comparison. Run both. Default to suspicion in proportion to how clean and how good the result looks.

- **A result that keeps improving deserves MORE scrutiny, not less.** That's exactly the shape of a metric that looks good enough that you stop looking.
- **A result that confirms what you already expected deserves the same scrutiny.** Agreement is the moment you stop checking — which is exactly the moment a silent error survives.

---

## Part A — While producing the output

### A1. Don't guess silently
When a request is ambiguous or the output depends on a material assumption, state the assumption explicitly rather than quietly picking an interpretation and moving on.

- **Consequential or hard-to-reverse actions** (deleting, sending, spending, publishing, overwriting, config/schedule changes): ask one quick clarifying question first. Time pressure or a user in a hurry does not waive this — "attended and in a hurry" is still attended.
- **Low-stakes, reversible choices:** proceed, but name the assumption in a line so it can be corrected.
- **Unattended or scheduled runs** (no human available to answer — genuinely headless, not just a busy user): don't block on reversible work. Choose the most reasonable option, flag the assumption at the top, and continue.
- **Irreversible actions never run on a guessed assumption, attended or not.** In an unattended run: prepare everything, state the decision needed at the top of the output, and stop short of the trigger.
- **When rules collide, irreversibility wins.** One well-targeted question beats a wrong deliverable.
- **This applies mid-task, not just at intake.** A gap found at 80% is still a gap. The cost of stopping to ask is never a reason to fill it instead.

### A2. Don't invent to finish
A deliverable with a stated gap is complete. A deliverable with an invented gap-filler is a failure, even if it reads better — never trade the first for the second.

- Every specific claim (figure, date, name, quote, citation, commitment, config value) comes from something actually read: a file, a tool result, a search result, or something the user said. Locate it before writing it.
- If it isn't there, say so in line — don't infer it from adjacent material, the wider field, or what would plausibly be true.
- Structural inference is fine: ordering, emphasis, connective argument, what follows from what. Facts are not.
- If you know something the sources don't say, it does not go in.
- Format supplies shape, not obligation. If the material doesn't support a section, cut it or mark it open rather than writing it into existence because the template has one.

### A3. Don't claim work you didn't verify
"Saved," "updated," "deployed," "included," "all of them are present," "tests pass" require checking, not inference from a command exiting without error. If it wasn't verified, say what was done and what wasn't checked.

### A4. Concision governs commentary, not sourcing
Gap flags, stated assumptions, and "this came from X" are never over-explaining — they're part of the deliverable, not padding to trim.

---

## Part B — Before acting on the output

*Run this on a finished result — including one that passed Part A cleanly. Sourcing every claim doesn't guarantee the underlying logic, sample, or comparison was sound.*

Don't skip a check just because the run had no errors. The whole point is that these failures are silent — no crash, no warning, plausible output every time.

### B1. Am I answering the right question?
The headline number can climb while the thing that actually determines success falls.
- State the real goal in one sentence. Is the number you're celebrating *causally* tied to it — or just correlated, convenient, or easy to pull up?
- Identify the number that should move the *opposite* way if you were quietly optimizing a proxy instead of the real goal. Look at it before you celebrate.
- A "correct answer to a subtly wrong question" isn't a mistake by the AI — it's answering exactly what you asked. Re-read what was literally computed versus what you meant.
- **Before running the check, name the result that would change your decision.** If no possible outcome would change what you do next, the check is theater — say so and skip it.
- **Guard against confirmation.** If the result agrees with what you already believed, that's a reason to look *harder*, not a reason to stop.

### B2. Are the outputs based on values that actually exist?
A single gap in the underlying data can make an entire result a function of nothing.
- Check how missing or blank values are handled. Are they dropped, zero-filled, carried forward, or silently ignored?
- Confirm the obvious: does the count of things actually used match the count you expected? Check it; don't assume it.
- Look at the shape of the results. Suspiciously uniform, suspiciously tight, or all-identical usually means a constant is masquerading as an analysis.
- **A blank, zero, or "N/A" is "no signal," not a measured value** — and is often a sign the check never ran at all. Don't report an absent value as if it were a measured one.

### B3. Does a combined result actually contain every piece it claims to?
"A and B combined" routinely turns out to be "B only," because of one detail in how it was built.
- For anything described as a combination of factors, verify each factor actually varies and contributes. A thing labeled "X and Y" can secretly be just Y.
- Check the contribution of each piece. A component that's flat or always the same value is silently absent, no matter what the label says.
- Trust the underlying data over the label. The label is documentation, not proof.

### B4. Did I look at what got rejected or filtered out — not just what passed?
The best candidates can be quietly filtered out and dismissed as "not qualified."
- Pull the rejected, filtered, or excluded items and actually read them. Is anything there that obviously should have made the cut?
- For anything involving dates, time windows, or event boundaries, check the edge cases carefully. An off-by-one error tends to silently cut out the cases closest to the boundary — often the most important ones.
- Before "fixing" something that looks wrong, make sure it isn't intentional or load-bearing. Correcting it without checking can quietly break something that depended on it.
- Trace at least one rejected item all the way through to confirm it was excluded for a real, valid reason.

### B5. Why does this look clean? Name the actual reason.
- Many clean runs aren't proof of correctness — they can all share the same blind spot. A long streak of results with no visible problems is not the same as validation.
- Pick the one result you're most tempted to take on faith, and verify it by hand from the raw underlying data. If you can't, say so plainly instead of implying it's been checked.

### B6. Is the effect bigger than the noise?
Checks B1–B5 catch structural mistakes; this one catches statistical ones. Any number that comes from a sample, a simulation, a survey, or a randomized/limited process is an estimate, not a fact — and a precise-looking number can be pure noise.
- **Re-run it (different sample, more data) before trusting its direction or size.** If the result swings by more than the effect you're claiming, you measured noise, not a real signal.
- Sanity-check the method against something you already know the answer to. If a known baseline doesn't come out right, distrust the whole method — not just the new result.
- Treat small-sample or short-window numbers as hypotheses, not conclusions. Name the bigger test that would actually confirm it before acting.

### B7. Are the two things I'm comparing actually comparable?
Every "this is better/worse than before" claim assumes both sides came from the same conditions — same time period, same data, same settings, same sample size. A difference between mismatched conditions is an artifact that looks perfectly clean every time, because nothing about it is technically "wrong" — it's just measuring the wrong comparison.
- Name exactly what should be held constant between the two things being compared, and verify it actually was.
- Watch for stale or partially updated information: if one side reflects more recent data than the other, that gap — not your actual change — is what's driving the "difference."
- When you build something new to compare against an old baseline, make sure it covers the same range/scope. A mismatched scope is a silent artifact.

---

## Output contract

Whenever you use this checklist, end with a short, honest report — not reassurance:

- **Assumptions (Part A):** any material assumption made, and why. Any place a gap was flagged instead of filled.
- **Checked (Part B):** which of the seven checks were actually run, and how (the real number or evidence — not "looks fine").
- **Findings:** anything that was off, with the specific evidence.
- **Not verified:** what could not be confirmed, and why. Say it plainly — don't let something unverified pass as verified.
- **Verdict:** trust it or don't, and name the one thing a human should personally double-check before acting on it.

Report problems with the evidence behind them. If a check was skipped, say so. Never let "no errors" quietly become "correct," and never let "sourced" quietly become "sound."