For anyone who wants to try contributing to an open source project, including beginners and heavy AI users. You do not need to know a community's process beforehand. Every term comes with an everyday picture, and the git walkthrough is marked so you can skip it if you do not need it yet.

Figures come from public sources, listed at the end. Examples labelled "illustration" are made up and name no real person or project. This post does not tell a personal contribution story, because I have none to tell. The git walkthrough is something I really ran on a practice repository on my machine. Nobody fully understands how AI models work inside, so the pictures below are memory aids only.

Quick summary#

• Maintainers of open source projects are receiving many low-quality contributions generated by AI. A Linux Foundation Research survey (20 October 2026) reports that maintainers estimate about 25% of pull requests are "AI slop", and 31% on large projects.
• One concrete example: curl ended its bug bounty at the end of January 2026. Its lead maintainer wrote that the share of security reports that turned out real fell from above 15% to below 5%.
• The way to avoid burdening them fits in four weeks: week 1 read and use the project, week 2 pick a small accepted issue and ask first, week 3 make a small tested change you can explain without AI, week 4 open a clear pull request, disclose AI and take feedback calmly.
• Every project has its own rules and they change fast. Some allow AI if you disclose it and understand it, some ban it. You must read the rules of the project you are contributing to, not rely on this post.
• In the git walkthrough I really ran (Git 2.56.0), one small change goes through three steps: test first, fix second, then amend after feedback. My first run also said "NO TESTS RAN", a good reason to always note how many tests ran.
• Rejection is normal. The goal of the four weeks is not to get merged at any cost, but to behave so that next time someone wants to read your work.

Hard words, plain meaning#

After this post you can explain#

1. Why open source maintainers dislike unchecked AI-generated contributions, and why projects' policies can differ completely.
2. The four-week order: read the rules, ask first, make a small tested change you understand line by line, then open the PR and disclose AI.
3. The git flow for this: your own branch, a commit with a test, a rebase when the project has moved on, and an amend after feedback.

1. A picture first: you are a guest in someone else's workshop#

Picture a small carpentry workshop run by one owner who lets visitors drop in. One visitor walks in and says, "I think this place could look nicer", then brings a cartload of things they made and puts them in three or four corners. Maybe the things are good. But the owner has to inspect every piece, make room, and if something breaks it is the owner who answers for it. Another visitor sits and watches for a week, asks, "May I sand that chair for you?", sands exactly one chair and brings it over to be looked at. The owner will remember the second name.

Open source is the same. Maintainers are often people working in their spare time or unpaid. Every pull request is a piece they have to check. AI makes sending a piece far cheaper, but checking it still costs the maintainer's time.

Takeaway: good contribution is not measured by how many lines you send but by how easy you make it for the maintainer to say yes.

2. Why maintainers are overwhelmed#

2.1 The Linux Foundation Research survey (20 October 2026)#

Linux Foundation Research published the report "Navigating Open Source Software Development in the Generative AI Era: Challenges and Opportunities for Project Communities" on 20 October 2026. According to their press release, maintainers estimate about 25% of incoming pull requests are AI slop, and 31% on large projects. The same release says 59% of maintainers see growing contribution volume (69% on large projects), about 35% of projects have a formal policy that permits AI, 32% have no policy and no plan for one, and 61% of contributors say they review and fully test AI-generated code before submitting it. It also reports respondents estimating an average of about 52% less time to write code.

Read this carefully. These are estimates by survey respondents, not a count of pull requests. The press release I read does not state how many people answered. I read the press release, not the full report. The sample may lean toward people willing to answer a survey.

Takeaway: the problem is real and measured, but most projects have no clear rule yet. You have to go and find the rule.

2.2 One project as an example: curl ends its bug bounty#

Daniel Stenberg, curl's lead maintainer, wrote in "The end of the curl bug-bounty" on 26 January 2026 that curl no longer has a bug bounty, effective 31 January 2026. He states that before 2025 the share of submissions confirmed as real vulnerabilities was "north of 15%", and since 2025 it is "below 5%", not even one in twenty. He names AI slop, low-quality human reports and a bad faith attitude from some reporters as reasons, and he suspects the cash rewards attracted that abuse. He also says people who submit AI slop continue to be banned.

This describes one project and the view of someone inside it. It does not prove every project is the same, and cash rewards may be specific to curl.

Takeaway: when the receiver is flooded, they close doors. When a door closes, good contributors suffer too.

2.3 GitHub gives maintainers more controls#

On 13 February 2026 GitHub added two repository settings: turn pull requests off entirely, or let only people with write access open new pull requests (everyone can still read and comment). They sit under Settings, General, Features. A Japanese developer blog I found reported that someone tested the "collaborators only" setting and saw it behave differently from the description at the time, so treat this as something that may have changed. GitHub is also reported to have added a cap on how many pull requests a user without write access can have open at once (June 2026, which I only saw in news coverage).

Takeaway: some repositories will not accept pull requests from strangers. First, find out whether the one you want does.

2.4 A real policy: Ghostty#

To see how specific a policy can get, I read the CONTRIBUTING.md and AI_POLICY.md files of the Ghostty project. I state only what I read at the time of reading, and the policy may have changed by the time you read this. According to those files: the project allows AI to write code, but requires disclosure of all AI use (naming the tool and the extent), requires contributors to understand all the code they submit and explain it without AI, and requires first-time contributors to be vouched for, otherwise pull requests are closed automatically. The project also keeps a public list of people turned away for submitting low-quality contributions. The rules apply to outside contributors, not maintainers.

This is one project. Another may be looser, or ban AI-generated contributions outright. This post relies on no project beyond the ones read above.

Takeaway: the rules live in the project itself. The first question is not "what do I do" but "what do they say here".

3. The four-week plan#

Picture four visits to the workshop: the first to look, the second to ask, the third to work, the fourth to hand something over. The time below is a suggestion, a few hours a week.

Week 1. Read and use the project#

Do:

1. Pick a project you actually use. If you do not use it, it is hard to know what would help.
2. Read the README, CONTRIBUTING, CODE_OF_CONDUCT and, if there is one, the AI policy. Find out whether the project takes pull requests from strangers at all.
3. Read about five accepted pull requests and five closed ones. Notice how people write descriptions and why things get closed.
4. Install and run the project on your machine, and run its test suite once.

Take away: five lines of "house rules of this project" in your own words, plus the command that runs the tests.

Week 2. Pick a small accepted job and ask first#

Do:

1. Find an issue with a newcomer-friendly label (such as good first issue) that nobody has claimed.
2. Read the issue and its comments carefully. An issue confirmed by a maintainer is worth doing.
3. Comment to ask to take it, describe your planned approach in two or three sentences, and wait for an answer before writing code.
4. If nobody answers for several days, do not push ahead with a large change. Ask again or pick another job.

Take away: one short, polite, clear comment that a maintainer has answered.

Week 3. Make a small, tested change you understand#

Do:

1. Create your own branch. Write a failing test that reproduces the problem, then fix it until the test passes.
2. Keep one single purpose. Do not reformat files or rename things in bulk in the same PR.
3. Run the project's whole test suite and note how many tests ran.
4. Use AI if the project allows it, but take this test: switch the AI off and explain every line you changed out loud, to someone else or to yourself. Where you cannot explain, you do not understand yet, and you are not ready to submit.

Take away: one branch with one clean commit and a test that runs.

Week 4. Open a clear PR, disclose AI and take feedback calmly#

Do:

1. Write the PR description with the template in section 5: problem, fix, how to check, and the AI disclosure line the project asks for.
2. Review your own diff before sending (see Reviewing AI-written code).
3. When feedback arrives, reply briefly, change exactly what was asked, and rebase if the project has moved on.
4. If it is rejected, say thanks, ask if it is unclear, and do not argue.

Take away: a PR whose purpose and checking method the maintainer sees at a glance.

4. A real walkthrough: the git flow of one small contribution (skip it if you do not use git yet)#

Picture practising in a mock workshop before entering a real one. I built a "mock workshop" on my machine: a bare repository as the project's "original", and a clone as "your fork". I really ran every command below with Git 2.56.0 and Python 3.14.8. Commit hashes will differ on your machine.

Note: this is not GitHub. On GitHub you press Fork on the web, push a branch to your fork and open a pull request on the web. What they share are the git commands: branch, commit, rebase, amend. In the example the "project" is a small library called textkit with a slugify function that turns a title into a short piece of a web address. The "problem" is that slugify("Hello World!") returns hello-world- with a stray dash at the end.

4.1 Before fixing: run the tests, create a branch#

$ python3 -m unittest -q
Ran 1 test in 0.000s

OK

$ git switch -c fix-slug-trailing-dash

My first run failed silently. It printed "Ran 0 tests" and "NO TESTS RAN", because the tests folder lacked an __init__.py file so the test runner found nothing. No line was red, no test ran. That is exactly why Make AI Output Prove Itself tells you to always note how many tests ran: "no errors" does not mean "checked".

4.2 Write the failing test first#

I added a test for a case with an exclamation mark and spaces at both ends, then ran it before fixing:

$ python3 -m unittest -q
FAIL: test_no_trailing_dash (tests.test_slug.SlugTests.test_no_trailing_dash)
Traceback (most recent call last):
  File "tests/test_slug.py", line 10, in test_no_trailing_dash
    self.assertEqual(slugify("Hello World!"), "hello-world")
AssertionError: 'hello-world-' != 'hello-world'

Ran 2 tests in 0.000s

FAILED (failures=1)

The test fails for exactly the expected reason. If a test passes straight away before the fix, or fails for another reason, it is checking something other than your problem.

4.3 Fix and rerun#

$ python3 -m unittest -q
Ran 2 tests in 0.000s

OK

$ git diff --stat
 tests/test_slug.py | 4 ++++
 textkit/slug.py    | 2 +-
 2 files changed, 5 insertions(+), 1 deletion(-)

Two files, five lines added and one changed: small enough for a maintainer to look at in a few minutes.

4.4 A commit with a clear message and a disclosure line#

fix(slug): strip leading and trailing dashes

slugify("Hello World!") returned "hello-world-". Strip dashes at both
ends and add a test for punctuation and surrounding spaces.

AI assistance: used an AI tool to draft the test; I read and ran it.

The last line is one way a project may ask for disclosure. Use the way of the project you contribute to, not this template.

4.5 The project moved on: fetch and rebase#

While you worked, the maintainer pushed one more commit to the main branch. You fetch it and move your change on top of the latest version:

$ git fetch origin
$ git log --oneline --graph --all
* 4cbdd85 docs: add a line on review etiquette
| * 0606d8e fix(slug): strip leading and trailing dashes
|/
* 7835b68 chore: initial textkit

$ git rebase origin/main
Successfully rebased and updated refs/heads/fix-slug-trailing-dash.

$ git log --oneline --graph --all
* ea41359 fix(slug): strip leading and trailing dashes
* 4cbdd85 docs: add a line on review etiquette
| * 0606d8e fix(slug): strip leading and trailing dashes
|/
* 7835b68 chore: initial textkit

The same change got a new hash (from 0606d8e to ea41359) because it now sits on a new base. The old commit 0606d8e still sits on the branch pushed earlier.

4.6 Reviewer feedback: amend and push the new version#

Suppose the reviewer asks for one more test, for the empty string. I add the test, run it, then amend the old commit rather than adding a patch-up commit:

$ python3 -m unittest -q
Ran 3 tests in 0.000s

OK

$ git commit --amend --no-edit
$ git push --force-with-lease origin fix-slug-trailing-dash
 + 0606d8e (old), d109624 (new): branch fix-slug-trailing-dash (forced update)

$ git diff origin/main --stat
 tests/test_slug.py | 7 +++++++
 textkit/slug.py    | 2 +-
 2 files changed, 8 insertions(+), 1 deletion(-)

Real Git prints one line joining the old and new hashes with its own symbols; I rewrote it for readability, and the result is that hash 0606d8e was replaced by d109624 on the server. --force-with-lease only overwrites when the branch on the server is what you expect. Use it only on your own branch (the PR branch), never on the project's main branch. Some projects want you to add new commits instead of amending, so check the project's guide.

4.7 Another way: make a patch file#

Some communities accept changes as patch files sent by email:

$ git format-patch -1 -o ../patches
../patches/0001-fix-slug-strip-leading-and-trailing-dashes.patch

This is rarer on GitHub but still used by many large projects. Each project's CONTRIBUTING says which way it uses.

A note on the example: it is a one-person practice repository, with no real reviewer and no web interface. It shows the commands run and give the output pasted here, not how a real conversation goes.

5. A PR description template and a self-check list#

5.1 Pull request description template#

## Problem
One or two sentences: what is wrong, when you hit it. Link the issue (Closes #123).

## Fix
One or two sentences: what you changed and why this way.

## How to check
- Test command: python3 -m unittest
- Tests run: 3, all pass
- New test failed before the fix: yes

## AI disclosure
As the project requires: whether AI was used, which tool, for which part.
I have read, run and understood the whole change.

5.2 Six-question self-check before you press send#

1. Can I explain every line of the change without AI?
2. Did I run the project's whole test suite, and do I know how many tests ran?
3. Does this PR have one purpose only?
4. Have I read the project's CONTRIBUTING and AI policy, and followed them?
5. Has a maintainer accepted this job in the issue?
6. Does the description say how to check, so the reviewer can follow along?

If any answer is "no", stop and finish that first.

6. Common mistakes#

Picture the visitor in the workshop: some things visitors do make the owner sigh.

7. Illustration: two mornings with the same issue#

This is an illustration that names no real person or project. Two contributors see the same small issue.

The first does not read the rules:

• 09:00. Opens the issue, sees an "easy job", asks AI to generate code.
• 09:40. The AI produces a 600-line change that includes reformatting three files and an extra optimisation. The contributor has not read it all, opens a PR with no AI disclosure.
• 14:00. The maintainer closes the PR with a reminder to read CONTRIBUTING: the work was not accepted and is too large. The contributor lost a morning and does not know what went wrong.

The second follows the four-week plan:

• 09:00. Reads CONTRIBUTING and sees the project requires AI disclosure and asking first.
• 09:20. Comments to ask to take the job, describing the fix in two sentences.
• Next day, 10:00. The maintainer replies "Fine, go ahead".
• Next day, 15:00. Writes the failing test first, fixes, runs the whole suite, rereads every line without AI, then opens a PR that adds only about 8 lines, with the disclosure line.
• Day three, 09:00. The reviewer asks for one more test and the contributor amends in 15 minutes.

What to do: stop yourself at the ask-first step and the explain-yourself step. These two steps cost about half an hour, but often save both sides a whole morning.

This illustration is not evidence. The evidence is the Linux Foundation survey and the curl case in section 2: real maintainers are under heavy load and closing doors, so a contribution that is easy to review is a real gift. Those two sources are estimates and an insider's account, not controlled experiments.

8. Exercises with answers#

Exercise A. Find the flaws in a PR description#

A PR description reads: "Fix stuff. Fixed the bug, used AI to be quick. Also reformatted the whole folder. Tested."

Find at least five weaknesses.

Answer: (1) it names no concrete problem and links no issue; (2) "used AI to be quick" names no tool, no part, and does not say you read and understood it; (3) reformatting the whole folder is a second purpose and belongs in a separate PR; (4) "Tested" gives no command, no test count, and does not say whether the new test failed before the fix; (5) there is no way for the reviewer to check it themselves; (6) the title "Fix stuff" says nothing.

Exercise B. Ask first or do it now?#

In the practice repository, slugify("Xin chào") returns xin-ch-o (I really ran this). You see accented letters disappear. Should you open a fixing PR straight away or comment first?

Answer: comment first. How to treat accented letters is a design decision: strip the accents (xin-chao), keep them, or transliterate. The maintainer may have a reason for the current behaviour or want a different approach. Opening a PR at once may get it closed even if your code works. A comment of two or three sentences describing what happens, a few options, and asking which direction they want is enough.

Exercise C. A test that passes before the fix#

You add a test slugify("Hello---World") == "hello-world". Running it passes at once with no change (I really ran this). What do you do?

Answer: this is not a bug. The function already collapses runs of dashes into one. A test that passes already may still be worth adding to protect existing behaviour, but it is not evidence that you fixed a bug, and it should not be packaged as a "fix". Go back and look at what the real problem is.

9. A balanced view: what is known, what is not#

Fairly solid: maintainers report growing contribution volume with a notable share of low quality; at least one large project (curl) closed a door for this reason; some projects (such as Ghostty) have clear AI policies.

Unclear or worth caution:

• The 25% and 31% figures are survey respondents' estimates, not a count. I have not read the full report.
• The curl case is one project, one insider, and the cash bounty may make it different from projects without one.
• Policies change fast. This post states only what I read at the time of reading.
• I have not seen research measuring directly whether following the four-week plan raises the rate of being merged. The plan is a synthesis of practice, not an experimental result.

Do not overstate. Nothing guarantees your PR will be accepted. A project may refuse for reasons unrelated to you: a release freeze, a change of direction, or simply running out of energy.

Do not downplay. Some projects ban AI-generated contributions outright. When you meet one, comply and do not try to get around it.

10. Conclusion#

Contributing to open source in the AI era is an act of respect: read the rules first, ask first, send little and send it easy to review, and understand what you send. AI helps you write faster, but it also helps you send things you do not understand faster. The four weeks above put the stopping points in the right places.

If you want to practise this git flow, do it in a practice repository first. See also Git for People Who Use Agents for rescue commands, Debugging Like a Detective for reproducing a bug before fixing it, and Learning an Unfamiliar Library with AI Without Losing the Skill for learning a new project while keeping your craft.

References#

Figures come from public sources as of October 2026. Individual projects' policies may have changed, so open the sources to check before you act.

• Linux Foundation, press release on the survey "Navigating Open Source Software Development in the Generative AI Era" (20 October 2026) and a news summary
• Daniel Stenberg, The end of the curl bug-bounty (26 January 2026)
• GitHub Changelog, New repository settings for configuring pull request access (13 February 2026)
• Ghostty, CONTRIBUTING.md and AI_POLICY.md (read in October 2026, the policy may have changed)
• Git, the git-rebase, git-push (--force-with-lease) and git-format-patch documentation

For anyone who hands work to an AI agent, beginners included. You need no background in monitoring or security. Every term comes with an everyday picture, and the code is marked so you can skip it for now.

Numbers come from public sources listed at the end. Examples marked "illustration" are reconstructed and name no real person or company. The worked example is a simulation I wrote in Python, not the log of a commercial tool. How an AI decides what to do inside is not fully understood by anyone, so the pictures below are memory aids only.

Quick summary#

• When you hand work to an agent and go to a meeting, you do not see each step. Without a trail, when something goes wrong the only option is to reread a long conversation and guess.
• An action trail is one line per thing the agent did: when, which tool or command, what result, which files changed. One JSON record per line (the JSON Lines format) can be searched with a few lines of code or the jq command.
• Record just enough. No secrets, no personal data, no whole file contents. Hide secrets before writing, because a log that leaks a key is as dangerous as code that leaks a key.
• The recording layer should sit outside the agent and accept appended lines only. A log the agent can edit is no longer evidence.
• In the worked example, 25 actions produced a 7,170 byte file. The three investigation questions (timeline, first failure, changed files) are answered as soon as the script runs, and an edited line is detected.
• A trail shows what happened, not whether it was right. It complements asking for evidence and testing, it does not replace them.

After this post you can explain#

1. Why a session nobody watched still needs a trail, and how a trail differs from a saved conversation.
2. Which fields each log line should have, and what must never be recorded.
3. How to use a trail after an incident: rebuild the timeline, find the first failure, list the changed files and detect an edited line.

Hard words, plain meaning#

1. A picture first: shop CCTV and a receipt book#

Imagine you open a shop and hire a very quick employee. Today you are away. When you return, the till is short by 200 thousand. If nothing recorded the day, all you can do is ask the employee, who can answer perfectly honestly and still remember wrongly.

Now imagine the shop has two things. A camera records what happens at the counter. A receipt book records each sale: what time, what was sold, how much was taken. You do not need to watch a whole day of video. You open the book, find the sale at 3 pm and see the gap at once. Aircraft work the same way: the flight recorder keeps the pilots' actions so that after an accident people can rebuild what happened even when nobody is left to ask.

An agent works like that quick employee: it runs dozens of commands in a few minutes. The conversation is the story it tells. The action trail is the receipt book: it records what was done, not what was said.

Takeaway: a trail is not for catching the employee. It lets you find the cause in minutes instead of hours of guessing.

2. Why a session nobody rewatches needs a trail#

The more work you delegate, the fewer steps you watch. The landing page of Anthropic's 2026 agentic coding trends report says engineering leaders are balancing productivity against oversight, quality and security, and lists multi-agent coordination among its topics. I could read only the landing page. The full report sits behind a sign-up form, so I have not read the body of the report and I draw nothing further from it.

Another number: the Stack Overflow 2026 survey reports that 69.9% of respondents ask an AI agent when they face a new challenge (a figure I read through a summary and did not check against each table). Agents are part of the daily routine, and that routine often runs while you are not looking.

Two kinds of loss follow. First: an incident happens and nobody knows which commands the agent ran, so the investigation is slow. Second: no incident, but a colleague asks "why was the test file changed?" and you reread a long conversation to answer. The post Make AI Output Prove Itself asks for evidence inside the agent's answer. A trail adds the rest: evidence outside the answer, written by another layer that the agent cannot write.

Takeaway: more delegation needs more trail. A trail is cheaper than rereading.

3. What to record and what not to#

3.1 Seven groups of information#

Picture a receipt: who sold, what, when, how much, under which order.

Add a prev field (the fingerprint of the previous line) if you want to detect edited lines, as section 5 does.

3.2 Three things never to record#

1. Secrets and passwords. Hide them before writing. Logs are usually read by more people than a config file, so a key in a log travels further than a key in .env.
2. Users' personal data. Record only masked identifiers or internal ids.
3. Whole file contents or long command output. Record a summary (number of lines, number of passing tests) and where to look it up, not the content. The log stays small and carries less risk.

If your team already has a one-page AI use policy, put these three rules in its "forbidden" column and set a review date for how long logs are kept.

3.3 What one line looks like#

{"seq": 12, "ts": "2026-10-31T09:01:27+07:00", "session": "s-001",
 "actor": "agent", "request": "R2: make the tests pass",
 "tool": "run_command", "args": {"cmd": "python -m pytest -q"},
 "exit_code": 1, "summary": "ran 20 tests, 18 passed, 2 failed",
 "files_changed": [], "prev": "5d5c9ffa0389"}

(In a real file each record sits on a single line. It is wrapped here for reading.)

Takeaway: the seven groups answer the five investigation questions: who, what, when, why, how far. The three prohibitions keep the notebook from becoming a new leak.

4. The mechanism: four links#

Picture a production line: action, recording layer, storage, query.

1. The agent acts. It runs a command, edits a file, calls a tool.
2. A recording layer captures the action. This layer lives in the tool or in the harness that runs the agent, not in the prompt. If you only tell the agent "remember to log yourself", it may forget or log wrongly.
3. Hide secrets, then write to an append-only place. Add lines, never reopen old lines to edit.
4. Query when needed. Read with a script or jq.

4.1 Does your tool have a place to attach#

According to the current hooks documentation of Claude Code (I opened the page), the tool lets you attach a command that runs before or after each tool call. The command receives a JSON object on standard input (stdin). The fields the documentation lists include session_id, transcript_path, cwd, hook_event_name, and for the after-call event also tool_name, tool_input, tool_response, tool_use_id and an optional duration_ms. A separate event covers a tool that fails, with an error field. The documentation also says a hook that exits with code 0 and prints no decision does not change the tool call, while exit code 2 blocks it, so never use exit code 2 for logging.

About session transcripts: the example paths in the documentation end in .jsonl, so transcripts are stored as JSON Lines. The documentation does not state the format in words, so this is inferred from the examples. It also says the transcript file may be written with a delay.

Here is a logging script I wrote. It uses jq to pick the fields and hide keys:

#!/bin/sh
# Append one redacted JSON line per tool call. Always exit 0.
jq -c '{
  ts: (now | todate),
  session: .session_id,
  event: .hook_event_name,
  tool: .tool_name,
  input: (.tool_input | tostring | gsub("sk-[A-Za-z0-9]{8,}"; "[REDACTED]")),
  ms: .duration_ms
}' >> "${TRAIL_FILE:-hook-trail.jsonl}"
exit 0

I ran this script with a fake JSON object shaped like the documentation, and it wrote one line, with the fake key sk-test1234567890abcd replaced by [REDACTED]. I have not run it inside a real Claude Code session. The syntax for declaring a hook (the matcher key and where it goes in the settings file) varies between versions, so read the documentation for your version before attaching it.

4.2 A shared standard: OpenTelemetry#

If you prefer a standard to naming fields yourself, the OpenTelemetry project has naming conventions for generative AI work, including agent actions and tool calls. According to the summaries I read, the set is still at the "Development" stage (not stable), so attribute names may change between versions. The official page I opened only says the conventions moved to a separate repository, so the "Development" status comes from secondary sources, please check the official repository. For a beginner, starting with your own JSON Lines names and migrating later is fine.

Takeaway: the recording layer sits outside the agent, writes after each action and hides secrets before storing.

5. Worked example run for real: 25 actions, one failure, one fake key#

This is a simulated agent session (Python 3.14.8, jq 1.7.1) that I wrote. The actions are scripted data, but the recording layer, the redaction, the hash chain and the queries are real code that ran, and the results below are real output. Both full scripts are in the appendix at the end.

The scenario (illustration): the agent receives two requests in a row. R1 is to add a 10 percent discount to checkout. R2 is to make the tests pass. During the session the agent reads a .env file containing a fake key (API_KEY=sk-test1234567890abcd), runs the tests once with a failure (action 12), fixes the code, reruns the tests successfully, then commits twice.

5.1 The fake key was hidden#

Action 7 reads .env. The recorded line:

{"actor": "agent", "args": {"path": ".env"}, "exit_code": 0, "files_changed": [], "prev": "2efa822af966", "request": "R1: add a 10 percent discount to checkout", "seq": 7, "session": "s-001", "summary": "loaded [REDACTED] (3 variables)", "tool": "read_file", "ts": "2026-10-31T09:00:13+07:00"}

Searching the whole file for sk-test finds 0 matching lines. The redaction is a simple regular expression, so it only catches the key shapes you predicted. Treat it as the first line of defence, not the only one.

A bug I hit while writing this post: my first version turned the arguments into a JSON string and only then hid secrets. When I tried a command containing password=abc123, the \S+ pattern swallowed the closing quote and brace, truncating the line and making the query script fail while reading it. The fix is to hide each string value before turning it into JSON, as the redact function in the scripts below does.

5.2 The timeline#

== timeline ==
 1 09:00:03 read_file    exit=0 src/checkout.py
 2 09:00:05 read_file    exit=0 src/pricing.py
 3 09:00:07 search       exit=0 discount
 4 09:00:09 read_file    exit=0 tests/test_discount.py
 5 09:00:10 run_command  exit=0 git status --short
 6 09:00:11 run_command  exit=0 git switch -c agent/discount
 7 09:00:13 read_file    exit=0 .env
 8 09:00:33 edit_file    exit=0 src/pricing.py
 9 09:00:45 edit_file    exit=0 src/checkout.py
10 09:00:54 run_command  exit=0 python -m pytest -q
11 09:01:19 edit_file    exit=0 tests/test_discount.py
12 09:01:27 run_command  exit=1 python -m pytest -q
13 09:01:29 read_file    exit=0 tests/test_discount.py
14 09:01:44 edit_file    exit=0 src/pricing.py
15 09:01:52 run_command  exit=0 python -m pytest -q
16 09:01:53 run_command  exit=0 git diff --stat
17 09:01:54 run_command  exit=0 git add -A
18 09:01:56 run_command  exit=0 git commit -m 'feat: add 10 percent discount'
19 09:01:58 read_file    exit=0 README.md
20 09:02:08 edit_file    exit=0 README.md
21 09:02:09 run_command  exit=0 git add README.md
22 09:02:11 run_command  exit=0 git commit -m 'docs: mention discount'
23 09:02:19 run_command  exit=0 python -m pytest -q
24 09:02:20 run_command  exit=0 git log --oneline -3
25 09:02:24 final_answer exit=0 -

5.3 Three investigation questions#

== first failure ==
12 2026-10-31T09:01:27+07:00 run_command ran 20 tests, 18 passed, 2 failed
== files changed ==
README.md: 1 edit(s)
src/checkout.py: 1 edit(s)
src/pricing.py: 2 edit(s)
tests/test_discount.py: 1 edit(s)
== edits to test files ==
11 R2: make the tests pass ['tests/test_discount.py']

Reading it: the first failure is action 12 at 09:01:27. Just before it (action 11) the agent added 22 lines to the test file, and after the failure it edited src/pricing.py (action 14) rather than editing the test file again. In other words the trail shows the agent fixed the code to make the tests pass, not the tests to match the code. If action 14 had been an edit to a test file, that would be the place to look closely.

The same questions with jq (no Python needed):

jq -r 'select(.exit_code != 0) | "\(.seq) \(.ts) \(.summary)"' trail.jsonl
# 12 2026-10-31T09:01:27+07:00 ran 20 tests, 18 passed, 2 failed

jq -r '.files_changed[]' trail.jsonl | sort | uniq -c
#    1 README.md
#    1 src/checkout.py
#    2 src/pricing.py
#    1 tests/test_discount.py

5.4 Detecting an edited line#

I copied the log file and changed the exit code on line 12 from 1 to 0 to "erase" the failure. Then I reran the chain check:

== chain check ==
line 13: chain broken, an earlier line was changed
chain NOT ok
== first failure ==
none found

Line 13 carries the fingerprint of the original line 12, so once line 12 changed the fingerprint no longer matched. The "none found" under first failure shows that if you skip the chain check you would conclude the session ran clean.

An honest limit: a hash chain only detects, it does not prevent. Someone who edits the log can recompute the whole chain from the edited line onward. For the chain to mean anything you must keep the fingerprint of the last line somewhere the editor cannot reach (another service, an append-only system, or a second person).

6. A storage calculation (illustration, skippable)#

The 25-line file weighs 7,170 bytes, about 286.8 bytes per line. Suppose a team has 200 sessions a day with 100 actions each (assumed numbers for the calculation):

per_line = 7170 / 25               # 286.8 bytes
per_day = 200 * 100 * per_line     # 5,736,000 bytes
print(round(per_day / 1e6, 2), round(per_day * 30 / 1e6, 1))
# 5.74 172.1   (MB per day, MB per 30 days)

About 5.74 MB a day and 172.1 MB for 30 days. Disk space is not the big problem for summary logs. The bigger questions are who may read them and how long to keep them. The 200 and 100 are my assumptions, while 286.8 bytes was measured on this simulation. A real log with longer output will be much heavier, which is why you do not store whole contents.

7. Reading the trail back after something went wrong#

Picture an investigator opening the receipt book. Five questions in order:

1. Is the book intact? Check the hash chain or compare with a copy kept elsewhere.
2. When did it start? Find the first failure and look at the actions just before it.
3. Who or what gave the order? Look at the request and actor fields.
4. How far did it reach? List changed files and commands with outside effects (commit, push, delete).
5. What stops it happening again? That lives outside the log: write an incident review. The story of the day an engineer deleted the wrong data at GitLab in 2017 is one example of how a team told the story of an event after it happened.

A trail helps with questions 1 to 4. Question 5 is human work.

8. Where trails break#

Picture a notebook that can have missing pages, wrong entries or erasures.

9. Illustration: one morning, two versions#

This is an illustrated situation that names no real person or product.

Without an action trail:

• 09:00. A dev gives the agent two jobs and goes into a meeting until 11:30.
• 11:35. The dev sees two commits on the branch. A colleague asks: "Why was the test file changed?"
• 11:40. The dev opens the long conversation and scrolls. The agent said "fixed" but not which files in which order.
• 12:20. The dev guesses an answer, unsure.

With an action trail:

• 09:00. Same two jobs, the recording layer runs outside the agent.
• 11:35. The colleague asks the same question.
• 11:37. The dev queries "test files edited": action 11 at 09:01:19 added new tests, action 14 edited the code, not the tests.
• 11:40. The answer goes back with action numbers attached.

What to do in both: tie the answer to a numbered action, not to memory. This is an illustration, so it proves nothing about frequency. The evidence is in the sources at the end.

10. Hands-on exercise (with answers)#

Run the two scripts from the appendix, then do four things.

1. How many run_command actions failed, and at which action number?
2. Which file was edited more than once?
3. Add two actions at the end of the steps list: a command echo password=abc123 and a command echo pwd=abc123. Rerun and count the lines in the file that contain abc123.
4. Edit one line in the middle of the file by hand and run the chain check. Which line is reported broken?

Answers:

1. One command failed, at action 12.
2. src/pricing.py, edited twice (actions 8 and 14).
3. Exactly one line contains abc123. The password=abc123 command was hidden as echo [REDACTED], while pwd=abc123 slipped through because the pattern has no pwd word (I ran this, and the result was 1 line). That is the lesson about the limits of hiding with a fixed pattern.
4. The line right after the one you edited, because it holds the fingerprint of the original line.

11. Sources and numbers used#

Reading it correctly: I did not find a study measuring how much an action trail shortens investigation time. The arguments in this post are good practice derived from how traces work, not experimental results.

12. Checklist: a trail for your agent#

1. Is there a recording layer outside the agent, or did you only tell the agent to log itself?
2. Does each line have time, session, actor, original request, tool, result and changed files?
3. Are secrets hidden before writing, and have you tested with a fake key?
4. Are you recording personal data or whole file contents?
5. Is the log append-only, or can the agent edit it?
6. Have you stored the last fingerprint or a copy somewhere else?
7. Do you have sample queries for three questions: timeline, first failure, changed files?
8. Who may read it, how long is it kept, and when is the review date?

13. A balanced view: what a trail does and does not do#

It does: shorten rebuilding an event, answer "what did the agent do at what time", give a colleague a numbered basis, and expose an edited line if you check the chain.

It does not:

• A trail does not say whether a result is right. A successful command can still do the wrong thing. For correctness you still need evidence handed in with the answer as in Make AI Output Prove Itself, tests and a human reader.
• It does not prevent damage. It only helps you know afterwards. For risky work you also need restricted permissions and a way back, like the save points in Git for People Who Use Agents.
• Hiding secrets with a fixed pattern misses key shapes you have not thought of.
• A trail costs storage and needs privacy care: who reads it and for how long.
• The worked example is only a simulation of one session. A real tool's trail has more fields and more exceptions.

Do not oversell it: having a trail does not mean being safe. Do not underrate it: when you need it most, during an incident, it is often the only thing left to rely on.

14. Conclusion#

Handing work to an agent and going to a meeting is normal. What is not normal is coming back and not knowing what it did. Give it a receipt book written by another layer: record just enough, hide secrets, allow appending only and know how to read it back. A trail does not replace evidence and tests, but it turns an afternoon of investigation into a few minutes.

Appendix: the two full scripts (skippable)#

agent_trail_demo.py, simulates a session and writes the trail:

import hashlib
import json
import re
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path

LOG = Path(sys.argv[1] if len(sys.argv) > 1 else "trail.jsonl")
SECRET = re.compile(r"sk-[A-Za-z0-9]{8,}|(password|token|api_key)\s*[=:]\s*\S+", re.I)


def redact(value):
    """Hide secrets in every string inside a value, before it is turned into JSON."""
    if isinstance(value, str):
        return SECRET.sub("[REDACTED]", value)
    if isinstance(value, dict):
        return {k: redact(v) for k, v in value.items()}
    return value


class Trail:
    """Append-only JSON Lines trail. Each line carries the hash of the previous line."""

    def __init__(self, path, session):
        self.path, self.session, self.prev, self.seq = path, session, "start", 0
        self.clock = datetime(2026, 10, 31, 9, 0, 0, tzinfo=timezone(timedelta(hours=7)))
        path.write_text("")

    def record(self, request, tool, args, exit_code, summary, files=(), seconds=2):
        self.clock += timedelta(seconds=seconds)
        self.seq += 1
        line = {
            "seq": self.seq,
            "ts": self.clock.isoformat(),
            "session": self.session,
            "actor": "agent",
            "request": request,
            "tool": tool,
            "args": redact(args),
            "exit_code": exit_code,
            "summary": redact(summary),
            "files_changed": list(files),
            "prev": self.prev,
        }
        text = json.dumps(line, ensure_ascii=False, sort_keys=True)
        self.prev = hashlib.sha256(text.encode()).hexdigest()[:12]
        with self.path.open("a") as f:
            f.write(text + "\n")


trail = Trail(LOG, "s-001")
R1 = "R1: add a 10 percent discount to checkout"
R2 = "R2: make the tests pass"
steps = [
    (R1, "read_file", {"path": "src/checkout.py"}, 0, "120 lines", (), 3),
    (R1, "read_file", {"path": "src/pricing.py"}, 0, "80 lines", (), 2),
    (R1, "search", {"pattern": "discount"}, 0, "4 matches in 2 files", (), 2),
    (R1, "read_file", {"path": "tests/test_discount.py"}, 0, "45 lines", (), 2),
    (R1, "run_command", {"cmd": "git status --short"}, 0, "clean", (), 1),
    (R1, "run_command", {"cmd": "git switch -c agent/discount"}, 0, "switched to new branch", (), 1),
    (R1, "read_file", {"path": ".env"}, 0, "loaded API_KEY=sk-test1234567890abcd (3 variables)", (), 2),
    (R1, "edit_file", {"path": "src/pricing.py"}, 0, "+14 -2 lines", ("src/pricing.py",), 20),
    (R1, "edit_file", {"path": "src/checkout.py"}, 0, "+6 -1 lines", ("src/checkout.py",), 12),
    (R1, "run_command", {"cmd": "python -m pytest -q"}, 0, "ran 18 tests, 18 passed", (), 9),
    (R2, "edit_file", {"path": "tests/test_discount.py"}, 0, "+22 -0 lines", ("tests/test_discount.py",), 25),
    (R2, "run_command", {"cmd": "python -m pytest -q"}, 1, "ran 20 tests, 18 passed, 2 failed", (), 8),
    (R2, "read_file", {"path": "tests/test_discount.py"}, 0, "67 lines", (), 2),
    (R2, "edit_file", {"path": "src/pricing.py"}, 0, "+3 -1 lines", ("src/pricing.py",), 15),
    (R2, "run_command", {"cmd": "python -m pytest -q"}, 0, "ran 20 tests, 20 passed", (), 8),
    (R2, "run_command", {"cmd": "git diff --stat"}, 0, "3 files changed, 45 insertions, 4 deletions", (), 1),
    (R2, "run_command", {"cmd": "git add -A"}, 0, "staged 3 files", (), 1),
    (R2, "run_command", {"cmd": "git commit -m 'feat: add 10 percent discount'"}, 0, "1 commit created", (), 2),
    (R2, "read_file", {"path": "README.md"}, 0, "30 lines", (), 2),
    (R2, "edit_file", {"path": "README.md"}, 0, "+5 -0 lines", ("README.md",), 10),
    (R2, "run_command", {"cmd": "git add README.md"}, 0, "staged 1 file", (), 1),
    (R2, "run_command", {"cmd": "git commit -m 'docs: mention discount'"}, 0, "1 commit created", (), 2),
    (R2, "run_command", {"cmd": "python -m pytest -q"}, 0, "ran 20 tests, 20 passed", (), 8),
    (R2, "run_command", {"cmd": "git log --oneline -3"}, 0, "3 commits listed", (), 1),
    (R2, "final_answer", {}, 0, "reported done, branch agent/discount", (), 4),
]
for step in steps:
    trail.record(*step)
print(f"wrote {trail.seq} lines to {LOG}")

query_trail.py, checks the chain and queries:

import hashlib
import json
import sys
from collections import Counter

path = sys.argv[1] if len(sys.argv) > 1 else "trail.jsonl"

print("== chain check ==")
prev, ok = "start", True
for i, line in enumerate(open(path), 1):
    if not line.strip():
        continue
    if json.loads(line)["prev"] != prev:
        print(f"line {i}: chain broken, an earlier line was changed")
        ok = False
        break
    prev = hashlib.sha256(line.rstrip("\n").encode()).hexdigest()[:12]
print("chain ok" if ok else "chain NOT ok")

rows = [json.loads(line) for line in open(path) if line.strip()]

print("== timeline ==")
for r in rows:
    args = r["args"]
    arg = args.get("cmd") or args.get("path") or args.get("pattern") or "-"
    print(f'{r["seq"]:>2} {r["ts"][11:19]} {r["tool"]:<12} exit={r["exit_code"]} {arg}')

print("== first failure ==")
first = next((r for r in rows if r["exit_code"] != 0), None)
print(f'{first["seq"]} {first["ts"]} {first["tool"]} {first["summary"]}' if first else "none found")

print("== files changed ==")
counts = Counter(f for r in rows for f in r["files_changed"])
for name, n in sorted(counts.items()):
    print(f"{name}: {n} edit(s)")

print("== edits to test files ==")
for r in rows:
    if any(f.startswith("tests/") for f in r["files_changed"]):
        print(r["seq"], r["request"], r["files_changed"])

Run: python3 agent_trail_demo.py trail.jsonl then python3 query_trail.py trail.jsonl. The simulated clock is fixed, so the fingerprints (the prev field) in your file will match mine if you use exactly this script.

References#

Numbers come from public sources as of October 2026. Some sources are secondary, so open the originals before quoting.

• Anthropic, 2026 Agentic Coding Trends Report (landing page, the full report is behind a form)
• Stack Overflow, 2026 Developer Survey (read through a summary, not checked table by table)
• Anthropic, Claude Code: Hooks reference (JSON fields, events, exit codes, transcript paths)
• OpenTelemetry, GenAI semantic conventions (page moved repositories) and secondary summaries of the Development status, for example Dash0
• jq, jq Manual (filters and gsub)
• Related posts on this site: Make AI Output Prove Itself, Git for People Who Use Agents, One-Page AI Use Policy for Small Teams

For anyone who writes code or asks an AI to write it, beginners included. No security background needed. Every term comes with an everyday picture, and the sections where you run commands are marked so you can skip them for now.

Every key in this post is a fake key, including the example key pair published in AWS's own documentation. Numbers come from public sources listed at the end. Examples marked "illustration" are reconstructed and name no real person or company. This is a practical guide, not security advice for your particular system.

Quick summary#

• API keys, passwords and access tokens are secrets: whoever holds one can use it, like a house key.
• Deleting a key from a file is not enough. Git remembers every old version. In this post's experiment, after the key was removed in the second commit, git log -S still found both commits and git show printed the key again.
• The right order when a key leaks: rotate or revoke first, clean history second, then read the usage logs. GitHub's documentation says the same.
• Rewriting history does not recall existing copies. In the experiment, a clone made before the rewrite still held the key.
• You can block leaks early with environment variables, .gitignore, a 30-line pre-commit hook and the platform's scanning. Each layer has gaps, so you need several.
• Working with AI agents adds three habits: do not paste real keys into the conversation, keep .env files out of the agent's reach, and read the diff before committing.

Hard words, plain meaning#

After this post you can explain#

1. Why deleting a key from a file does not make it disappear, and how to check with three git commands.
2. The right order when a key leaks (rotate first, clean later) and why rewriting history is not enough.
3. Three layers of defense (keep it out of code, block it at commit, scan after push) and where each one falls short.

1. A picture first: the key under the doormat#

Imagine you keep your house key under the doormat. The next day you realize that is a bad idea and move the key to a drawer. But during that one day a neighbor walking by had a copy made. Putting the key away does not make the copy disappear. The only sure fix is to change the lock.

An API key in code works the same way. When you commit it, git has already stored a copy. Someone may have cloned the repo. An automated scanner may have read it from a public repo. Deleting the line in a later commit is like putting the key in the drawer: good, but it does not deal with the copy.

Takeaway: once a secret has left its hiding place, treat it as exposed and change the lock, not just put it back.

2. What counts as a secret, and where it leaks#

Common places leaks happen:

• Committed straight into git, then deleted in a later commit (section 3 runs this for real).
• A .env file committed because it was never added to .gitignore, or was committed before it was.
• Logs and screenshots containing a key, then sent to a teammate or posted in a forum.
• Pasted into a chat with an AI to "show it the error". That text may be stored and shown again in the conversation history.
• Code that runs in the browser (a client-side bundle): anything sent down to a user's browser can be read, so never put a secret key there.

Takeaway: secrets leak through more than one door, so prevention needs layers.

3. A real experiment: delete it, it is still there#

Every command and output in this section was run for real in a throwaway repo, with Git 2.56.0 and Python 3.14.8, using fake keys only. Commit hashes differ on every run. Timings are from my machine on a 2-commit repo, so they only give you the order of magnitude.

3.1 Commit a fake key, then delete it in the next commit#

git init -q -b main repo
cd repo

# config.py holds the example key pair from the AWS documentation (fake keys)
#   AWS_ACCESS_KEY_ID = "AKIAIOSFODNN7EXAMPLE"
#   AWS_SECRET_ACCESS_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
git add .
git commit -m "add config"             # about 0.03 s

# the next commit switches to environment variables
git add .
git commit -m "remove secret from config"

git log --oneline

2bfea15 remove secret from config
a0d01a2 add config

3.2 It looks clean#

grep -n AKIA config.py          # no match
git grep AKIA                   # no match, it only searches current files

Both commands print nothing. The current file really is clean.

3.3 But the history still has it#

git log --oneline -S AKIAIOSFODNN7EXAMPLE

2bfea15 remove secret from config
a0d01a2 add config

The -S option finds commits that changed how many times a string appears, so it shows both the commit that added it and the commit that removed it. Next, search every commit and print the old file:

git grep -n AKIA $(git rev-list --all)      # about 0.02 s
git show a0d01a2:config.py

a0d01a2473ddcb923dbfd69164c1df8a11de379b:config.py:3:AWS_ACCESS_KEY_ID = "AKIAIOSFODNN7EXAMPLE"

import os

AWS_ACCESS_KEY_ID = "AKIAIOSFODNN7EXAMPLE"
AWS_SECRET_ACCESS_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
REGION = "ap-southeast-1"

Three commands and under a second, and anyone holding this repo can read both keys again.

Takeaway: a clean file does not mean a clean repo. You have to search the history.

4. The right order when a key leaks#

Imagine someone tells you "a copy of your house key was made". You change the lock first, and only then go find out who has been inside.

1. Revoke or rotate the key now. GitHub's secret scanning documentation recommends rotating "immediately" and says removing a secret from git history is slow and often unnecessary once the key has been revoked.
2. Read the usage logs of the key on the service: who used it, from where, when, during the exposure window.
3. Clean the history (section 6), if needed, so the next push does not carry the key.
4. Write it down and put guardrails in place so it does not happen again (section 5).

4.1 Rotating an AWS key: the documented procedure#

The AWS documentation on updating access keys describes a rotation that does not interrupt applications:

1. While the first access key is still active, create a second one (now two keys are active).
2. Update every application and tool to use the new key.
3. Check the "last used" information of the old key to see whether anything still uses it. The documentation suggests you can wait several days.
4. Do not delete the old key right away. Set it to inactive first.
5. Use only the new key to confirm applications still work. Anything still using the old key will break, and you can reactivate the old key temporarily.
6. After waiting a while, delete the old key.

That procedure is written for routine rotation, with time to wait. For a key that has already leaked, disabling the old key immediately is my own inference from GitHub's "revoke first" principle, not something the AWS page says. The cost is that anything still using the old key breaks at once, so have the new key ready.

4.2 What fits in 15 minutes, and what does not#

This is a target I propose, not a measurement. Real timing depends on whether you already have access and know where the key is used.

Takeaway: 15 minutes is enough to close the door. Checking the damage is a job for the hours after.

5. Block it early: several layers, each with gaps#

Imagine a house with three layers: never carry the key out, someone reminds you at the door, and a camera on the street.

5.1 Layer one: keep secrets out of code#

Put secrets in environment variables or a secret store, and let the config file only say "read from this variable". In the experiment, the second commit did exactly that:

AWS_ACCESS_KEY_ID = os.environ["AWS_ACCESS_KEY_ID"]

If you use a .env file locally, add it to .gitignore before creating the file. A real test (I used the name dev.env instead of .env, because my run environment does not let me create a file named .env; it works the same way):

git status --short

?? dev.env
?? tools/

echo "dev.env" > .gitignore
git status --short
git check-ignore -v dev.env

?? .gitignore
?? tools/
.gitignore:1:dev.env    dev.env

dev.env disappears from the list, and check-ignore tells you which line of .gitignore is blocking it. But .gitignore only affects files git is not already tracking. I tried it with a file that was committed earlier:

git add -f old.env
git commit -m "oops: commit old.env"
echo "old.env" >> .gitignore
git ls-files old.env

old.env

Even after adding it to .gitignore, git still tracks the file. To stop tracking it you need git rm --cached old.env, and the key in history stays, as in section 3.

5.2 Layer two: block at commit with a pre-commit hook#

Here is a tiny scanner (about 30 lines of Python) that reads the lines about to be added and looks for three shapes: AWS-style keys, the header of a private key, and lines assigning a string to a name like api_key, secret, token or password:

PATTERNS = {
    "aws access key id": re.compile(r"AKIA[0-9A-Z]{16}"),
    "private key header": re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"),
    "key or password assigned to a string": re.compile(
        r"""(?i)(api[_-]?key|secret|token|passwd|password)\w*\s*[=:]\s*['"][^'"\s]{8,}['"]"""
    ),
}
# reads: git diff --cached -U0, checks lines starting with "+", exits with code 1 on a match

Hook it up in .git/hooks/pre-commit:

#!/bin/sh
exec python3 tools/scan_secrets.py

Trying to commit a file with a fake key:

Blocked: this commit adds something that looks like a secret.
  deploy.py: key or password assigned to a string
Use an environment variable, then rotate the key if it was ever real.
exit code: 1

The commit is refused (the whole attempt took about 0.14 seconds). But this scanner has two weak spots, and I tested both:

• A miss: when a key is split into two strings (part1 = "AKIA", part2 = "IOSFODNN7EXAMPLE", then joined), the commit goes through with exit code 0.
• A false alarm: a line password = "change-me-please" in a sample document is blocked, even though it is just a placeholder.

Takeaway: a scanner that matches text patterns catches common mistakes, not deliberate or clever ones, and sometimes cries wolf. Do not treat it as a guarantee.

The hook lives in .git/hooks, which does not travel with the repo. To share it with a team you need a way to distribute hooks (for example a hook manager in the project), and anyone can still skip it with --no-verify.

5.3 Layer three: scan after the push, and block at the push#

Platforms offer features for this. According to the GitHub documentation pages I read (through a page summarizer):

• Secret scanning automatically detects exposed credentials such as API keys and passwords, and scans "your entire Git history on all branches", plus the text of issues, pull requests, discussions and wikis.
• Push protection blocks a push that contains a secret before it reaches the repository. It recognizes provider patterns, generic patterns (such as private keys and database connection strings) and custom patterns. At the repository level it is off by default, and anyone with write access can bypass it by giving a reason. At the user level, the documentation says it is on by default to stop you pushing secrets to public repositories on GitHub.com.

I read these pages through a summarizer and did not check every word, so open the original pages (listed under references) and look at the settings of your own account. Independent tools such as gitleaks, trufflehog and git-filter-repo exist too, but not run here: I did not install them in my test environment.

6. Cleaning history: what it does and does not do#

After you have rotated the key, you may want the repo to stop containing it. The tool recommended today is git filter-repo, but I could not run it because my test machine does not have it. So I demonstrate with git filter-branch, which ships with Git (the Git documentation page says using it "is not recommended" and suggests git filter-repo, but it runs and is enough to illustrate):

git clone -q repo clone-before       # take a copy BEFORE rewriting
cd repo
git filter-branch -f --tree-filter "python3 redact.py" -- --all   # about 0.8 s, 5 commits

Here redact.py replaces the two fake key strings with REDACTED in every commit. Check:

git log --oneline -S AKIAIOSFODNN7EXAMPLE main

It prints nothing: the main branch is clean. But the key still survives in other places:

git for-each-ref --format='%(refname)'

refs/heads/main
refs/original/refs/heads/main

The tool leaves a backup ref refs/original/refs/heads/main, and git log -S AKIAIOSFODNN7EXAMPLE --all still sees the two old commits. To clean up you must delete the backup ref, expire the reflog and prune objects nothing points to:

git for-each-ref --format='%(refname)' refs/original | xargs -n1 git update-ref -d
git reflog expire --expire=now --all
git gc -q --prune=now                          # about 0.1 s
git log --oneline -S AKIAIOSFODNN7EXAMPLE --all --reflog   # prints nothing
git cat-file -t a0d01a2        # fatal: Not a valid object name

Now this repo is clean for good. And the copy taken before the rewrite?

cd ../clone-before
git log --oneline -S AKIAIOSFODNN7EXAMPLE
git cat-file -t a0d01a2

2bfea15 remove secret from config
a0d01a2 add config
commit

That copy still has the key. This is why cleaning history cannot replace rotating the key: anyone who pulled or forked the repo earlier keeps the old version, and a public repo may have been copied or archived by others.

One more warning: rewriting history changes the commit hashes, so teammates must pull again and may hit conflicts. Warn them first. The post Git for people who use agents explains why force pushing needs care.

7. Working with AI agents: six more habits#

Imagine hiring a contractor: you hand over a key that expires and opens one room, not the master key.

1. Do not paste real keys into the conversation. If the AI needs to see an error, replace the key with a placeholder such as <API_KEY>.
2. Keep .env files out of paths the agent can read, or use your tool's setting that blocks reads. An agent that can read the whole project folder can read the secrets in it.
3. Use narrow, short-lived keys for anything an agent does: just enough for the task, expiring soon.
4. Read the diff before committing. An agent may hard-code a connection string so it "works right away". One git add is enough to put it into history. See Reviewing AI-written code.
5. Do not let agents push to the main branch or run installs and high-privilege commands unattended. See also OWASP Top 10 for AI apps.
6. Put the rule into team policy. The post A one-page AI use policy has a "forbidden" column for this kind of data.

7.1 Illustration: a morning asking an agent to connect a database#

This is an illustration, naming no real person or product. Names and values are invented.

Without those habits:

• 10:00. A dev asks the agent: "Connect the database for me, here is the connection string" and pastes the real one into the conversation.
• 10:10. The agent writes config.py, hard-coding the connection string so it "works right away", and commits.
• 10:20. The dev skims it, sees it run, and pushes.
• 10:45. The platform sends an alert that a secret was detected in the repo. The dev deletes the line and pushes again, thinking it is done.

With those habits:

• 10:00. The dev puts the connection string in an environment variable on their machine and tells the agent: "read it from DATABASE_URL, do not write the value into code".
• 10:10. The agent writes config.py reading the environment variable. The pre-commit hook finds nothing odd.
• 10:20. The dev reads the diff, sees only a variable name, and pushes.

If you landed in the first version, what to do at 10:45 is: rotate the database password immediately, then read the access logs, and only then think about cleaning history. Deleting the line in a later commit solves nothing: history already holds the old string, and the conversation with the AI already contains it.

8. Where people go wrong#

9. Numbers and documents used#

A number I did not use: I tried to check a figure of "41% of 100 apps had exposed secrets or API keys" attributed to a VibeWrench study (March 2026). I could not find a source that confirms it. VibeWrench's product page only says they scanned 100 apps and found 318 vulnerabilities, with no rate of exposed keys. So this post does not use that number.

Reading it correctly: there is good evidence for the mechanism (git remembers history, copies persist) because you can rerun it yourself. There is no trustworthy figure for how often keys leak in general, and I found no study measuring the effect of pre-commit hooks on their own.

10. An exercise with answers (30 minutes)#

Task. Create a throwaway repo and use fake keys (like the AWS example pair above, never real ones), then:

1. Commit a file with a fake key, then remove the key in the next commit.
2. Use git log -S and git show to prove the key is still there.
3. Write a five-step order of actions as if it were a real key.
4. Install a scanning pre-commit hook, then try one commit with a directly assigned key and one with a key split into two strings.
5. Take a copy of the repo, rewrite history with git filter-branch, then check both repos.

Answers.

1. After two commits, git log --oneline shows two lines.
2. git log --oneline -S <key> prints both commits. git show <first commit>:<file> prints the key.
3. A reasonable order: (1) disable the old key on the service, (2) create a new key and update where it is used, (3) check the application, (4) read usage logs, (5) clean history if needed and tell teammates. Grading point: rotation comes before cleaning history.
4. The commit with the directly assigned key is blocked (exit code 1). The one with the split key is not blocked, because the scanner only looks for whole patterns.
5. The rewritten repo is clean after deleting the backup ref, expiring the reflog and running gc; the copy taken before still has the key. Ideas to improve the scanner: scan whole files instead of only added lines, flag strings by randomness, or use a dedicated tool.

11. A checklist before you push#

1. Is there any real secret in this change (key, password, connection string, private key)?
2. Do secrets live in environment variables or a secret store, not in code?
3. Are .env and similar files in .gitignore before the file was created?
4. Did I read the diff and look for odd long strings before committing?
5. Is a scanning hook running locally, and is platform scanning on?
6. Are the keys I use narrow in scope and short-lived?
7. Have I pasted a real key into a conversation with AI? If so, rotate it.
8. If a key leaked right now, do I know where to rotate it, how long it takes and who to tell?

12. A balanced view: what we know, what we do not#

Fairly solid: git keeps history and can reprint a deleted key; rewriting history does not recall existing copies; scanners that match text patterns can miss things and raise false alarms; rotating the key is the root step. All four were run for real in this post.

Unclear or limited:

• How often keys leak in the wild and how much damage follows. Available numbers mostly come from security vendors scanning their own samples, and one number I could not confirm was dropped.
• The real effectiveness of each defensive layer. I found no study measuring pre-commit hooks or push protection on their own.
• The dedicated tools (git-filter-repo, gitleaks, trufflehog) I did not run. Read their documentation before using them.
• Platforms differ in features and defaults. I read the GitHub pages through a summarizer and they may have changed.

Do not overstate: no single layer is enough. Do not play it down: the cost of prevention is a few minutes of setup, while a leaked key can let someone else reach your data or resources.

13. Conclusion#

Once an API key has gone into git, treat it as out of your hands. Deleting the line in a later commit is putting the key in a drawer while the copy is still out there. Rotate the key first, read the logs, and only then clean history. To keep it from happening again, keep secrets out of code, block at commit, scan after the push, and with AI agents hand over narrow, short-lived keys and always read the diff before committing.

References#

Numbers come from public sources as of October 2026. I read the GitHub and AWS pages through a page summarizer, so open the originals to check before quoting.

• GitHub Docs, About secret scanning
• GitHub Docs, About push protection
• AWS Documentation, Update access keys (IAM User Guide)
• Escape, The State of Security of Vibe Coded Apps (undated, done by a security vendor on its own)
• Git, git-filter-branch and git-log (the -S option)

For anyone who uses AI to write code or hands work to an AI agent, beginners included. You do not need to know how models work. Every term comes with an everyday picture, and the calculation is marked so you can skip it if you do not need it.

Numbers come from public sources, listed at the end. Examples marked "illustration" are reconstructed and name no real person or company. Every price in this post is an assumption chosen to make the arithmetic easy, not a current price list, because prices change by provider and over time. Nobody fully understands how a model works inside, so the pictures are memory aids only. Billing, however, is documented, and the post follows the documentation.

Quick summary#

• AI bills by the token, a piece of text the model reads in and writes out. The bill is input tokens times the input price, plus output tokens times the output price.
• In a conversation with many turns, the whole history is sent again at every turn. So cost grows faster than the number of turns. In the post's worked example, 15 turns send 322,500 input tokens but only 7,500 output tokens, and doubling the turns raises input by about 3.4 times.
• Prompt caching can sharply cut the cost of re-reading. In the example, cost falls to about 31%. But rules and prices differ between providers, and the cache expires if you pause for long.
• Three ways to stay on budget: short sessions, then a new session from a summary, cheaper models for routine work, spending caps and monitoring.
• The trend is toward usage-based pricing. Gartner (24 June 2026) forecasts that AI coding costs could exceed an average developer's salary by 2028, and GitHub moved Copilot to token-based credits on 1 June 2026.
• Cheapest per token is not cheapest per finished task. Measure cost and rework together.

Hard words, plain meaning#

After this post you can explain#

1. What a token is, and why the same content can take different numbers of tokens depending on the language and how text is split.
2. Why a long conversation gets more expensive faster than the number of turns, and what caching does to soften it.
3. Three ways to keep cost on budget, and why cheapest per token is not necessarily cheapest per finished task.

1. A picture first: the taxi meter and the notebook that must be reread#

Imagine you hire a very good answerer who remembers nothing between questions. Each time you ask, they must reread the whole notebook of everything said so far before writing an answer. They charge by the pages read plus the pages written, the way a taxi meter ticks by the kilometre.

Question one: the notebook is thin, quick and cheap to read. Question fifteen: the notebook has grown, and each reread costs more. What you add each turn may be only a few lines, but what has to be reread keeps growing.

This picture is only a memory aid. What is clearly documented is the billing: AI services charge for the tokens sent in and the tokens produced, and in multi-turn conversations the earlier history is part of the input at every turn (see also The context window).

Takeaway: the bill does not grow with the number of sentences you type. It grows with the number of times the notebook is reread multiplied by how thick it is.

2. What a token is#

2.1 Pieces of text, not words#

A model does not read letter by letter or word by word, but in pieces. A common word may be one piece, a rare word may be cut into several, and punctuation is a piece too. How text is cut differs between models, so I give no specific splits here, because they would be easy to get wrong.

A rule of thumb often quoted for English is about 4 characters per token. I have not verified this against the original page (OpenAI's tokenizer page returned a 403 error when I opened it) and treat it as a rough assumption.

2.2 Same content, different token counts#

Petrov, La Malfa, Torr and Bibi (2023) compared the tokenized length of the same text translated into many languages and found gaps of up to about 15 times in some cases, even for tokenizers built to support many languages. They argue this can lead to higher costs for some language communities.

What this means for you: if you ask AI in Vietnamese or Japanese, the number of tokens per character may be higher than in English. I could not run a token counter in the environment where I wrote this (no tokenizer library was installed), so I give no figure of my own for Vietnamese or Japanese. The right approach is to use the token counter of the tool you actually use and measure your own text. The study measured 2023-era models, and newer ones may differ.

Takeaway: do not guess token counts from word counts. Measure with the provider's counter, on your own content.

3. Input and output prices#

Most providers set different prices for input and output tokens, and output is usually more expensive per token. An example from the price table in Anthropic's documentation, which I read on 30 October 2026 (price per million tokens):

In these three rows output is 5 times input, and the small model is many times cheaper than the large one. Treat this as an example from one provider at one moment. Prices change, model names change, and other providers price differently.

Still, do not conclude that "output is pricey so ignore input". The calculation in section 4 shows that in a long conversation input is the largest part of the bill, because it keeps being repeated.

4. Why the bill balloons: a script you can run#

Picture it: each turn the notebook grows a little and is reread.

4.1 Assumptions (illustration, you can skip the calculation)#

This is an illustrative example with made-up numbers, not a measurement from any particular tool:

• At the start you send 6,000 tokens of instructions and files.
• Each turn adds 1,500 tokens of new content (what you type, tool output).
• Each turn the model writes 500 tokens, which join the history.
• Assumed prices: $2 per million input tokens, $10 per million output tokens.
• With caching: rereading the cached part costs 0.1 times the input price, writing the new part costs 1.25 times. These two factors are the defaults in Anthropic's documentation (some models have a lower read factor).

4.2 The script, rerunnable#

# Illustrative numbers only. Change them to match your own tool and price list.
BASE = 6000           # tokens of instructions and files sent at the start
NEW_IN = 1500         # new tokens added each turn (your message, tool output)
OUT = 500             # tokens the model writes each turn
PRICE_IN = 2.00       # dollars per million input tokens (assumption)
PRICE_OUT = 10.00     # dollars per million output tokens (assumption)
READ = 0.10           # cache read costs 0.1 x input price (assumption)
WRITE = 1.25          # cache write costs 1.25 x input price (assumption)

def run(turns, base=BASE, new_in=NEW_IN, out=OUT):
    sent = 0                      # input tokens sent in total
    cost_plain = 0.0              # no caching: everything at full input price
    cost_cache = 0.0              # with caching: read old part cheaply, write new part
    context = base                # tokens already in the conversation
    prev_in = 0                   # what the previous call sent (it is in the cache)
    for t in range(1, turns + 1):
        this_in = context + new_in            # whole history is sent again
        sent += this_in
        cost_plain += this_in * PRICE_IN / 1e6
        if t == 1:
            cost_cache += this_in * PRICE_IN * WRITE / 1e6
        else:
            cost_cache += prev_in * PRICE_IN * READ / 1e6            # read what was cached
            fresh = this_in - prev_in                                # last output + new content
            cost_cache += fresh * PRICE_IN * WRITE / 1e6             # write the new part
        cost_out = out * PRICE_OUT / 1e6
        cost_plain += cost_out
        cost_cache += cost_out
        prev_in = this_in
        context = this_in + out               # output joins the history
    return sent, turns * out, cost_plain, cost_cache

for turns in (15, 30):
    sent, out_total, plain, cache = run(turns)
    print(f"{turns} turns: input sent {sent:,} tokens, output {out_total:,} tokens, "
          f"no cache ${plain:.4f}, with cache ${cache:.4f}")

# restart halfway: a fresh session instead of one long history (no caching in this line)
first = run(8)
fresh_short = run(7, base=1000)   # starts from a 1,000-token summary
fresh_full = run(7)               # starts again from the full 6,000-token base
one_long = run(15)
print(f"one session of 15 turns:               ${one_long[2]:.4f}")
print(f"8 turns + fresh 7 (1,000 summary):     ${first[2] + fresh_short[2]:.4f}")
print(f"8 turns + fresh 7 (full 6,000 again):  ${first[2] + fresh_full[2]:.4f}")

The real output when I ran it with Python 3.14.8 (I also recomputed it separately with a different snippet and the two agree):

15 turns: input sent 322,500 tokens, output 7,500 tokens, no cache $0.7200, with cache $0.2212
30 turns: input sent 1,095,000 tokens, output 15,000 tokens, no cache $2.3400, with cache $0.5197
one session of 15 turns:               $0.7200
8 turns + fresh 7 (1,000 summary):     $0.4260
8 turns + fresh 7 (full 6,000 again):  $0.4960

4.3 Reading the result#

• Input dominates. With 15 turns, the total input sent is 322,500 tokens against only 7,500 output tokens, so input is 43 times larger. Without caching, input is $0.645 of the $0.72 total (about 90%), and output is $0.075.
• Cost grows faster than turns. Doubling the turns (15 to 30) takes input from 322,500 to 1,095,000 tokens, about 3.4 times, and uncached cost from $0.72 to $2.34, about 3.25 times.
• Caching cuts the rereading sharply. With 15 turns, cost drops to $0.2212, about 31% of $0.72. With 30 turns it is $0.5197, about 22% of $2.34. It still grows with turns, only more slowly.
• A fresh session at the right moment helps. Splitting 15 turns into 8, then 7 in a new session that starts from a 1,000-token summary, costs $0.426, about 41% lower than $0.72 (same uncached accounting). If the new session has to resend the full 6,000 starting tokens, it is $0.496, about 31% lower.

4.4 Limits of the calculation#

This is a simplified model. It ignores real things: the cache expires if you pause for long (Anthropic's documentation gives a default lifetime of 5 minutes, refreshed each time it is used), a minimum length for caching (from 512 to 4,096 tokens depending on the model), other charges (tools, search), and whether your tool uses caching automatically at all. I have not verified this for specific tools. Your absolute numbers will differ. What matters is the shape of the curve: many turns in one history make cost climb faster than the number of turns.

Takeaway: the main dial is not typing less, but keeping the notebook shorter and rereading it fewer times.

5. Three ways to stay on budget#

Picture managing the team's lunch fund: pick the right place for each meal, do not reorder the whole menu every time, and keep a record.

Way 1. Short, fresh sessions with a summary#

When a task is done, do not drag the same conversation into an unrelated one. Ask the agent to summarize what must carry over (decisions made, relevant files, work left), then open a new session. In the example this cuts about 31% to 41% of the uncached cost. A handover summary template is in The context window. Gartner likewise suggests training developers to send only relevant context and to summarize or trim input.

Way 2. Cheaper models for routine work#

Simple, repetitive work (renaming, generating sample data, formatting) often does not need the strongest model. With the assumed prices divided by 4 ($0.50 input and $2.50 output per million tokens), the same 15 turns cost $0.18 instead of $0.72. Gartner advises routing simple, high-volume tasks to smaller models and saving stronger ones for complex work. But keep the balance section below in mind: if a cheaper model needs more turns or leaves mistakes, total cost may not fall.

Way 3. Caps and monitoring#

• Set spending caps per person or team. According to GitHub's post of 27 April 2026, organizations can pool credits and administrators can set budgets at the enterprise, cost center and user level, and when the pool runs out they can allow extra use at published rates or cap spending.
• Look at the tool's usage page every week instead of waiting for the monthly invoice. GitHub says a preview bill was available from early May 2026.
• Gartner suggests setting token thresholds, escalation rules and automated monitoring, and reviewing high-token workflows in sprint retrospectives.

Takeaway: the three work together. Short sessions reduce the tokens, cheaper models reduce the price per token, and caps plus monitoring stop you being surprised.

6. Why this topic is heating up: the move to usage-based pricing#

6.1 Gartner's forecast (24 June 2026)#

According to a reprint of Gartner's release on DevOps Digest (Gartner's own page returned a 403 error when I opened it), Gartner forecasts that by 2028 AI coding costs will exceed an average developer's salary. The reasons listed include rising token consumption, the shift from per-seat licenses to usage-based billing, weak governance of usage (agents running on their own, bloated context windows) and vendors not yet having mature cost-optimization tools. The reprint gives no percentages or dollar amounts for that comparison, so treat it as a forecast of direction, not a measured number.

6.2 GitHub Copilot moves to token-based credits (1 June 2026)#

According to GitHub's post of 27 April 2026, which I read through a page-summarizing tool, from 1 June 2026 Copilot usage consumes GitHub AI Credits, calculated from token consumption (input, output and cached tokens) at each model's published API rates. Monthly plans include credits equal to the plan price (for example Pro $10, Business $19 per user per month). Code completions and next edit suggestions do not consume credits. The fallback to a cheaper model when a limit is reached is removed. The page does not state how many dollars one credit is worth (secondary sources give different figures), so I do not use that number.

6.3 Dollar figures per developer: rough reference only#

Blogs such as CloudZero, Eden AI or Exceeds AI give ranges from a few hundred dollars per developer per month to much higher for heavy agent users. These are companies that sell cost-management tools, so they have a commercial interest, and the methods are unclear. I do not draw conclusions from those numbers. Look at your own team's real bill.

Takeaway: the direction is clear, from fixed prices to pay for use. The only certain approach is to measure your own usage.

7. Where the budget swells#

Picture a tap running when nobody is looking.

7.1 Illustration: one morning, two ways#

This is an illustrative situation that names no real person or product. Numbers follow the model in section 4.

Way one, stretching one conversation:

• 09:00. A developer asks an agent to fix a display bug and keeps going in the same conversation.
• 09:45. The first task is done after about 15 turns. The developer asks for an unrelated task in the same conversation, believing the agent "already understands the project".
• 11:30. About 30 turns in total. With the post's assumed numbers, uncached cost is about $2.34 instead of $0.72 for 15 turns.

Way two, close the task and open a new session:

• 09:45. The first task is done. The developer asks the agent to summarize the decisions made and the relevant files in about 1,000 tokens, then opens a new session for the second task.
• 11:30. Both tasks are done, each in a short history. With the assumed numbers, two 15-turn sessions cost about $1.44 (two times $0.72) instead of $2.34, about 38% lower. This is multiplication from the script's results, not a real measurement.

What to do: make opening a new session at the boundary between tasks a habit, and check the usage page at the end of the day. This illustration proves nothing. The evidence is in how billing is documented by providers and in Gartner's recommendations above.

8. Exercise with an answer key#

What to do: run the script in section 4.2 and compare.

1. Before running, guess: is the cost of 15 uncached turns close to $0.5, $1 or $2?
2. Run it and note four numbers: input sent, output, uncached cost, cached cost.
3. Change exactly one assumption, rerun and compare. Three suggestions: change NEW_IN from 1500 to 3000 (a tool prints a long log), change the number of turns from 15 to 30, or divide both prices by 4.

Answer key to check against (same assumptions, Python 3.14.8):

How to read it: doubling the new content per turn raises cost by about 50%, doubling the turns raises it by about 3.25 times, and a model 4 times cheaper cuts cost exactly 4 times (because prices are multipliers). Compare your guess from step 1 with the real result: the gap tells you whether you tend to guess low or high, and which part you overlook when you forget that history is resent.

A real exercise (30 minutes): open the usage page of the tool you actually use. Pick a task you just finished and note the number of turns, input tokens, output tokens and cost. If the page does not show them, note that. It is a finding too.

9. Checklist: eight questions#

1. Does my tool bill by token or at a fixed price, and does the plan have a usage allowance?
2. Do I know this week's usage, or am I just waiting for the invoice?
3. How many turns is the current conversation, and is the task still related to the first turns?
4. Am I sending a whole file or a long log when I need only part of it?
5. Does this task need the strongest model, or is a cheaper one enough?
6. Does the tool set a maximum number of steps or a spending cap for the agent?
7. Is caching on, and how long does it last?
8. How does the cost of this task compare with the rework and the risk of getting it wrong?

10. A balanced view: what is known and what is not#

Fairly solid: AI services bill by input and output tokens, output is usually more expensive per token, earlier conversation history is sent again at later turns, and caching can lower the price of rereading (according to Anthropic's and GitHub's documentation on billing).

Unclear:

• Your team's real cost. The model in section 4 is simplified with assumed prices.
• How much your tool uses caching on its own. I did not verify each tool.
• The token gap between Vietnamese, Japanese and English on today's models. The study I cite is from 2023 and I had no counter to measure it myself.
• Gartner's 2028 forecast is a forecast, not a measurement.

Do not overstate. Cheapest per token is not necessarily cheapest per finished task. A weaker model may need more turns or leave mistakes, and rework costs more than tokens. Do not choose a model by price without measuring quality. The post Measuring an AI feature covers measuring quality, cost and trust together.

Do not dismiss it. Cost is a real axis of daily agent use, and usage-based pricing is spreading. Put cost into your team's one-page AI use policy, together with asking for evidence as in Make AI Output Prove Itself, so saving money does not lower quality.

Do not use it to compare colleagues. Token use depends on the kind of work and does not measure a person's productivity.

11. Conclusion#

The AI bill does not grow with the number of sentences you type, but with how many times the history is reread multiplied by how thick it is. Three things keep cost within reach: open a new session at the boundary between tasks, use the right size of model for each task, and set caps and check usage every week. Run the script once with your own numbers, then compare it with your real usage page.

References#

Figures come from public sources as of 30 October 2026. Prices and billing rules change, so open the originals before quoting.

• Anthropic, Prompt caching (cache write and read factors, minimum length, default lifetime, example price table; read through a page-summarizing tool)
• Gartner, Gartner Predicts AI Coding Costs Will Surpass Average Developer's Salary by 2028 as Token Consumption Surges (24 June 2026, the original page returned a 403 error when I opened it) and the reprint on DevOps Digest (the source I actually read)
• GitHub, GitHub Copilot is moving to usage-based billing (27 April 2026, read through a page-summarizing tool; secondary sources give different values for one credit, so I do not use it)
• Petrov, La Malfa, Torr, Bibi, Language Model Tokenizers Introduce Unfairness Between Languages (2023)
• Per-developer cost ranges from blogs of vendors that sell cost-management tools, as a rough reference only: CloudZero, Eden AI

For anyone who uses AI to write code or to work day to day, beginners included. No statistics background needed. Every term comes with an everyday picture, and the calculation parts are marked so you can skip them.

Numbers come from public sources listed at the end, and I opened the originals for the METR figures used here. Examples marked "illustration" are numbers I made up, not anyone's real measurements. Nobody fully understands how AI works inside, and this post says nothing about that: it only teaches how to measure outcomes from the outside. The two studies cited are other people's, on specific tools and dates, not conclusions about you.

Quick summary#

• Feeling "AI makes me faster" and actually being faster can differ. In METR's 2025 randomized trial, 16 experienced developers did 246 real tasks. Tasks with AI took about 19% longer, although the developers had predicted a 24% speedup and afterwards still believed in about 20%.
• METR's early-2026 follow-up (57 developers, over 800 tasks) leaned toward AI helping, but METR itself calls the data "very weak evidence" about the size of the gain, because many developers refused to work without AI.
• You can measure yourself in two weeks: about ten tasks of similar size, randomly assigned with or without AI, timed including rework, then compare medians.
• Three things most easily spoil the measurement: choosing which tasks get AI, forgetting rework time, and peeking at results and changing course.
• Ten tasks prove nothing. They give you a measured hint instead of just a feeling.
• The post includes a CSV template, a Python script that runs, a worked illustration with answers and a checklist. Do not use the result to judge colleagues.

Hard words, plain meaning#

After this post you can explain#

1. Why feeling faster can differ from measured speed, and why you need a stopwatch.
2. Why you must randomize and include rework time for the comparison to be roughly fair.
3. Why ten tasks are only a hint, and how to read the result without fooling yourself.

1. Picture it first: new running shoes#

Picture buying a new pair of running shoes. After the run your feet feel light and you think you were faster. The watch says you were two minutes slower than usual, because it was hotter today and the route was hillier. The feeling is real, the watch is real, but they measure different things.

To learn whether the shoes help, a careful runner does many runs, picks at random which ones get the new pair, times every run and compares. They also count not just "running time" but "how sore my feet were afterwards".

AI is the same. The feeling of "less typing, less thinking" is real. But the time from receiving a task to really being done and not sent back is a different quantity. This post teaches how to start that second stopwatch.

Takeaway: to find out whether something helps you, do not ask your feelings. Run a small, fair comparison.

2. What others have measured#

2.1 METR's 2025 trial#

METR is an independent research organization. It ran a randomized trial: 16 experienced open-source developers (about 5 years of prior experience with their own projects) did 246 real tasks such as bug fixes, features and refactors, on large codebases (on average over 22,000 stars and over a million lines of code). Each task was randomly assigned "AI allowed" or "AI not allowed". The AI tools were mainly Cursor Pro with Claude 3.5 and 3.7 Sonnet. The period was February to June 2025, each task averaged about two hours, and developers were paid 150 USD an hour.

Result: tasks with AI took 19% longer. Before the work, the developers predicted AI would make them 24% faster. After finishing, they still believed AI had made them about 20% faster. Economics and machine learning experts who were asked predicted roughly 39% and 38% speedups.

METR lists its own limits: only 16 people so it may not generalize, possible sampling bias (people who expected large gains may not have signed up), learning effects beyond about 50 hours of tool use cannot be ruled out, part of the time was self-reported, and it covers only work on familiar codebases. The METR page now also labels these results as out of date.

Takeaway: 19% does not mean "AI slows you down". It is one experiment on one group, tools and date. What is worth learning is the gap between feeling and measurement.

2.2 The early-2026 follow-up#

On 24 February 2026 METR posted "We are Changing our Developer Productivity Experiment Design". They reran with 57 developers (10 returning and 47 new), 143 repositories and over 800 tasks. According to the post, returning developers were about 18% faster (confidence interval from 38% faster to 9% slower) and new developers about 4% faster (from 15% faster to 9% slower). Both intervals include zero.

More important: METR says the data is "very weak evidence" about the size of the gain, because more and more developers declined to take part since they did not want to work without AI (this likely biases the speedup downward), the pay cut from 150 to 50 USD an hour may have added selection effects, and timing became unreliable when developers ran several agents at once. METR still considers it likely that developers are more sped up by AI than in early 2025, and is redesigning the experiment.

Takeaway: even people who measure for a living have to change their method when tools and habits change. Someone else's number does not replace yours.

3. Designing the two-week experiment#

Picture a small kitchen experiment: the same dish cooked two ways, tasted blind.

3.1 Eight steps#

1. Pick about ten tasks of similar size. For example each expected to take 30 to 90 minutes. Ten tasks fit two work weeks, roughly one a day.
2. Write your prediction first. Record "I think AI will make me this many percent faster". METR used this step and it is the one most often skipped.
3. Define "done" before you start. For example: tests pass, you re-read the code, you ran it. The same definition for both groups.
4. Randomize. Flip a coin or use the script below. About half the tasks get AI, half do not. Never switch groups after seeing the task.
5. Time from start to "done" as defined in step 3. Record real start and end times, not estimates.
6. Record rework too. If a task is sent back, or you realize something is missing, add the fixing time to that same task.
7. Do not peek at results midway. Peeking makes you want to change your method or choose tasks.
8. Analyze once you have ten tasks. Compare the medians of both groups, then compare with the prediction from step 2.

3.2 CSV template#

One task per row. minutes is the time until done by your definition, rework_minutes is the fixing time afterwards:

task_id,arm,minutes,rework_minutes,done_checks
T1,no_ai,70,5,pass
T2,ai,35,20,pass

Also keep elsewhere: a description of the task, your prediction, your impression afterwards (how many percent faster or slower) and odd notes. Write the impression before you open the table of numbers.

3.3 Script for randomizing and analysis (Python 3, nothing to install)#

The plan command randomizes tasks. The analyze command computes medians, the difference, an exact permutation test and a 95% bootstrap interval:

#!/usr/bin/env python3
"""Self-experiment helper: randomize which tasks use AI, then compare the times.

  python3 ai_experiment.py plan 10 [seed]    : prints a CSV with the arm of each task
  python3 ai_experiment.py analyze data.csv [--no-rework] : medians, difference, exact permutation test, bootstrap interval

Needs only the Python standard library.
"""
import csv, itertools, random, statistics, sys


def plan(n, seed=None):
    rng = random.Random(seed)
    arms = ["ai"] * (n // 2) + ["no_ai"] * (n - n // 2)
    rng.shuffle(arms)
    print("task_id,arm")
    for i, a in enumerate(arms, 1):
        print(f"T{i},{a}")


def load(path, rework=True):
    rows = list(csv.DictReader(open(path)))
    cost = lambda r: float(r["minutes"]) + (float(r["rework_minutes"]) if rework else 0)
    ai = [cost(r) for r in rows if r["arm"] == "ai"]
    no = [cost(r) for r in rows if r["arm"] == "no_ai"]
    return ai, no


def diff(a, b):
    return statistics.median(a) - statistics.median(b)


def analyze(path, rework=True, seed=1, boots=10000):
    ai, no = load(path, rework)
    obs = diff(ai, no)
    # exact permutation test: every way of choosing which tasks would have been "ai"
    pooled = ai + no
    n, k = len(pooled), len(ai)
    count = total = 0
    for idx in itertools.combinations(range(n), k):
        s = set(idx)
        a = [pooled[i] for i in s]
        b = [pooled[i] for i in range(n) if i not in s]
        total += 1
        if abs(diff(a, b)) >= abs(obs):
            count += 1
    # bootstrap interval for the difference of medians
    rng = random.Random(seed)
    ds = sorted(
        diff([rng.choice(ai) for _ in ai], [rng.choice(no) for _ in no]) for _ in range(boots)
    )
    lo, hi = ds[int(0.025 * boots)], ds[int(0.975 * boots) - 1]
    print(f"tasks with ai: {len(ai)}, without ai: {len(no)}")
    print(f"median minutes with ai:    {statistics.median(ai):.1f}")
    print(f"median minutes without ai: {statistics.median(no):.1f}")
    print(f"difference (ai - no_ai):   {obs:+.1f} min ({obs / statistics.median(no) * 100:+.0f}%)")
    print(f"exact permutation p-value: {count}/{total} = {count / total:.3f}")
    print(f"bootstrap 95% interval:    {lo:+.1f} to {hi:+.1f} min")


if __name__ == "__main__":
    cmd = sys.argv[1]
    if cmd == "plan":
        plan(int(sys.argv[2]), int(sys.argv[3]) if len(sys.argv) > 3 else None)
    elif cmd == "analyze":
        analyze(sys.argv[2], rework="--no-rework" not in sys.argv)

In a real run, do not pass a seed to plan, so that you cannot predict it either. Save the output the moment it runs and never edit it afterwards.

4. A worked example (illustration, skippable)#

This is an illustration with numbers I made up, not anyone's real measurements. I ran the planner with seed 7 (only so the example is repeatable) and then invented a time for each task:

$ python3 ai_experiment.py plan 10 7
task_id,arm
T1,no_ai
T2,ai
T3,ai
T4,ai
T5,no_ai
T6,ai
T7,no_ai
T8,no_ai
T9,ai
T10,no_ai

After two weeks, the table sample.csv (invented minutes) looks like this:

4.1 Running the analysis, two ways#

Real output of the script on the table above, counting rework:

$ python3 ai_experiment.py analyze sample.csv
tasks with ai: 5, without ai: 5
median minutes with ai:    55.0
median minutes without ai: 60.0
difference (ai - no_ai):   -5.0 min (-8%)
exact permutation p-value: 252/252 = 1.000
bootstrap 95% interval:    -45.0 to +25.0 min

And forgetting rework (the --no-rework flag):

$ python3 ai_experiment.py analyze sample.csv --no-rework
tasks with ai: 5, without ai: 5
median minutes with ai:    40.0
median minutes without ai: 60.0
difference (ai - no_ai):   -20.0 min (-33%)
exact permutation p-value: 48/252 = 0.190
bootstrap 95% interval:    -50.0 to +10.0 min

4.2 Answers and how to read them#

The honest reading: in this example, AI looks slightly ahead, but ten tasks cannot confirm it, and the initial feeling is bigger than the measurement. More important: if you forget to add rework time, you will believe the prettier 33%.

Note: these numbers are invented. Real work can give a completely different result, including the opposite direction.

5. Where the measurement breaks#

Picture timing a run but forgetting to start the watch again after the red light.

5.1 Illustration: one developer's week#

This is an invented scenario with no real person or product.

• Monday, 09:00. The developer runs plan. The first task lands in the AI group. They write the prediction: 30% faster.
• 09:05. Start the timer, ask the AI to write a data-cleaning function.
• 10:20. The AI says it is done, tests pass. The developer stops the clock: 75 minutes.
• 10:45. On re-reading, they find the empty-date case is missing. Fixing takes another 25 minutes. They enter it in the rework column rather than skipping it.
• Friday, 17:30. They promise themselves not to look at the totals until the end of next week, even though a few nice numbers are already visible.
• Next Friday, 16:00. All ten tasks are done. They run the analysis, compare it with the prediction and write the conclusion: AI is slightly ahead on tasks with a clear pattern, not ahead on tasks that need business understanding.

What to do: add rework time to the same task, even when it felt "almost done". This is an illustration, so it proves nothing. The evidence is in the research: the METR trial above is one example of feeling and measurement differing.

6. Numbers and documents used#

Reading it correctly: the sources show that feeling and measurement can diverge and that measuring AI productivity is hard. They do not say AI is slow or fast for you.

7. Checklist: eight questions before trusting your number#

1. Did I write my prediction before starting?
2. Was which tasks get AI decided by a coin or script, and did I save it?
3. Are the tasks of similar size?
4. Is "done" defined the same way for both groups?
5. Did I add all rework time to the right task?
6. Did I peek at results midway and change my method?
7. How wide is the interval, and am I concluding more than it allows?
8. Am I using this only to adjust my own habits, not to judge anyone?

8. A balanced view: what it measures, what it does not#

It measures: a hint about which tasks AI seems to help with and which it does not, and the gap between your feeling and the measurement.

It does not measure:

• Ten tasks prove nothing. The interval is wide and the difficulty of individual tasks can skew the result.
• Time is not everything. AI may help you learn, tire you less or make you willing to take harder work, none of which the stopwatch captures. And faster does not mean you understand the code you hand in. See Learning an Unfamiliar Library with AI Without Losing the Skill.
• Today's result may not hold for the tools three months from now.
• Do not use the result to evaluate colleagues. The sample is small, everyone does different work, and a single number is easy to misuse. Read Do Not Measure Productivity by Lines of Code before putting any metric in front of a team.
• If you want to read numbers about people entering the field with care, see Entry-Level Developers: The Numbers Contradict Each Other, So Which Ones Do You Believe?.

Do not overstate it: a personal experiment is not rigorous science. Do not dismiss it: it still beats trusting a feeling, and the cost is a time log for two weeks. If you want to check the quality of what AI hands you, read Make AI Output Prove Itself too.

9. Conclusion#

Feeling faster is a feeling. The time from receiving a task to really being done and not sent back is a measurement. The two can differ, as the METR trial showed for one specific group. Run a small experiment: ten tasks, randomized, timed with rework included, compare medians, and read the result with exactly the confidence it deserves.

References#

Numbers come from public sources as of October 2026. I opened the METR pages and the arXiv paper and checked the METR figures against them. Open the originals to check before quoting.

• METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (arXiv 2507.09089) and METR blog post, 10 July 2025
• METR, We are Changing our Developer Productivity Experiment Design (24 February 2026)
• Python Software Foundation, documentation of the statistics and itertools modules (for the median and combinations)

For beginners, for people who build products, and for anyone who asks AI to fix a slow web page without knowing where the problem sits. No networking background is needed. Every hard word comes with an everyday picture, and the calculation parts are marked so you can skip them.

Numbers and definitions come from public documentation listed at the end. Examples labeled "illustration" or "simulation" are reconstructed and name no real person or company. Real systems have more detail than this article, so the pictures below are only aids for remembering.

What I could not do, up front: the environment where I wrote this blocks opening local network ports and blocks outbound https. So I could not run curl against any server. The "real run" section is a simulation I wrote and really ran, but the delays are set by me, not measured on a network. The curl commands in the article are marked "not run here". Run them on your own machine to get real numbers.

Quick summary#

• When you open a page, the request passes many legs: address lookup, connection, security handshake, the request, the application, the database, the reply. Each leg can be slow or fail on its own.
• To find the slow leg, measure, do not guess. The curl command with the -w option prints timing marks.
• curl's marks are cumulative from the start. The time of one leg is a mark minus the mark just before it.
• In this article's simulation, two cases have similar totals (0.973 s and 1.383 s) but very different culprits: waiting for the database (about 83%) in one, DNS lookup (about 87%) in the other.
• A 200 code only says the request succeeded at the protocol level. It does not say the content is right.
• When you ask AI to fix a slow page, measure first and give it the numbers. AI-written code can hide many small queries that are hard to see, but I found no study measuring how often.

After this article you can explain#

1. Which legs a web request has, and why each can be slow or fail by itself.
2. Why curl's timing marks must be subtracted, and how to compute each leg.
3. How to narrow a problem down to the network, the application or the database before asking AI to fix it.

Hard words, plain meaning#

1. A picture first: sending a parcel#

Picture ordering goods from a shop in another city. Before the goods reach you, seven things happen:

1. You look up the phone book for the shop's address (DNS).
2. You call to say there is an order and wait for them to pick up (TCP connection).
3. Both sides agree on a locked box (TLS).
4. You send the order slip saying what you want (HTTP request).
5. The front desk receives the slip and handles it (the application on the server).
6. The front desk goes to the warehouse to find the goods (the database).
7. The goods and a slip come back to you (the reply).

If the goods arrive late, you cannot tell at once where the problem is: an old phone book, a busy line, the front desk, or the warehouse. The most trustworthy person is the one who times each step. Computers are the same.

MDN describes it more briefly with a road picture: your device is the house, the server is the shop, DNS is the address book, and HTTP is the language you order in. MDN lists four steps: DNS lookup, request, response, assembly. This article splits it finer so you can see which parts are measurable.

Takeaway: "the page is slow" is not one fault. It is the sum of seven legs, and usually one leg takes most of the time.

2. The seven legs, one by one#

The table is a simplified map. The last column says how each leg tends to fail, so you can recognize it from an error message.

2.1 Leg 1: DNS, the phone book#

Picture it as: asking directory enquiries for the shop's number. You know the name, not the number.

Computers talk to each other with numeric addresses. The browser asks a DNS server to turn "example.com" into a numeric address. The result is usually remembered for a while (cached), so the next time is faster. That is why the first measurement is often slower than later ones.

2.2 Legs 2 and 3: connection and security handshake#

Picture it as: phoning ahead, then both sides agreeing on a locked box.

According to MDN, HTTP runs on TCP and is usually encrypted with TLS. HTTP needs a reliable channel, so it opens a connection first. For an https address there is an extra TLS handshake where both sides agree on a key. The old HTTP/1.0 opened a new connection for each request and reply, while later versions reuse the connection, so a second request to the same server often skips legs 1 to 3.

2.3 Leg 4: sending the HTTP request#

Picture it as: the order slip. It names the item (the path), the way of ordering (the method) and a few notes (headers).

MDN describes a request as a method (such as GET to fetch, POST to send), the path of the resource, the HTTP version, optional headers and possibly a body. An example following MDN's pattern:

GET / HTTP/1.1
Host: developer.mozilla.org
Accept-Language: fr

A reply has the version, a status code with a few words, headers and possibly a body (the page content):

HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 29769

The five status code classes according to MDN: 1xx informational, 2xx success, 3xx redirection, 4xx a fault on the side that sent the request, 5xx a fault on the server side. Some common codes: 200 success, 301 moved permanently to another address, 304 not changed so use the saved copy, 404 not found, 500 a general server error, 503 temporarily unable to serve.

2.4 Legs 5 and 6: the application and the database#

Picture it as: the front desk reads the slip, then goes to the warehouse. Every trip to the warehouse takes time there and back.

For a dynamic page, time often piles up in these two legs. The application runs your code, and if it needs data it asks the database. Each ask is a round trip, especially when the database sits on another machine. Asking 100 small times is usually slower than asking one big time, because you pay for 100 trips. Section 4.1 has a worked example.

2.5 Leg 7: the reply comes back#

The bigger the content, the longer it takes, especially if it is not compressed. For a text page this is usually the shortest leg, but it can be long for large images or downloads.

Takeaway: every leg has its own "failure signature". The error message is a clue worth reading carefully before asking AI to guess.

3. Timing each leg with curl#

Picture it as: putting a stopwatch at every door along the parcel's route.

curl is a command-line tool for sending HTTP requests. Two options matter here:

• -v (verbose): prints the whole process. According to curl's documentation, a line starting with * is information from curl itself (name lookup, connection), a line starting with > is what curl sent, and a line starting with < is what curl received.
• -w (write-out): prints measurements after the transfer, in a template you set.

Five marks to know. All are in seconds and all count from the start, according to the libcurl documentation:

A sample command (not run here, because my environment blocks outbound https; run it on your machine):

curl -s -o /dev/null -w 'dns=%{time_namelookup} tcp=%{time_connect} tls=%{time_appconnect} first_byte=%{time_starttransfer} total=%{time_total} code=%{http_code}\n' https://example.com/

I did run a command of this form with curl 8.7.1 and curl accepted the variable names, but the connection was blocked, so the status code came out as 000 and the numbers are meaningless. I do not use them.

3.1 The subtraction formulas#

Because the marks are cumulative, the time of each leg is:

DNS             = time_namelookup
TCP connection  = time_connect       - time_namelookup
TLS handshake   = time_appconnect    - time_connect
Server wait     = time_starttransfer - time_appconnect
Download        = time_total         - time_starttransfer

For an http address (no TLS), drop the TLS line and use time_connect in place of time_appconnect on the "Server wait" line (an approximation).

An important limit: the "server wait" interval contains both the application (leg 5) and the database (leg 6). curl cannot separate them. To separate them you need query logs or a dedicated monitoring tool.

4. A real run: simulating the seven legs (the calculation can be skipped)#

This is a simulation I wrote in Python 3.14.8 and really ran in my environment. Each leg is a wait I set in advance. The program prints cumulative marks like curl does, then computes each leg by subtraction. The delays are numbers I chose, not measured from a network. What is being tested is whether the subtraction returns what I set.

Case A: a slow database (DNS 0.040 s, connect 0.030, TLS 0.060, server wait 0.800, download 0.020). Case B: slow DNS (DNS 1.200, connect 0.030, TLS 0.060, server wait 0.050, download 0.020).

The real output of the run:

A: slow database
  time_namelookup      0.045   leg =  0.045
  time_connect         0.080   leg =  0.035
  time_appconnect      0.144   leg =  0.064
  time_starttransfer   0.949   leg =  0.805
  time_total           0.973   leg =  0.024
  longest leg: time_starttransfer (0.805 s)
B: slow DNS, fast database
  time_namelookup      1.205   leg =  1.205
  time_connect         1.240   leg =  0.035
  time_appconnect      1.305   leg =  0.065
  time_starttransfer   1.360   leg =  0.055
  time_total           1.383   leg =  0.023
  longest leg: time_namelookup (1.205 s)

Reading the result:

• Case A: the server-wait leg is 0.805 s out of 0.973, about 83%. The 0.805 is a little above the 0.800 I set because a real wait always overshoots by a few milliseconds.
• Case B: the DNS leg is 1.205 s out of 1.383, about 87%. The totals of the two cases are close (under 1.4 s), but what needs fixing is completely different.
• If you only look at time_total, you cannot tell the two cases apart. Fixing the database in case B would not help.

To be clear: this is a simulation with delays I set. It proves the method of calculation, not what a real network looks like. The numbers on your machine will differ.

4.1 Extra queries: a real count, an assumed time#

The second example also ran for real, in Python with SQLite 3.53.4 in memory. There are 50 customers and 200 orders. Way one: ask for the list of customers, then one separate query per customer. Way two: one query with a join and a group.

same answer: True
queries, one per row: 51
queries, one join   : 1
waiting for the database at 5 ms per query: 255 ms vs 5 ms

Both ways give the same answer, but one costs 51 queries and the other costs 1. The counts are real. The 5 milliseconds per round trip is my assumption for a database on another machine (SQLite in memory has almost no network delay). Under that assumption, 51 queries wait 255 ms and 1 query waits 5 ms. This pattern is called N+1 (one query for the list, then N queries, one per row).

Takeaway: when the "server wait" leg is long, the first question is not "which query is slow" but "how many queries does this page send".

5. Exercises with answers#

These exercises use a fake server on your own machine. I could not run this server because my environment blocks opening network ports (the command that opens the port reports "Operation not permitted"). I could only check that the code compiles. The answers below follow from how I wrote the code, so run it for real and compare.

5.1 The fake server#

Save it as slow_app.py:

import time
from http.server import BaseHTTPRequestHandler, HTTPServer

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/slow":
            time.sleep(0.8)          # simulate a slow query
            body = b"orders: 3\n"
        elif self.path == "/missing":
            self.send_response(404)
            self.send_header("Content-Length", "0")
            self.end_headers()
            return
        else:
            body = b"hello\n"
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, *a):
        pass

HTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

Run python3 slow_app.py in one terminal window and use another window for the commands below.

5.2 Exercise 1: find the slow leg#

curl -s -o /dev/null -w 'tcp=%{time_connect} first_byte=%{time_starttransfer} total=%{time_total} code=%{http_code}\n' http://127.0.0.1:8765/slow
curl -s -o /dev/null -w 'tcp=%{time_connect} first_byte=%{time_starttransfer} total=%{time_total} code=%{http_code}\n' http://127.0.0.1:8765/

Question: in each command, which leg takes almost all the time? Work it out by subtraction.

Answer (derived from the code, compare with your numbers): the first command has first_byte minus tcp of about 0.8 seconds (plus a few milliseconds of error), because the code waits 0.8 s before replying. The rest is close to zero since both ends are the same machine. The second command has no wait, so all the numbers are tiny. Because the address is http, with no TLS, you use time_connect in place of time_appconnect.

5.3 Exercise 2: a 404 and the timing#

curl -s -o /dev/null -w 'code=%{http_code} total=%{time_total}\n' http://127.0.0.1:8765/missing

Answer: code=404 and a very small time. The lesson: an error reply can still be fast. A short time does not mean correct.

5.4 Exercise 3: one measurement is not enough#

Run the exercise 1 command (the /slow path) five times and write down the first_byte values. Take the middle value when sorted from smallest to largest (the median).

Answer: the numbers should be near 0.8 seconds but differ by a few milliseconds. On a real network the spread is much larger, so do not conclude from a single run.

5.5 Try it with https (not run here)#

curl -v https://example.com/ -o /dev/null
curl -s -o /dev/null -w 'dns=%{time_namelookup} tcp=%{time_connect} tls=%{time_appconnect} first_byte=%{time_starttransfer} total=%{time_total}\n' https://example.com/

In the -v command, look for the * lines (connection and handshake), the > lines (the order slip curl sends) and the < lines (the reply slip). With the -w command, apply the formulas in section 3.1. The first run often has slower DNS, and the second is often faster thanks to caching.

6. Where people get mixed up#

Picture checking a parcel: knowing it arrived late is not enough, you must know at which door.

7. Illustration: one morning with a slow report page#

This is an illustration, naming no real person or product. The names, counts and times are reconstructed.

• 09:30. A user reports the report page takes about 3 seconds to appear.
• 09:40. The dev asks an agent: "Make the query faster." The agent adds an index and says "optimized".
• 10:05. The page is still slow. The dev runs curl with -w and sees tcp=0.02, first_byte=3.05, total=3.08. Server wait takes nearly 3 seconds. DNS, connection and download are negligible.
• 10:15. The dev turns on the query log: one page load sends 601 queries (1 query for the list and 600 queries, one per row). Each query waits about 5 milliseconds because the database is on another machine. The arithmetic: 601 times 5 milliseconds is 3,005 milliseconds, which matches about 3 seconds.
• 10:50. The dev asks the agent to merge them into one query with a join, and to print the query count before and after. first_byte drops to about 0.1 seconds (an assumed number in this example).

The action to take: measure first, then give the numbers to the AI. If you only say "make it faster", the AI can fix the wrong place, such as adding an index for a query that was never the culprit, and still report done. Ask for evidence as in the post Make AI Output Prove Itself: the query count and the first_byte before and after.

This illustration proves nothing about frequency. The evidence is elsewhere: section 4.1 really counted 51 queries versus 1 for the same answer. As for whether AI-written code often causes N+1, I found no study that measures it directly. A 2025 study (Unveiling Inefficiencies in LLM-Generated Code) analyzed 492 small code snippets produced by three open-source models on HumanEval+ and found logic and performance problems among the common inefficiencies, but those are small algorithm tasks, not web applications with a database. Treat N+1 as a possible risk and measure for yourself, not as a certainty. For the security of AI-written code, see also OWASP Top 10 for AI Apps.

8. Numbers and documents used#

Read it correctly: the documents support the way of splitting into legs and the way of computing. I did not measure on a real network and found no study measuring N+1 in AI-written code.

9. Checklist: eight questions when a page is slow#

1. Did I measure with numbers, or am I guessing?
2. Did I take the marks and subtract them, or am I using time_total?
3. Which leg is biggest: DNS, connection, TLS, server wait or download?
4. Was this the first run or a later one (with caches)? Did I measure several times?
5. What is the status code, and is the returned content right?
6. If the "server wait" leg is long, how many queries does this page send?
7. When asking AI to fix it, did I give it the measurements and ask for measurements after the fix?
8. Where did I measure from, and are the users somewhere similar?

10. A balanced view: what this map simplifies#

What it does: gives you a way to split the problem, a set of measuring formulas and a habit of checking before asking AI.

What it does not do:

• This is a simplified map of HTTP/1.1 over TCP with TLS. In practice there are also CDNs (content delivery networks), proxies and load balancers. Each adds a leg or removes one.
• HTTP/3, per RFC 9114, runs on QUIC, a protocol built on UDP, and merges the connection part with the security handshake. According to the secondary sources I read, this reduces the round trips when opening a connection. In that case time_connect and time_appconnect may no longer separate as cleanly as the table in section 3. Read the RFC itself if you need precision.
• curl measures from one place, one time. A single measurement is noise. Dedicated monitoring tools (APM) see more, such as separating the application from the database.
• The simulation in section 4 only illustrates the arithmetic, and I did not run the exercises in section 5 for real.

Do not overstate it: knowing seven legs does not make you an infrastructure expert, but it is enough to ask the right question. Do not wave it off: fixing the wrong place still costs time, and someone will still report "done".

11. Conclusion#

"The page is slow" is the sum of several legs, and usually one leg takes nearly all of it. Measure with curl -w, subtract each mark from the next and look at the biggest leg. If it is the server wait, count the queries before guessing. And when you ask AI to fix it, give it the measurements, ask for measurements after the fix, then open those numbers yourself.

References#

Definitions come from public documentation as of October 2026. The HTTP/3 part rests on secondary sources, so open the originals to check before quoting.

• MDN, How the web works
• MDN, HTTP overview and HTTP response status codes
• curl, CURLINFO_STARTTRANSFER_TIME and CURLINFO_APPCONNECT_TIME
• Everything curl, Verbose and the curl man page (I could not read the -w part of this page, the variable names were confirmed by running curl 8.7.1)
• IETF, RFC 9114 (HTTP/3) and RFC 9000 (QUIC)
• Unveiling Inefficiencies in LLM-Generated Code: Toward a Comprehensive Taxonomy, arXiv 2503.06327 (2025)

For anyone who writes or reads code, including beginners. No advanced Git or prior debugging knowledge needed. Every term comes with an everyday picture, and the calculation parts are marked so you can skip them.

Every command and output in this post comes from a real run on a practice repo (Git 2.56.0, Python 3.14.8), and there is a script so you can recreate it. Commit hashes will differ on your machine because they depend on time. There is no single agreed theory of how skilled people debug, so the five steps below are a synthesis of practice, not a procedure tested in controlled studies. Examples labeled "illustration" are reconstructed and name no real person or company.

Quick summary#

• Debugging is its own skill, and it does not go away because AI exists. If you ask AI before you know where the bug is, you will not have grounds to judge its answer.
• Five steps: reproduce the bug reliably, shrink the failing case, form one hypothesis that could be wrong, halve the suspect range, then fix and add a test so the bug cannot return.
• The halving step has a built-in Git tool, git bisect. In this post's real example, 19 suspect commits needed 4 checks to find the culprit, versus up to 19 if you walk through them one by one.
• The culprit in the example is a commit named "refactor: simplify discount check", a change that looks harmless. The bug is one character: >= became >.
• After the five steps you hand AI a short bug report (template at the end). Then AI helps faster and you can check its explanation.
• Limits: this needs a test that gives the same answer every time. Bugs that depend on timing, data or environment are much harder.

Hard words, plain meaning#

After this post you can explain#

1. Why a bug needs to be reproduced and shrunk before you fix it, and what a hypothesis that "can be wrong" looks like.
2. How git bisect finds the bad commit by halving, and why 19 commits need only about 5 checks.
3. What a good bug report for AI contains, and why to send it after you have narrowed the range yourself.

1. A picture first: the detective at the scene#

Picture a detective arriving at a scene. They do not rush to arrest the first suspicious person. They leave the scene as it is, write down what happened, drop details that do not matter so the core shows, then form a specific hypothesis: "if the culprit came through the back door there will be mud on the kitchen floor". They go and check exactly that. If it is wrong, they drop the hypothesis and try another. At the end they have evidence others can check.

Debugging is the same. Each of the five steps matches something the detective does:

Takeaway: a good detective is not the one who guesses right first, but the one with a way to rule out wrong possibilities one by one.

2. Why still learn this when AI exists#

2.1 A survey suggests the skill is still valued#

BairesDev's Dev Barometer Q2 2026 (1,569 developers in 77 countries, 1,059 juniors and 510 seniors, published June 2026) reports that 48% of junior developers rank problem-solving and analytical thinking as the most important skill for getting hired, almost three times the 18% who name proficiency with AI tools. The same survey found that only 16% of senior developers say juniors fully understand the AI-generated code they submit.

Three cautions. First, BairesDev is a software outsourcing company with a commercial interest in the topic. Second, the original page refused access when I opened it, so I read the numbers through search results quoting their press release and data deck, not from the original tables. Third, the 16% is seniors' opinion of juniors, not a measurement of real understanding. Read it as a signal, not hard evidence.

2.2 Why asking AI first can cost you your judgment#

Say you paste the whole failing file to AI and get a fix back. If you do not know where the bug is, you have two choices: trust it or not. A Stanford study (Perry et al., 2022, 47 participants) found that the group using an AI assistant wrote less secure code on four of five tasks but was more confident that its code was secure. It used older models and a small group, but it hints at a risk worth noting: feeling sure does not go with being right.

Takeaway: using AI is fine. But narrow the suspect range yourself first, so the AI's answer has something to be checked against.

3. Five steps, with a real example#

The example: a small shopping cart program. Rule: orders of 100 or more get 10% off. The test reports the failure: an order totaling exactly 100 gets no discount. The repo has 20 commits and exactly one breaks the rule. The script that creates it is in section 5.

Step 1. Reproduce: one command, the same result#

Picture: keep the scene untouched.

The first job is not fixing but having one command that makes the bug happen every time:

python3 test_cart.py
echo "exit=$?"

FAIL total([(50, 2)]) = 100.0, want 90.0
exit=1

These two lines are the scene: an order of 2 items at 50 (total 100) should come out as 90.0 but the program gives 100.0. The exit code is 1, meaning "fail". If two runs give two different results, you have not reproduced it yet, and that comes first (see the limits section).

Step 2. Shrink: remove what does not matter#

Picture: send people out of the room until only those involved remain.

Remove parts of the failing case one at a time and ask: is the bug still there? This is the result of a real run, from a big cart down to the smallest case:

The number of lines, items and prices do not matter. What remains is: the total before discount is exactly 100. The minimal failing case is now one line: total([(100, 1)]).

Step 3. Form one hypothesis that can be wrong, and predict the result first#

Picture: "if the culprit came through the back door there will be mud on the kitchen floor".

From step 2, the hypothesis: the discount function handles the boundary at exactly 100 wrongly. A good hypothesis lets you predict. If it is right: below 100 no discount, exactly 100 should get a discount but does not, above 100 gets a discount. Only then run it:

discount(99.99)   # 0.0
discount(100)     # 0.0 (should have been 10.0)
discount(100.01)  # 10.0

As predicted: the discount starts after 100, not at 100. The hypothesis holds, and you now know you are looking for a "greater than" that should be "greater than or equal". A hypothesis that fits every outcome is useless. If the result here had been discount(100) = 10.0, you would have to drop this hypothesis.

You may already see the fix. But in a real project that function may have been touched by many people, and the question "which change broke it" matters as much as "where is it broken". That is step 4.

Step 4. Halve: remove half of the suspects each time#

Picture: guess a number from 1 to 100. You do not ask "is it 1, is it 2", you ask "is it above 50?". Each question removes half.

The history has 20 commits. The first works and the latest is bad. That leaves 19 suspect commits. git bisect picks the one in the middle for you to test, then removes half. This is the real session:

git bisect start HEAD $(git rev-list --max-parents=0 HEAD)

status: waiting for both 'good' and 'bad' commits
status: waiting for 'good' commit(s), 'bad' commit known
Bisecting: 9 revisions left to test after this (roughly 3 steps)
[5f01574] feat: add free shipping limit

Git picked the middle commit and moved you there. You run the test and mark git bisect good or git bisect bad. With git bisect run, Git runs the test at every step and decides from the exit code (0 is good, 1 to 127 except 125 is bad, per the Git docs):

git bisect run python3 test_cart.py

running 'python3' 'test_cart.py'
Bisecting: 4 revisions left to test after this (roughly 2 steps)
[f1da7cd] feat: add min order note
running 'python3' 'test_cart.py'
FAIL total([(50, 2)]) = 100.0, want 90.0
Bisecting: 2 revisions left to test after this (roughly 1 step)
[2004cf0] feat: add coupon placeholder
running 'python3' 'test_cart.py'
FAIL total([(50, 2)]) = 100.0, want 90.0
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[6b693ef] refactor: simplify discount check
running 'python3' 'test_cart.py'
FAIL total([(50, 2)]) = 100.0, want 90.0
6b693ef is the first 'bad' commit
commit 6b693ef

    refactor: simplify discount check

 cart.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
bisect found first 'bad' commit

This session made 4 checks: the middle commit (pass), then three commits that all fail, until one commit was left. The culprit is refactor: simplify discount check. Look at its change:

git diff 6b693ef~1 6b693ef

-    if total >= 100:
+    if total > 100:

One character. The commit name says "tidy up", but it changed behavior. This is where a commit's name and its real content can disagree, and why you should find the commit with a test rather than by reading names.

Do not forget to end the session with git bisect reset so Git takes you back to your original branch.

Halving works outside Git too#

The "remove half" idea works everywhere: temporarily switch off half the lines of a long function and see if the bug remains, print a value in the middle of the flow to learn whether things break before or after that point, or turn off half of the feature flags. Each halving gets you much closer than staring at every line.

Step 5. Fix, recheck, and stop it coming back#

Picture: file a report so the case does not reopen.

Fix one character (> back to >=), then run the test:

python3 test_cart.py
echo "exit=$?"

exit=0

Pass. You also check the neighboring boundaries, and the real results are:

total([(99.99, 1)])   # 99.99  (no discount)
total([(100, 1)])     # 90.0   (10% off)
total([(100.01, 1)])  # 90.01  (10% off)

The test already existed in the repo (it checks an order of exactly 100), so this bug will be caught if someone breaks it again. If it had not, adding the test is this step: every bug should leave a test behind for itself. Better ways to write tests are in the post Testing AI-written code.

Takeaway: the five steps take you from "the program is wrong" to "one character in one specific commit", with evidence at every step. The rest is just typing the fix.

4. A calculation to check yourself: how fast halving is (optional)#

With N suspect commits, halving needs at most about log2(N) checks, rounded up. This is a real calculation:

import math
for n in (19, 99, 999, 9999):
    print(n, math.ceil(math.log2(n)), round(math.log2(n), 2))

In the real run Git needed 4, not 5. The 5 is a maximum, the real number depends on where the bug is. If you walked from the oldest commit one by one, you would need 10 checks to reach commit 11 (the culprit is the 11th commit), and up to 19. Note that these numbers assume each check gives a trustworthy answer and every commit runs. If some commits cannot be built, Git lets you skip them with code 125, but you will need extra checks.

5. Exercise: do it yourself, with an answer key#

Save this script as make_repo.sh and run bash make_repo.sh. It creates a cart-demo folder with 20 commits, exactly one of which breaks the discount rule.

#!/usr/bin/env bash
# Create a practice repo with 20 commits, exactly one of which breaks the discount rule.
set -e
rm -rf cart-demo
mkdir cart-demo
cd cart-demo
git init -q -b main
git config user.name "debug-demo"
git config user.email "demo@example.com"

c() { cat >> "$2"; git add -A; git commit -q -m "$1"; }   # c "message" file < content

cat > test_cart.py <<'EOF'
import sys
from cart import total

checks = [
    ([(10, 3)], 30.0),     # under 100: no discount
    ([(50, 2)], 90.0),     # exactly 100: 10% off
    ([(60, 2)], 108.0),    # over 100: 10% off
]
bad = [(i, want, total(i)) for i, want in checks if total(i) != want]
for i, want, got in bad:
    print(f"FAIL total({i}) = {got}, want {want}")
sys.exit(1 if bad else 0)
EOF
c "feat: add subtotal, discount, total and a check" cart.py <<'EOF'
def subtotal(items):
    return sum(price * qty for price, qty in items)

def discount(total):
    if total >= 100:
        return round(total * 0.10, 2)
    return 0.0

def total(items):
    s = subtotal(items)
    return round(s - discount(s), 2)
EOF
c "feat: add shipping fee" cart.py <<'EOF'

SHIPPING = 5.0
EOF
c "docs: explain discount rule" cart.py <<'EOF'

# 10% off for orders of 100 or more.
EOF
c "feat: add currency format" cart.py <<'EOF'

def money(x):
    return f"{x:,.2f}"
EOF
c "feat: add empty cart helper" cart.py <<'EOF'

def is_empty(items):
    return len(items) == 0
EOF
c "feat: add item count" cart.py <<'EOF'

def count(items):
    return sum(qty for _, qty in items)
EOF
c "feat: add max quantity" cart.py <<'EOF'

MAX_QTY = 99
EOF
c "feat: add tax rate constant" cart.py <<'EOF'

TAX_RATE = 0.08
EOF
c "feat: add receipt title" cart.py <<'EOF'

def title(name):
    return f"Receipt for {name}"
EOF
c "feat: add free shipping limit" cart.py <<'EOF'

FREE_SHIPPING_FROM = 200
EOF
perl -pi -e 's/total >= 100/total > 100/' cart.py
git add -A; git commit -q -m "refactor: simplify discount check"
c "feat: add coupon placeholder" cart.py <<'EOF'

COUPONS = {}
EOF
c "feat: add rounding helper" cart.py <<'EOF'

def r2(x):
    return round(x, 2)
EOF
c "feat: add item label" cart.py <<'EOF'

def label(price, qty):
    return f"{qty} x {price}"
EOF
c "feat: add min order note" cart.py <<'EOF'

MIN_ORDER = 1
EOF
c "feat: add currency code" cart.py <<'EOF'

CURRENCY = "USD"
EOF
c "feat: add store name" cart.py <<'EOF'

STORE = "Demo Shop"
EOF
c "feat: add version string" cart.py <<'EOF'

VERSION = "1.0"
EOF
c "feat: add free item note" cart.py <<'EOF'

FREE_ITEM_NOTE = "none"
EOF
c "chore: add end marker" cart.py <<'EOF'

# end of file
EOF

Try it before reading the answer:

1. Go into cart-demo, run python3 test_cart.py and write down the result (step 1).
2. Shrink the failing case by hand, trying a few other values (step 2).
3. Write your hypothesis as one sentence, with the results you predict for 99.99, 100 and 100.01 (step 3).
4. Run git bisect by hand (mark each commit) once, then again with git bisect run python3 test_cart.py. Count the checks (step 4).
5. Fix it and rerun the test (step 5).

Answer key:

• The bad commit is named refactor: simplify discount check, the 11th of the 20 commits.
• Its change: >= became > in the discount function of cart.py.
• Number of checks: at most 5, and 4 in my run. Yours may be 4 or 5, with different hashes, and that is normal.
• One trap: the test must exist in every commit you check. Here test_cart.py exists from the first commit, so it is safe. If the test were only added in a middle commit, older commits would report "bad" because the file is missing, and Git would conclude wrongly. To avoid it, keep the test script outside the repo (the Git docs also advise keeping the test script out of the repo) or pick a good commit after the test appeared.

6. A short bug report for AI: do it after the five steps#

After the five steps you have what you need to write a bug report that AI can use and you can check. A template to paste:

Bug: <one sentence>
Reproduction (smallest): <command or data>
Expected: <correct result>
Actual: <verbatim output>
Ruled out: <hypotheses tried and the results>
Suspect: <current hypothesis, with evidence>
Scope: <files, suspect commit>
Request: explain the cause before fixing,
propose the smallest change, and say what you did not check.

For this post's example the filled-in report looks like this:

Bug: an order totaling exactly 100 does not get the 10% discount.
Reproduction (smallest): total([(100, 1)])
Expected: 90.0
Actual: 100.0
Ruled out: number of lines and the price of each item do not matter.
discount(99.99) = 0.0, discount(100) = 0.0, discount(100.01) = 10.0.
Suspect: the comparison at the boundary 100 in discount().
Scope: git bisect points to the commit "refactor: simplify discount check".
Request: explain the cause before fixing and propose the smallest change.

With this report the right answer is almost too obvious: AI will point at the comparison at the boundary. More important: you can check the answer with the same test and boundary values you already have. If AI proposes a fix somewhere else, you know right away it is off. For making AI hand in evidence for its fix, see the post Make AI Output Prove Itself.

7. Where beginners trip#

7.1 Illustration: one morning, two ways#

This is an illustrative scenario and names no real person or product.

Way one: ask AI right away.

• 09:30. A user reports: "My order of 100 did not get the discount." The dev pastes the whole cart.py to AI and asks how to fix it.
• 09:40. AI proposes adding a special condition for a total equal to 100 inside the total function. The test passes, the dev merges.
• 11:15. The discount function still uses >, so another place that calls discount directly (the end-of-day report) still gets it wrong. The dev loses time hunting again.

Way two: five steps first, then ask.

• 09:30. Same report. The dev runs the test and sees the bug reproduces with one command.
• 09:40. Shrinks it to total([(100, 1)]), predicts and tries three boundary values, and the hypothesis holds.
• 09:50. git bisect run points to the commit "refactor: simplify discount check", a one-character change.
• 10:00. The dev gives AI a short bug report to make sure no other place uses the same condition, fixes it at the root and merges.

What to do: build the habit of shrinking and hypothesizing before asking AI, so you can check what it says. This is an illustration, so it proves nothing. The evidence is in the studies cited: Perry et al. (2022) show how easy it is to feel overconfident with an AI assistant, and the BairesDev survey suggests problem-solving is still valued. See also the post Git for People Who Use Agents on leaving save points that give bisect something to work with.

8. Eight questions before you hand a bug to AI#

1. Do I have one command that makes the bug happen every time?
2. Is the case as small as it gets: if I remove anything more, does the bug disappear?
3. What is my hypothesis, and did I predict the result first?
4. Did the result match the prediction, or am I bending it to fit?
5. Is this a regression (it used to work), and do I have a good commit as a starting point?
6. What have I ruled out, and did I write it down?
7. Does my bug report have the expected result, the actual result verbatim and what was ruled out?
8. When AI answers, what will I use to check it: a test, boundary values, or just a feeling?

9. A balanced view: what this does and does not do#

What it does: turns "the program is wrong" into a specific place with evidence, narrows the suspect range very fast through halving, makes bug reports short and checkable, and leaves a test that stops the bug returning.

What it does not do:

• git bisect needs a test that gives the same answer every time and a history with commits that run. Bugs that depend on timing, real data, run order or environment are much harder to reproduce.
• If the bug has existed since the first commit, there is no good commit to start from. If the history was squashed into a few large commits, halving only points you at a big chunk.
• Some bugs need careful code reading or domain knowledge, and are not regressions.
• AI sometimes finds the cause faster than a person, even when you have not done all the steps. The post does not say do not ask AI, it says ask afterwards so you can check.
• I did not find a controlled study measuring how much faster these five steps find bugs. They are a synthesis of practice, plus one real example on a small repo I built.
• The real example is one bug in a small 20-commit repo. Real repos are far more complicated.

Do not overstate it: the five steps do not guarantee finding every bug. Do not play it down: the cost of fixing without understanding usually shows up later, when the bug returns somewhere else.

10. Conclusion#

Debugging works like a detective's job: keep the scene, shrink it, form a hypothesis that can be wrong, rule things out by halving, then file a report so the bug does not come back. Once you have done those steps, the bug report for AI becomes short, and you have something to check its answer against. Try the exercise with the 20-commit repo: it takes about half an hour and leaves you with a habit that lasts a career.

References#

Figures come from public sources as of October 2026. Every command and output in the post comes from a real run on Git 2.56.0 and Python 3.14.8, and commit hashes differ between runs.

• Git, git-bisect documentation (usage, git bisect run, exit codes 0, 1 to 127 except 125, and 125 to skip)
• BairesDev, Dev Barometer Q2 2026 and the press release "16 percent of juniors fully understand AI code" (a survey by an outsourcing company; I read the figures through search results because the original page blocked access, so open the original before quoting)
• Perry et al., Do Users Write More Insecure Code with AI Assistants? (Stanford, 2022)

For anyone who writes SQL queries or asks an AI to write them, including beginners. You do not need to know how a database works inside. Every term comes with an everyday picture, and the script and code are marked optional if you only want the results.

Every number in this article comes from an experiment I really ran on SQLite 3.53.4 (the version bundled with Python 3.14.8, macOS, arm64 chip), and the output is pasted as it came. The experiment is small, on one machine, with randomly generated data, so treat it as a way to learn, not as a performance benchmark. How a database really chooses a plan is complicated, and the pictures below are memory aids, not the whole story.

Quick summary#

• With no index, a database often has to read every row of the table to find a result. In the experiment with 100,000 books, finding one author took about 2.2 ms.
• After adding an index on the author column, the same query took about 0.03 ms, roughly 79 times faster (other runs gave 75 to 82 times).
• An index does not always help. For a column with only two values (in stock or not), the query used the index and still took 43.6 ms against 39.9 ms for a full scan.
• With a multi-column index, column order decides which queries can use it. An index on (genre, year) did not help a query that filters only on year, while an index on (year, genre) did.
• An index has a price: inserting 20,000 rows was about 2.2 times slower with one index and about 5 times slower with three, and the file grew by about 2.5 MB for the first index.
• The cheap check is to read the query plan (SCAN means a full scan, SEARCH means only part of the rows). It is also the evidence worth asking for when an AI writes a query for you.

Hard words, plain meaning#

After this article you can explain#

1. Why queries slow down as a table grows: without an index the time grows with the number of rows, with an index it grows very slowly.
2. A good index needs three things: the right leading column, a condition that rules out many rows, and a query that does not "hide" the column behind a function or a leading %.
3. Every index has a price: it slows writes and takes space, so add one when you can measure the benefit and drop it when nobody uses it.

1. A picture first: the library card catalogue#

Imagine a library with 100,000 books. You want every book by the author "Author 1234". One way is to walk past every shelf and open each book to see who wrote it. The other is to open the card catalogue, where the cards are already sorted by author, find "Author 1234" and read the cards there, each saying which shelf the book is on.

The second way is much faster, but it has conditions. The catalogue must have been made, and it must be kept correct every time the library receives new books. If you ask in a way the catalogue is not sorted for (for example "authors whose name ends in 1234"), the librarian still has to walk every shelf.

A database index is that catalogue. The rest of this article measures how much faster it is, when it does not help and what it costs, with real numbers.

Takeaway: an index does not make the table faster, it gives the database a shortcut when the question matches the way the shortcut is sorted.

2. The real experiment: 100,000 books#

2.1 Setup#

The books table has 100,000 randomly generated rows with a fixed seed, so you can rebuild exactly the same data: title, author (2,000 different authors, so about 50 books each), year of publication from 1950 to 2024, genre (8 kinds) and an in-stock column (about 90% equal to 1). Each query ran 200 times and I took the median. The database ran in memory (:memory:), so the numbers do not reflect a real disk; see the limits further down.

The full script is in section 7 so you can rerun it. Below is the output of the run I used as the main numbers of this article.

2.2 Finding one author: full scan against an index#

SELECT * FROM books WHERE author = 'Author 1234'    (59 result rows)

No index:
    plan : SCAN books
    median: 2.204 ms

After CREATE INDEX idx_author ON books(author):
    plan : SEARCH books USING INDEX idx_author (author=?)
    median: 0.028 ms

The plan changed from SCAN (walk the whole table) to SEARCH books USING INDEX (use the index), and the time fell from 2.204 ms to 0.028 ms, about 79 times faster. I ran the whole script four times and the ratio stayed between 75 and 82 times. I measured only one table size (100,000 rows), so I cannot show how the time changes with a bigger table. In principle a scan must pass through every row, so it takes longer as the table grows, while the PostgreSQL docs describe an index lookup as walking a few levels of a search tree.

Takeaway: for a condition that rules out almost every row, an index gives a very large improvement.

2.3 A column with two values: the index is used and it is still not faster#

SELECT * FROM books WHERE in_stock = 1              (89,966 result rows)

No index:
    plan : SCAN books
    median: 39.896 ms

After CREATE INDEX idx_stock ON books(in_stock):
    plan : SEARCH books USING INDEX idx_stock (in_stock=?)
    median: 43.571 ms

Here the plan does use the index (SEARCH books USING INDEX) but the time did not fall, it even got a little slower (43.571 against 39.896 ms). The most reasonable explanation: the query still has to fetch almost 90% of the rows, so the main work is reading and passing on 89,966 rows, and an index lookup does not reduce that part. This is my explanation; I did not separate the cost of each step to prove it.

I also tried the other side: in_stock = 0 returns only 10,034 rows (about 10%). On a fresh database, the scan took 5.551 ms and the index 5.029 ms, only about 9% faster. Even when the query rules out 90% of the rows, the benefit is small compared with the 79 times for one author, because an author matches only 59 rows out of 100,000.

Takeaway: selectivity decides the value of an index. The question to ask is "how many rows does this condition rule out?", not "is this column often filtered on?". Seeing USING INDEX in a plan does not mean the query is fast.

2.4 A multi-column index: column order decides#

I tried two indexes on genre (genre) and year (year), with two queries.

Query A: WHERE genre = 'tech' AND year >= 2020    (827 rows)
Query B: WHERE year = 2015                        (1,321 rows)

No multi-column index:
    A: SCAN books                                         3.124 ms

Index (genre, year):
    A: SEARCH books USING INDEX idx_genre_year (genre=? AND year>?)   0.437 ms
    B: SCAN books                                         2.067 ms

Index (year, genre):
    A: SEARCH books USING INDEX idx_year_genre (year>?)     0.561 ms
    B: SEARCH books USING INDEX idx_year_genre (year=?)     0.694 ms

Reading it:

• Query A can use both indexes, but the (genre, year) index uses both conditions (genre=? AND year>?) while the (year, genre) index can use only the first condition, year>?. The times are 0.437 against 0.561 ms: different, but not by much.
• The big difference is in query B. The (genre, year) index cannot be used for a query that filters only on year (the plan is still SCAN, 2.067 ms), while the (year, genre) index can (0.694 ms).

Picture a catalogue sorted in two levels: first by genre, then by year inside each genre. Asking for "genre tech, year 2020 or later" goes straight to the tech drawer and reads on. Asking for "every book from 2015" finds 2015 scattered across every genre drawer, so the catalogue cannot help. The SQLite docs state this rule: the columns of an index are used from left to right, without gaps. Use The Index, Luke (Markus Winand) compares it with a phone book sorted by surname, where you cannot look someone up by first name alone.

One caution so you do not over-read this: for a query that uses equality on both columns (for example year = 2015 AND genre = 'tech'), I tried both orders and both used both columns, about 0.07 ms each against 1.6 ms for a scan. Column order matters most when there are queries filtering on a single column, or range conditions (>=, BETWEEN) on the leading column. The PostgreSQL docs also advise using multi-column indexes sparingly and say an index with more than three columns is rarely helpful unless table usage is very regular. PostgreSQL 18 can even "skip" the leading column in some cases, so the column order rule here is a general rule, not a law for every system and every version.

Takeaway: write down your real queries first, then choose the column order so that the leading column appears in as many of them as possible.

2.5 When an index exists and the query still scans#

I kept idx_author and tried four ways of asking for the same author:

WHERE lower(author) = 'author 1234'     plan: SCAN books                  5.539 ms
WHERE author LIKE '%1234'               plan: SCAN books                  3.798 ms
WHERE author LIKE 'Author 1234%'        plan: SCAN books                  2.852 ms
WHERE author GLOB 'Author 1234*'        plan: SEARCH books USING INDEX idx_author (author>? AND author<?)   0.029 ms

• Wrapping the column in a function (lower(author)): the index is sorted by author, not by lower(author), so it is ignored. Like a catalogue sorted by the original name while you ask using the lowercase version of the name.
• A % at the start (LIKE '%1234'): the catalogue is sorted by the first letters, so asking by the end of the name cannot use it.
• LIKE with a fixed beginning (LIKE 'Author 1234%'): in theory it can use an index, but here it is still SCAN. According to the SQLite docs, LIKE is case-insensitive by default, so it can use an index only when the column uses the NOCASE collation. My default index compares byte by byte (BINARY), so it cannot be used. When I also created an index on author COLLATE NOCASE, the plan changed to SEARCH.
• GLOB is case-sensitive, so it matches the default index and took 0.029 ms.

This detail is specific to SQLite. PostgreSQL has its own rules about LIKE and indexes (they depend on the collation and the index type), and I did not check them in this round. The general lesson still holds: do not guess, read the plan.

2.6 The price: slower writes and more space#

Insert 20,000 rows into a 100,000 row table (median of 7 runs, in memory):

0 indexes                                  6.4 ms   (x1.00)   file 4.98 MB
1 index (author)                          13.9 ms   (x2.18)   file 7.48 MB
3 indexes (author, genre+year, title)     31.8 ms   (x5.00)   file 12.23 MB

Each time a row is inserted, the database must update the table and every index. Three indexes made inserting about 5 times slower, and the file nearly 2.5 times bigger (from 4.98 MB to 12.23 MB). The PostgreSQL docs also say an index adds overhead to data modification and advise dropping indexes that are rarely or never used.

Note: the example ran in memory, with bulk inserts in one transaction, no concurrent access and no disk. A real system can differ quite a lot, in either direction. Use this number to see the direction (writes cost more with more indexes), not to forecast.

Takeaway: add indexes for the queries you really run often, and measure again. An index "just in case" on every column is a way of paying a lot for something that goes unused.

3. How to read a query plan#

Picture the librarian telling you in advance how they will search. In SQLite you type EXPLAIN QUERY PLAN before the query:

EXPLAIN QUERY PLAN SELECT * FROM books WHERE author = 'Author 1234';

According to the SQLite docs, there are two words to know:

Three easy mistakes:

1. SEARCH does not guarantee speed (the example in 2.3 used the index and still took about as long as a scan).
2. The brackets show which condition the index uses. In 2.4, (year>?) shows that only half of the conditions were used.
3. The SQLite docs state that this output is meant for human readers and may change between versions, so do not write code that depends on it.

Other systems have similar tools with different formats. PostgreSQL uses EXPLAIN (and EXPLAIN ANALYZE to run and measure), so do not carry the SQLite way of reading plans over to it unchanged.

4. Where indexes often go wrong#

Picture a library with a single librarian: the more catalogues, the busier they are whenever books arrive.

4.1 What about AI-written queries#

An AI assistant writing a query usually does not know whether your table will have 100 or 100 million rows unless you say so. The query it writes is usually correct in its results and fast on test data, so a reviewer easily lets it pass. I have not found independent measurements of how often AI-generated queries lack an index, so this is a warning from the mechanism, not from data. The cheapest remedy is to ask for evidence, as in way 2 of Make AI Output Prove Itself: ask the AI to paste the real query plan and the assumed number of rows.

5. Illustration: a query that was fine at first and slow months later#

This is an illustration. It names no real person or product, and the scaled-up numbers are multiplications I made up, not measurements.

• 10:00. A dev asks an agent: "Add a screen to search books by author."
• 10:20. The agent writes the query WHERE author = ?, tries it on 1,000 books and it runs in under 1 ms. The dev approves because the result is right and the tests are green.
• Three months later, 09:30. The catalogue has 5 million books. If scan time grew in proportion to the row count (an assumption for illustration), 2.2 ms at 100,000 rows becomes about 110 ms at 5 million rows (2.2 times 50). Each author search now occupies the machine for a noticeable time, and when many people search at once the screen feels slow.
• The fix: run the query plan, see SCAN books, add an index on author and measure again. With the assumption that an index lookup grows very slowly with row count, the time would stay very small.

What to do from 10:20: when reviewing a new query, ask for the query plan and the expected number of rows. The illustration proves nothing. The evidence is the experiment in section 2 and the PostgreSQL docs: an index costs something on writes, so choose it on purpose and measure, instead of adding them everywhere. As for the need to learn this, ITviec (through search results, the figure being from the job platform's own data) reports job postings requiring SQL up about 38.7% year on year in Q3 2025, so this is not a skill made obsolete by AI.

6. Exercises with an answer key#

Run the script in section 7 once to have the data, then do three exercises. Try to guess the result before you run it.

Exercise 1. Add an index for the query WHERE year = 2015 AND genre = 'tech'. Try both (year, genre) and (genre, year). Which index uses both conditions, and how do the times differ?

Exercise 2. Create an index on in_stock, then run WHERE in_stock = 0 and WHERE in_stock = 1. Why is the benefit so different from searching by author?

Exercise 3. Run WHERE author LIKE 'Author 1234%' when only the default index on author exists. Why is it still a SCAN? There are two ways to fix it; can you find them?

Answers (from my run)#

Exercise 1. Both orders use both conditions, because both are equalities:

no index:      SCAN books                                    1.602 ms (149 rows)
(year, genre): SEARCH books USING INDEX a (year=? AND genre=?) 0.071 ms
(genre, year): SEARCH books USING INDEX b (genre=? AND year=?) 0.074 ms

The difference is almost nothing. The point to remember: for a query using only equality on every column, the order matters little. It matters when a query filters on a single column (as in 2.4) or has a range.

Exercise 2. On a fresh database for each measurement:

in_stock = 0  (10,034 rows): scan 5.551 ms, index 5.029 ms
in_stock = 1  (89,966 rows): scan 39.209 ms, index 41.749 ms

The benefit is tiny for = 0 (about 9%) and absent for = 1. Fetching and passing on many rows takes most of the time. The lesson: selectivity, and the number of rows returned, decide the value of an index. Some other queries on this column may still benefit (for example counting the books in stock could use a covering index); I did not measure that case separately in this article.

Exercise 3. According to the SQLite docs, LIKE is case-insensitive by default, so the LIKE optimization uses only an index with the NOCASE collation. The default index is BINARY, so it is not used. Two fixes: write GLOB 'Author 1234*' (GLOB is case-sensitive and can use a BINARY index, 0.029 ms), or create the index CREATE INDEX nc ON books(author COLLATE NOCASE), after which LIKE 'Author 1234%' switches to SEARCH (I tried it: 0.029 ms).

7. The script to rerun (optional)#

All you need is Python 3 (the sqlite3 library is included). Save it as index_demo.py and run python3 index_demo.py. It takes about half a minute. The numbers will differ by machine, but the shape of the results (SCAN against SEARCH, which times are larger or smaller) should be similar.

import itertools
import random
import sqlite3
import statistics
import time

N = 100_000
REPS = 200
GENRES = ["novel", "science", "history", "children", "cooking", "tech", "poetry", "comic"]


def make_rows(n, seed):
    rnd = random.Random(seed)
    rows = []
    for i in range(n):
        rows.append((
            f"Book {rnd.randrange(10**6):06d}",
            f"Author {rnd.randrange(2000):04d}",
            rnd.randrange(1950, 2025),
            rnd.choice(GENRES),
            1 if rnd.random() < 0.9 else 0,
        ))
    return rows


def new_db(rows):
    db = sqlite3.connect(":memory:")
    db.execute("CREATE TABLE books (id INTEGER PRIMARY KEY, title TEXT, author TEXT, "
               "year INTEGER, genre TEXT, in_stock INTEGER)")
    db.executemany("INSERT INTO books (title, author, year, genre, in_stock) VALUES (?,?,?,?,?)", rows)
    db.commit()
    return db


_n = itertools.count()


def plan(db, sql):
    # the counter comment makes each EXPLAIN text unique, so Python never reuses a stale cached plan
    return [r[3] for r in db.execute(f"EXPLAIN QUERY PLAN /*{next(_n)}*/ " + sql)]


def timed(db, sql):
    samples = []
    count = 0
    for _ in range(REPS):
        t = time.perf_counter()
        count = len(db.execute(sql).fetchall())
        samples.append((time.perf_counter() - t) * 1000)
    return statistics.median(samples), count


def show(db, label, sql):
    p = plan(db, sql)
    ms, count = timed(db, sql)
    print(f"[{label}] {sql}")
    print(f"    plan : {' | '.join(p)}")
    print(f"    rows : {count}   median: {ms:.3f} ms over {REPS} runs")


rows = make_rows(N, seed=42)
db = new_db(rows)
print("SQLite", sqlite3.sqlite_version, "rows", N)
print()

print("=== 1. One author: scan then index ===")
q1 = "SELECT * FROM books WHERE author = 'Author 1234'"
show(db, "no index", q1)
db.execute("CREATE INDEX idx_author ON books(author)")
show(db, "index on author", q1)
print()

print("=== 2. A column with two values (in_stock: 90% are 1) ===")
q2 = "SELECT * FROM books WHERE in_stock = 1"
show(db, "no index", q2)
db.execute("CREATE INDEX idx_stock ON books(in_stock)")
show(db, "index on in_stock", q2)
db.execute("DROP INDEX idx_stock")
print()

print("=== 3. Two columns: order matters ===")
q3 = "SELECT * FROM books WHERE genre = 'tech' AND year >= 2020"
q4 = "SELECT * FROM books WHERE year = 2015"
show(db, "no composite index, q3", q3)
db.execute("CREATE INDEX idx_genre_year ON books(genre, year)")
show(db, "index (genre, year), q3", q3)
show(db, "index (genre, year), q4 uses year only", q4)
db.execute("DROP INDEX idx_genre_year")
db.execute("CREATE INDEX idx_year_genre ON books(year, genre)")
show(db, "index (year, genre), q3", q3)
show(db, "index (year, genre), q4 uses year only", q4)
db.execute("DROP INDEX idx_year_genre")
print()

print("=== 4. Queries an index on author cannot help ===")
q5 = "SELECT * FROM books WHERE lower(author) = 'author 1234'"
q6 = "SELECT * FROM books WHERE author LIKE '%1234'"
q7 = "SELECT * FROM books WHERE author LIKE 'Author 1234%'"
show(db, "index on author, lower()", q5)
show(db, "index on author, LIKE '%1234'", q6)
show(db, "index on author, LIKE 'Author 1234%'", q7)
q8 = "SELECT * FROM books WHERE author GLOB 'Author 1234*'"
show(db, "index on author, GLOB 'Author 1234*'", q8)
print()

print("=== 5. The price: size and inserts (20,000 new rows, 7 runs each) ===")
extra = make_rows(20_000, seed=7)


def insert_time(index_sqls):
    samples = []
    pages = 0
    for _ in range(7):
        d = new_db(rows)
        for s in index_sqls:
            d.execute(s)
        t = time.perf_counter()
        d.executemany("INSERT INTO books (title, author, year, genre, in_stock) VALUES (?,?,?,?,?)", extra)
        d.commit()
        samples.append((time.perf_counter() - t) * 1000)
        pages = d.execute("PRAGMA page_count").fetchone()[0] * d.execute("PRAGMA page_size").fetchone()[0]
        d.close()
    return statistics.median(samples), pages / (1024 * 1024)


setups = [
    ("0 indexes", []),
    ("1 index (author)", ["CREATE INDEX i1 ON books(author)"]),
    ("3 indexes (author, genre+year, title)", [
        "CREATE INDEX i1 ON books(author)",
        "CREATE INDEX i2 ON books(genre, year)",
        "CREATE INDEX i3 ON books(title)",
    ]),
]
base = None
for name, sqls in setups:
    ms, mb = insert_time(sqls)
    base = base or ms
    print(f"{name:40s} insert 20,000 rows: {ms:8.1f} ms  (x{ms / base:.2f})   size after: {mb:6.2f} MB")

8. A checklist before adding or reviewing a query#

1. How many rows will this table have after a year, and did I try it at that size?
2. Did I read the query plan? Is the first line SCAN or SEARCH?
3. About what percentage of rows does the filter rule out?
4. Does the query wrap the column in a function, or put % at the start of a LIKE?
5. For a multi-column index, does the leading column appear in the real queries?
6. Which conditions do the brackets in the plan show the index using?
7. Is the write cost of the new index acceptable for a table that changes often?
8. Is there an old index that no longer appears in the plan of any real query and can be dropped?
9. If an AI wrote the query, did it paste the plan and the assumed number of rows for me?

9. A balanced view#

Fairly solid (measured or in the docs): an index can make a selective query tens of times faster at around 100,000 rows; selectivity, column order and the way the condition is written decide whether an index helps; indexes make writes slower and take space.

Handle with care:

• The experiment ran on one machine, in memory, with randomly generated data distributed evenly. Real data is often skewed (a few authors have very many books), and disks, caches and concurrent access change the results.
• SQLite is not PostgreSQL or MySQL. Each has its own planner, LIKE rules, index types and optimizations (for example PostgreSQL 18 can skip the leading column in some cases). I did not test PostgreSQL directly in this round, and what is said about it comes from its docs.
• The article does not cover planner statistics (ANALYZE), partial indexes, deeper covering indexes, caching or how a table is stored on disk. Those are good topics to learn next.
• The 79 times figure belongs to this experiment. Do not use it as a promise for your system.

Do not overstate: an index is not a cure for every slow query. Sometimes a query is slow because it fetches too many rows or columns, lacks a LIMIT, or because of the table design. Do not dismiss it either: on a large table, not reading the query plan is often the cheapest cause to fix.

10. Conclusion#

An index is the database's card catalogue: very fast when the question matches the way it is sorted, useless when it does not, and always costly to maintain. To use it well you need three habits: read the query plan, ask how many rows the condition rules out, and measure again after adding one. For a query written by AI, ask for that same plan as evidence. These habits will outlast any tool.

To go further: Reviewing AI-Written Code has a checklist for changes, and Testing AI-Written Code helps you have tests protecting you when you change a query.

References#

The measurements in this article are mine, run on SQLite 3.53.4 (Python 3.14.8, macOS arm64), and may differ on your machine. I opened the documents below to check the points attributed to them. The ITviec figure was seen only through search results and I did not open the original report.

• SQLite, The SQLite Query Optimizer Overview (multi-column indexes used left to right, the conditions for the LIKE and GLOB optimizations)
• SQLite, EXPLAIN QUERY PLAN (meaning of SCAN, SEARCH and USING INDEX, the note that the format may change)
• PostgreSQL, Multicolumn Indexes and Introduction to indexes (version 18 docs)
• Markus Winand, Use The Index, Luke: Concatenated Indexes
• ITviec, Vietnam IT recruitment market overview Q3 2025 (seen only through search results, measures only job postings on its own platform)

For anyone who hands code work to an AI agent, including people who have never used Git. No background needed. Every term comes with an everyday picture, and the parts with commands are marked so you can copy them or skip them.

Every command and output in the worked example was really run in a practice repository (Git version 2.56.0, 30 October 2026). Scenarios marked "illustration" are made up and name no real person or company. How an agent goes wrong inside is not fully understood by anyone, so the pictures below are memory aids only.

Quick summary#

• An agent can edit many files in minutes. Without a way back, one bad edit can swallow a whole working session. Git is the save point: it remembers only what you have committed.
• The core habit: one branch per task you give the agent, ask the agent for small commits, and read git status and then git diff before accepting the result.
• There are four ways back, each for its own situation: git restore (not committed yet), git revert (already committed, the safest), git reflog (rescue after a slip), git worktree (let the agent work in its own folder from the start).
• The biggest blind spot: uncommitted changes that are discarded with git restore cannot be recovered. In this post's practice repository, the reflog had no entry for them.
• Save points only cover files in the repository. A deleted database or a sent email lies outside Git.
• Destructive commands (push --force, reset --hard, clean) belong in the agent's instructions as do not run without asking.

Hard words, plain meaning#

1. A picture first: a game with save points#

Picture a long game. If you never save, dying once costs you an hour of play. If you save every few minutes, dying costs a few minutes.

Handing work to an agent is the same. An agent is a very fast, slightly hasty player: it can edit ten files before you finish reading the first. It is good but not always right, and the documentation of agent tools says as much (see section 6). A save point does not make the agent err less. It makes the error cheaper.

Git plays that role, with one important rule: Git remembers only what has been committed. Changes sitting in the folder without a commit are like playing mid-level without saving.

Takeaway: before you hand over a task, have a clean save point. While the agent works, ask it to save often.

2. The five-step routine: one branch per task#

Picture a dinner table with several trays: you do not try a new dish on the main plate, you put it on a separate plate. If it fails, you throw the plate away and the main plate is untouched.

Everything below was run in a practice repository with one file, prices.py, which adds up an order total. There is one first save point on the branch main.

Step 1. Make a branch for this task#

git switch -c agent/add-discount

Switched to a new branch 'agent/add-discount'

The naming habit agent/task-name lets you see at a glance which branches belong to the agent. It is my habit, not a Git rule.

Step 2. Hand over the task and let the agent work#

You ask the agent: "add an optional discount parameter to the total function". The agent edits the file.

Step 3. Read git status and then git diff before accepting#

git status -sb
git diff

## agent/add-discount
 M prices.py
diff --git a/prices.py b/prices.py
index de87eb8..d7bc76f 100644
--- a/prices.py
+++ b/prices.py
@@ -1,2 +1,3 @@
-def total(items):
-    return sum(item["price"] * item["qty"] for item in items)
+def total(items, discount=0):
+    subtotal = sum(item["price"] * item["qty"] for item in items)
+    return subtotal * (1 - discount)

M means one file was modified. A line starting with - was removed, a line starting with + is new. This reading is the "open the evidence" step from Make AI Output Prove Itself, and the first part of reviewing AI-written code.

Step 4. Commit small, like a save point#

git add prices.py
git commit -q -m "feat: add optional discount to total"
git log --oneline

c9c7cf3 feat: add optional discount to total
6dd4973 feat: add total function

The -q flag only keeps the command quiet. One job per commit, describable in a sentence, in the feat: job name style of Conventional Commits (a popular note-writing convention, not mandatory).

Step 5. Merge when you are satisfied#

When you have read the diff and run the checks, you are the one who merges the branch into main:

git switch main
git merge --no-ff -q agent/add-discount -m "merge: add discount"
git log --oneline --graph

*   04cbf8d merge: add discount
|\  
| * 5aee66b feat: add extra script
| * c5b6fbf Revert "refactor: simplify total"
| * 6f88dd0 refactor: simplify total
| * c9c7cf3 feat: add optional discount to total
|/  
* 6dd4973 feat: add total function

The main branch only receives what you have looked at. The middle save points (refactor, Revert, extra script) come from the experiments in section 3.

2.1 A calculation you can redo (illustration, skippable)#

This is an example with made-up numbers. Suppose an agent works for 3 hours (180 minutes) without a break and something goes wrong in the last minute.

session = 180
no_commit_loss = session            # never saved: lose up to 180 minutes
every = 20
saves = session // every            # 9 save points
worst_loss = every                  # lose at most 20 minutes
print(saves, worst_loss, session - worst_loss)
# 9 20 160

Reading the result: committing every 20 minutes gives 9 save points, and the worst accident costs at most 20 minutes instead of 180, so 160 minutes are kept. The 20 minutes is my assumption for the illustration. Real life has no formula, but the rule is to make steps small enough to look back at and dare to throw away.

3. Going back: the right command for the right situation#

Picture accidentally deleting a paragraph. How you get it back depends on whether you had saved the document.

3.1 git restore: discard uncommitted changes#

The agent edits the file into a useless function:

git status -sb
git diff --stat
git restore prices.py
git status -sb

## agent/add-discount
 M prices.py
 prices.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
## agent/add-discount

After restore, git status no longer shows the M line, meaning the file is back to the committed version. Warning: if the discarded change was something you wanted to keep, there is no way back. I tested it: edit the file, run git restore, then look at the reflog and the stash list.

git reflog | head -2
git stash list | wc -l

5aee66b HEAD@{0}: reset: moving to HEAD
5aee66b HEAD@{1}: reset: moving to HEAD@{1}
       0

The two reflog lines belong to earlier operations. There is no record for the discarded line, and the stash list is empty (0 entries). The git restore manual page describes the command as restoring content from a source, so the current content is replaced. To keep it, git stash first, or commit first.

3.2 git revert: undo an existing commit#

The agent commits a "simplification" that breaks the function:

git commit -qam "refactor: simplify total"
git revert --no-edit HEAD
git log --oneline

[agent/add-discount c5b6fbf] Revert "refactor: simplify total"
 Date: Fri Oct 30 17:31:17 2026 +0700
 1 file changed, 2 insertions(+), 1 deletion(-)
c5b6fbf Revert "refactor: simplify total"
6f88dd0 refactor: simplify total
c9c7cf3 feat: add optional discount to total
6dd4973 feat: add total function

The broken commit stays in the history, and a new commit undoes it. According to the Git documentation, git revert creates new commits that undo earlier ones instead of deleting them. Because it does not rewrite history, it is the safe choice on a shared branch.

3.3 git reset --hard then git reflog: a slip can still be rescued#

This is the scariest part, so I ran it for real. Add one more commit, then step back two:

git reset --hard HEAD~2
git log --oneline
ls

HEAD is now at 6f88dd0 refactor: simplify total
6f88dd0 refactor: simplify total
c9c7cf3 feat: add optional discount to total
6dd4973 feat: add total function
prices.py

The file extra.py disappeared from the folder, and the function in prices.py went back to the broken version. But the reflog remembers:

git reflog | head -8

6f88dd0 HEAD@{0}: reset: moving to HEAD~2
5aee66b HEAD@{1}: commit: feat: add extra script
c5b6fbf HEAD@{2}: revert: Revert "refactor: simplify total"
6f88dd0 HEAD@{3}: commit: refactor: simplify total
c9c7cf3 HEAD@{4}: commit: feat: add optional discount to total
6dd4973 HEAD@{5}: checkout: moving from main to agent/add-discount
6dd4973 HEAD@{6}: commit (initial): feat: add total function

The save point 5aee66b is still there. Set the branch back to it:

git reset --hard HEAD@{1}
git log --oneline
ls

HEAD is now at 5aee66b feat: add extra script
5aee66b feat: add extra script
c5b6fbf Revert "refactor: simplify total"
6f88dd0 refactor: simplify total
c9c7cf3 feat: add optional discount to total
6dd4973 feat: add total function
extra.py
prices.py

Everything is back. According to the Git documentation, reflog entries are kept for 90 days by default, and 30 days for entries no longer on any branch, so rescue early. The reflog is a notebook on your machine and is not pushed to the server.

3.4 git stash: put work in progress aside#

git stash push -m "half done idea"
git stash list
git stash pop

Saved working directory and index state On agent/add-discount: half done idea
stash@{0}: On agent/add-discount: half done idea
Dropped refs/stash@{0} (377cf291f166fc06e84e9ac41e27da37fa79d8d9)

Between the two commands, git status was clean. After pop, the changes came back. Use it when you want free hands to let the agent do something else, without making a half-baked commit.

Takeaway: remember one sentence: what is committed can almost always be rescued, what is not committed cannot. That is why you commit small.

4. Worktree: give the agent its own desk#

Picture a kitchen with two counters: you cook at one, the assistant cooks at the other. Same kitchen, same pantry, but you do not bump elbows.

A worktree is a second working folder of the same repository. According to the Git documentation, it shares the repository with the main folder but has its own HEAD and index, and by default Git does not let two worktrees stand on the same branch.

git worktree add ../demo-git-wt -b agent/refactor-names
git worktree list

Preparing worktree (new branch 'agent/refactor-names')
HEAD is now at 5aee66b feat: add extra script
/path/to/demo-git    5aee66b [agent/add-discount]
/path/to/demo-git-wt 5aee66b [agent/refactor-names]

(I shortened the paths to /path/to/ for readability.) Now the agent works in demo-git-wt while you keep editing in the main folder. When it is done, clean up:

git worktree remove ../demo-git-wt
git branch -d agent/refactor-names

Deleted branch agent/refactor-names (was 5aee66b).

Without --force, git worktree remove only removes a clean folder. Claude Code has a shortcut, claude --worktree job-name, that opens a session in a new worktree, according to its Common workflows page. Other tools may have their own way, so check the documentation of the tool you use.

5. House rules for the agent#

Picture handing a house key to a helper: you say which rooms they may enter and which not.

5.1 Habits worth writing down as rules#

• Do not commit straight to main. One branch per task.
• No push --force, and no pushing without asking. Pushing to a server is an action other people see.
• No reset --hard, no clean, no deleting branches without asking. git clean deletes files Git is not tracking, so I treat them as having no copy to rescue.
• Do not touch files that hold keys, such as .env. Let .gitignore block them, and check with git check-ignore -v .env.

git check-ignore -v .env

.gitignore:1:.env	.env

The line says the rule on line 1 of .gitignore is blocking the file .env. (In the practice repository my environment would not let me create a real .env file, so I only checked the rule and did not try git status with a real file.)

5.2 Five lines to paste into the agent's instructions#

Put this block in AGENTS.md or your tool's instruction file:

## Git
- Work on a branch named agent/<task-name>, never commit straight to main.
- Commit small, one job per commit, run the checks before committing.
- Do not use push --force, reset --hard, clean or delete branches without asking.
- Do not push. Do not edit .env or any file that holds keys.
- At the end: paste the output of git status and git diff --stat.

This is guidance written in words, and an agent may ignore it. It is the first barrier, not the only one: also use the tool's permission settings and the branch protection of your code hosting service, if available.

5.3 What an agent's commit note should look like#

feat: add optional discount to total

- discount defaults to 0, so existing callers keep working
- checked with: python -m pytest tests/test_prices.py (3 passed)

The first line says what, the body says why and how it was checked. The note is itself a small piece of evidence, in line with way 2 in Make AI Output Prove Itself.

6. Numbers and documents used#

Read it correctly: the Git and Claude Code documents support using save points and separate branches. I did not find a study measuring how much a separate branch reduces incidents when using agents, so the habits above are good practice, not measured results.

7. Where save points still fail#

Picture saving the game on this machine while the character has already done something in the real world outside.

7.1 Illustration: a morning with an agent#

This is an illustration, naming no real person or product.

• 09:00. The dev creates the branch agent/add-discount and assigns the job "add a discount to the total function".
• 09:20. The agent says it is done. The dev runs git diff --stat and sees that one file changed more than expected. On a closer look, the agent rewrote the whole old function.
• 09:25. Nothing is committed yet, so the dev has two choices: git restore (lose everything, including the good part) or git stash and then pick what is needed. The dev chooses stash, keeps the good part, and asks the agent to redo the rest more tightly.
• 10:10. The agent commits a second time. The dev notices this commit breaks a test and runs git revert HEAD. The history stays, the function goes back as before.
• 10:30. The dev merges into main after reading the diff.

What to do: commit early, read the diff before accepting, and get used to stash and revert more than restore and reset. This illustration proves nothing. The evidence lies in the documents above, and in one real case worth remembering: 2025 news reports said that during a trial by a founder, an agent was reported to have deleted a production database despite a code freeze. The sources disagree on details such as recoverability, and most of the information comes from the person involved. If it happened as told, it was about a database, not files, which lies outside what Git save points cover.

8. Habits to apply from tomorrow#

8.1 Eight questions before handing over and before accepting#

1. Am I on this task's own branch, or on main?
2. Before handing over, is git status clean, or is there unsaved work in progress?
3. Does the agent know the Git rules (own branch, small commits, no push)?
4. After the agent worked, have I read git status and git diff?
5. Was any file changed that I did not ask for?
6. Is there anything that looks like a key or a password in the diff?
7. If I drop this edit, do I use stash, restore or revert, and am I losing anything uncommitted?
8. Does this job touch anything outside the repository (database, mail, API)?

8.2 Three things to do right now#

1. Paste the five-line block from 5.2 into the instruction file of the tool you use.
2. Try once in a scratch repository: reset --hard, then reflog, then set it back, so you do not panic when it matters.
3. Turn on protection for the main branch on your code hosting service, if it has one.

9. A balanced view: what Git does and does not do#

What it does: gives you save points, lets you try things without fear, allows several jobs in parallel through worktrees, and turns every change the agent makes into a readable diff.

What it does not do:

• It does not cover things outside the repository: databases, sent mail, money moved.
• It cannot rescue uncommitted changes.
• It does not stop an agent from doing something bad, it only helps you go back after it happens. Prevention needs permissions and rules.
• Learning Git has a cost. For a newcomer, the few commands in this post are enough to start. Harder cases such as merge conflicts take more time to learn.
• I have not seen a controlled trial measuring the effect of each habit. Try them in your team and adjust.

Do not exaggerate: Git is not a safety switch. Do not play it down: most accidents when using agents on files can shrink to "lost 20 minutes" if a save point is close by.

10. Conclusion#

Agents are fast, and fast even when wrong. Give each task its own branch, ask it to save often, read the diff before accepting, and remember four ways back: restore for the uncommitted part (and understand it is a real loss), revert for an existing commit, reflog to rescue after a slip, worktree so each agent has its own desk. A save point does not make the agent err less, but it makes you less afraid.

References#

Commands and facts rest on public documentation and a practice repository run on 30 October 2026 (Git 2.56.0). The news reports on the Replit case mostly retell the posts of the person involved, so open the originals before quoting.

• Git, git-restore, git-revert, git-reflog, git-worktree
• Anthropic, Checkpointing and Common workflows, section Run parallel sessions with worktrees
• Tom's Hardware, AI coding platform goes rogue during code freeze and deletes entire company database and heise (July 2025)
• Conventional Commits, Specification 1.0.0

For team leads, freelancers, small business owners and anyone in a team of a few people who uses AI at work, including beginners. No legal or security background is needed. Every term comes with an everyday picture, and the calculation is marked so you can skip it.

This is not legal advice. Figures come from public sources listed at the end, and in some places I only read a news report. Examples labeled "illustration" are made up and do not refer to real people or companies. Nobody fully understands the inside of AI, so the pictures here are memory aids. The template is my practical suggestion, not a mandatory standard.

Quick summary#

• Small teams often have no rules for AI. A news report on the Stack Overflow 2026 survey says about 29.7% of respondents left AI use to the individual and about 24% said their workplace had approved tools or an official policy. These are numbers read through a news report.
• The biggest risk is often not that AI is wrong, but that someone pastes data they should not. In 2023, according to Bloomberg's report and outlets repeating it, Samsung employees were reported to have put internal source code into ChatGPT, and the company then restricted generative AI tools.
• One page is enough to start: three columns (allowed, ask first, forbidden), four data levels in plain words, a list of approved tools, one accountable person for each AI-written change, incident steps and a review date.
• A way to test the policy: take 20 real tasks from last week and sort them into the three columns. If too many land in "ask first", the policy is too strict or too vague.
• A policy that is too strict makes people work around it, a vague one leaves everyone guessing, and one nobody reads is the same as none. The post shows how to spot each.
• A policy is not a safety guarantee. It only makes what to do clear and gives someone responsibility.

Hard words, plain meaning#

1. A picture first: house rules for a shared kitchen#

Picture a shared kitchen for six people. Nobody wants to read a thick rulebook. But with nothing at all, one person uses the same board for raw and cooked food and another leaves expired food in the fridge. One sheet of paper on the fridge door is enough: this you can just use, this you ask about first (say, the landlord's pressure cooker), this never (leaving the gas on when you go out). A short sheet gets read. A review date keeps it from going stale. A named person means someone to ask.

An AI use policy for a small team works the same way. The goal is not to cover every situation, but to give everyone the same answer to the questions that come up most: can I paste this here, who reads what the AI wrote, what do I do if I slip.

Takeaway: a good policy for a small team is a sheet on the fridge door, not a binder in a drawer.

2. Why it matters: two things with data#

2.1 Most teams have no rules yet (Stack Overflow 2026 survey, via a news report)#

A news report on the Stack Overflow Developer Survey 2026 (IT Brief) says about 29.7% of respondents said AI use at work was optional and left to the individual, and only about 24% said their employer had approved tools or an official policy or guidance. I only read these numbers through the news report and did not check them against the survey pages, and the sample is people who chose to answer, not every small team. Read it as "quite a few places do not have one", not as an exact figure.

Takeaway: if your team has no policy you are not unusual, and the gap is common, not specific to you.

2.2 A reported case: source code into a chatbot (Samsung, 2023)#

According to Bloomberg's first report in early May 2023 and outlets repeating it (Forbes, TechRadar), Samsung engineers used ChatGPT to deal with problems in source code and were reported to have put internal source code into it around April 2023. The company then sent a notice restricting generative AI tools on company computers, phones and networks, allowing them on personal devices as long as company intellectual property was not at risk, and saying violations could lead to discipline. According to these reports the notice did not say what the leaked code contained. Sources also disagree on whether the restriction was temporary or lasting, and I could not open Bloomberg's original, so keep this at the level of "according to reports".

What matters is not the name of a big company. It happened where the engineers were skilled: one very ordinary task (asking a chatbot to look at a bug) plus one very ordinary habit (pasting the code in) is enough.

Takeaway: the risk is in the act of pasting data, and it needs a rule that says which data must not be pasted.

3. The one sheet: three columns and seven items#

Picture the sheet on the fridge door, split into green, yellow and red columns, a few lines each.

3.1 Three columns#

• Allowed (no need to ask): use approved tools with public and internal data for things like explaining, drafting, writing tests, renaming and writing docs.
• Ask first: using a tool not on the list, putting client data into any tool, letting an agent run install, delete or deploy commands, using AI on work the client said not to use it on.
• Forbidden: pasting keys, passwords, tokens and real personal data, using a personal free account for client data, merging AI-written code that no second person has read and no check has run on.

3.2 Four data levels in plain words#

Picture a house with four places to keep things.

3.3 The other seven items#

1. Approved tools: name, account type, who approved, date, and whether the data terms were read.
2. Who reviews the result: every AI-written change is read by a second person. The ways of asking for proof in Make AI Output Prove Itself help the reader go faster.
3. Who is accountable: the person who merges or sends it to the client answers for it. See also Who is accountable when AI is wrong.
4. Telling the client: when to tell, what to say, as agreed in the contract.
5. Logging: for new AI use, one line: tool, purpose, data level.
6. When something goes wrong: the short steps in section 6.
7. Review date: usually about 90 days out, with a named keeper.

3.4 A template to paste and edit#

AI USE POLICY OF TEAM [team name]
Keeper: [name]    In effect: [date]    Review on: [date + 90 days]

ALLOWED (no need to ask)
- Use the tools on the list below with Public and Internal data.
- Ask AI to explain, draft, write tests, rename, write docs.

ASK FIRST
- Use a tool not on the list.
- Put Customer data into any AI tool.
- Let an agent run install, delete or deploy commands.
- Use AI on work where the client asked us not to.

FORBIDDEN
- Pasting keys, passwords, tokens, real personal data.
- Using a personal free AI account for client data.
- Merging AI-written code before a second person has read it and a check has run.

DATA: Public / Internal / Customer / Secret (examples in section 3.2)
APPROVED TOOLS: [name, account type, approver, date]
ACCOUNTABLE: the person who merges or sends it to the client
TELLING THE CLIENT: [when, how]
LOGGING: new AI use, one line (tool, purpose, data level)
INCIDENT: stop, report within 1 hour, write down what was done, rotate keys if exposed, tell the client if needed

4. An illustrated example for a six-person team#

This example is made up and does not refer to any real company. Picture a team of six building software under contract for a few clients. They fill in the template:

• Approved tools: one paid service on a team account created by the lead, with the approver having read the terms on whether the team's data is used for training (you need to read the real terms of whatever service you pick). Personal free accounts are not on the list.
• Ask first: anything touching client data, and every time an agent is allowed to run an install or a deploy.
• Telling the client: the quote states that the team uses AI to help write code, every change is read by a person, and the client's data is not put into the tool without agreement.
• Review: every 90 days, kept by the lead.

4.1 The test: sort 20 real tasks into the three columns (illustration, can be skipped)#

A way to test the policy: each person lists real tasks where they used AI last week, and the team sorts them into the three columns. The numbers below are made up to be easy to compute, not measurements.

tasks, people, minutes_per_ask = 20, 6, 3
allowed, ask, banned = 12, 6, 2
print(allowed / tasks, ask / tasks, banned / tasks)
# 0.6 0.3 0.1
weekly = tasks * people            # 120 tasks a week for the team
asks = weekly * ask / tasks        # 36 ask-first questions
minutes = asks * minutes_per_ask   # 108 minutes
print(weekly, asks, minutes, minutes / 60)
# 120 36.0 108.0 1.8

Reading the result: of 20 tasks, 12 (60%) go ahead at once, 6 (30%) need a question and 2 (10%) are forbidden. With 6 people at 20 tasks each a week, that is about 36 questions, roughly 108 minutes, or about 1.8 hours a week for the whole team. If 12 of the 20 tasks (60%) landed in "ask first", the same calculation gives 72 questions and 3.6 hours a week, and people would stop asking.

Be clear about this: three minutes per question and the 12, 6, 2 split are assumptions. The rule of thumb that "ask first should not exceed about a third of tasks" is mine, not a measurement from any study. Try it with your team's real tasks and adjust.

5. Where a policy fails#

Picture the house rules: too long and nobody reads them, too strict and people cook in secret, too vague and everyone reads them their own way.

5.1 Illustration: one morning with and without a policy#

This is an illustrative scenario that names no real person or product.

• 10:00. A dev needs to find a bug in a report exported for a client. The error log contains emails and names of the client's end users.
• 10:05. The dev pastes the whole log into a public chatbot on a personal free account, because nobody ever said not to.
• 10:30. The dev notices real emails in what was pasted. There is no procedure, so the dev stays quiet.
• Next week. The client asks where their data went. Nobody has an answer.

The version with a policy: at 10:05, the line "customer data means ask first" makes the dev message the lead. Together they strip the emails from the log, use an approved tool and write one line. If the paste had still happened, the "report within an hour" step lets the team deal with it early: note what was pasted, rotate keys if any, tell the client if needed.

What to do: write the incident section before there is an incident, because people rarely think of the steps in a panic.

This illustration is not evidence. The Samsung case reported above is a real example of the same kind of behavior, and the Stack Overflow 2026 survey (via a news report) suggests the policy gap is common. See also OWASP Top 10 for AI Apps for technical risks.

6. When something goes wrong: five short steps#

1. Stop: stop what you are doing with that tool.
2. Report within an hour: to the accountable person, even if you are not sure it is an incident.
3. Write it down: what was put in, into which tool, when.
4. Deal with it: rotate keys if they were exposed, delete the conversation if the tool allows, ask the provider about data retention.
5. Tell the client if the contract or the law requires it, and put the lesson back into the policy.

Allow reports of near misses as well. If reporting is punished, people will stay silent.

7. Numbers, standards and law: how to read them#

On the legal side: according to the sources I found, the law was passed by the National Assembly on 10 December 2025 and takes effect on 1 March 2026. It defines a deployer as an organization or individual using an AI system in professional, commercial or service activity, and use for personal, non-commercial purposes falls outside that scope. I have not verified the specific obligations of each party, the implementing rules or the transition periods. If your team supplies products or services that use AI to clients, read the full law and ask a legal professional. This post only helps you know what to ask.

Read it fairly: the sources above support having clear rules and a named person accountable. I found no study measuring how many incidents a one-page policy prevents. The template here is a synthesis of practice.

8. Habits to apply from tomorrow#

8.1 Eight questions before posting the policy#

1. Does the policy fit on one page?
2. Do the three columns have concrete examples, not only concepts?
3. Do the four data levels have examples familiar to your team?
4. Is there a list of approved tools with dates and approvers?
5. Does every AI-written change get read by a second person?
6. Is there a named accountable person and a named keeper of the policy?
7. Are the incident steps written down, and is reporting a near miss allowed?
8. Is there a review date?

8.2 Three things to do now#

1. Spend 30 minutes filling in the template in 3.4 and cut what your team does not need.
2. Run the 20-task test in 4.1 with last week's real tasks.
3. Set a review date about 90 days out and write the keeper's name.

9. A balanced view: what a policy does and does not do#

It does: give everyone the same answer to recurring questions, tell newcomers what to do, create a place to report when someone slips and leave a trail when a client asks.

It does not:

• A policy is not a safety guarantee. People can still paste by mistake, and tools can change their terms.
• I have not seen a study measuring how well a one-page policy works. Track your own count of questions and near misses.
• One sheet of paper does not replace training, checking tools and a culture where people dare to report mistakes.
• For teams that supply services to clients or handle personal data, one page is a starting point, not the full set of legal obligations.

Do not oversell it: no template fits every team. Do not wave it off: the policy gap tends to show up when a client asks, and by then it is too late to write one.

10. Conclusion#

A small team does not need a thick rulebook. It needs a sheet on the fridge door: allowed, ask first, forbidden, four data levels, one accountable person, five steps when something goes wrong and a review date. Draft it in 30 minutes, test it on 20 real tasks and revise it after 90 days.

References#

Figures come from public sources as of October 2026. Some sources I read only through news reports or secondary summaries, so open the originals before quoting. This is not legal advice.

• Stack Overflow, 2026 Developer Survey and the IT Brief report on the survey (the source of the 29.7% and 24% figures)
• Reports on Samsung's restriction of generative AI (May 2023): Forbes Australia and TechRadar (repeating Bloomberg's report)
• NIST, AI Risk Management Framework (AI RMF 1.0, January 2023)
• ISO, ISO/IEC 42001:2023
• Law on Artificial Intelligence, No. 134/2025/QH15 on Thu Vien Phap Luat and the Baker McKenzie overview (read the full text before relying on it)

For anyone who works in a team with code, including newcomers and people who hand work to AI agents. No software design background is needed. Every term comes with an everyday example, and the calculation is marked so you can skip it.

The numbers come from public sources listed at the end. Examples labeled "illustration" are made up for this post and do not mention real people or companies. How an AI agent reads and uses documents internally is not fully understood by anyone, so the pictures below are memory aids only. The AI advice in this post is a synthesis of practice, not the result of a controlled experiment.

Quick summary#

• The hardest part of a project is often not the code but the reasons behind the code. Reasons tend to live in a few heads, then leave with them.
• An ADR (architecture decision record) is one short page that records one decision: context, decision, consequences and status. Michael Nygard proposed the format in 2011 and recommended one to two pages.
• A newcomer without the reasons has two paths, both bad: accept without understanding, or reverse without knowing the consequences. Nygard describes exactly those two paths.
• An AI agent can end up in a similar spot: it sees a "roundabout" piece of code and tidies it up, not knowing the roundabout shape was deliberate. An ADR linked from the agent's instruction file is a place to check before editing.
• A good ADR needs only one page: written for a reader in the future, naming the options that were rejected and what we accept to lose. A template to paste is included.
• An ADR is not a ritual for every change. Small, easily reversed decisions do not need one. And an ADR only helps if someone keeps its status current and someone reads it.

Hard words, plain meaning#

1. A picture first: the note in the recipe#

Picture a restaurant with a recipe book. The soup entry says: "No salt, season with fish sauce at the end." A new kitchen helper finds the soup bland and adds salt. That day a guest allergic to salt orders the soup. Nobody meant harm. The book was missing one line: "why no salt".

Now picture the book with that line: "No salt, because this dish also serves guests on restricted diets. Guests add salt at the table." The helper reads it and understands, and if the restaurant changes its policy, there is a place to edit.

An ADR is that note, but for code. And for an AI agent walking into a repository, it is a note the agent can read before it "adds some salt".

Takeaway: what needs saving is not a thick book but the reason for each decision worth remembering, short enough that someone reads it.

2. What an ADR is, from the sources#

2.1 Michael Nygard's original (2011)#

In "Documenting Architecture Decisions", published on the Cognitect blog on 15 November 2011, Michael Nygard writes that architecture decisions often come with no record of why. A newcomer to the project has only two options: accept the decision without understanding it, or reverse it without knowing the consequences. He says both can hurt the project, and too many unexplained decisions make a team afraid to change anything.

He proposes that each record has five parts:

• Title: a short noun phrase.
• Context: the forces at play (technological, political, social, project local), stated as fact without judgment.
• Decision: full sentences, active voice, starting with "We will".
• Status: "proposed" until the stakeholders agree, then "accepted". Later it may become "deprecated" or "superseded", with a link to the replacement.
• Consequences: the context after the decision, including the positive, negative and neutral effects.

On length: one to two pages. On voice: write as if having a conversation with a future developer, in full sentences, using bullets only for layout and not to replace sentences with fragments. On storage: in the project repository, for example doc/arch/adr-NNN.md, numbered sequentially and never reused. A reversed decision is not deleted: the old record stays and is marked superseded.

Nygard also shares an experience: after about three months of use on several projects, six to ten developers who rotated through them said the context in the records was valuable. This is the author's own account from a small group, not a controlled study.

2.2 MADR: a popular Markdown template#

MADR (Markdown Architectural Decision Records) is a lean template written in Markdown. According to the official page, its parts are: title, optional front matter (status, date, decision makers, consulted, informed), context and problem statement, optional decision drivers, considered options, decision outcome (with good and bad consequences), optional confirmation of compliance, pros and cons of the options, and more information (which can state conditions for revisiting). The storage convention: the directory docs/decisions, with files named NNNN-title-with-dashes.md in sequence. The latest version the page describes is 4.0.0, released 17 September 2024, dual licensed MIT or CC0.

2.3 Y-statement: an ADR in one sentence#

The Y-statement is a short form popularized by Olaf Zimmermann and the authors associated with it, a single sentence in the pattern: "In the context of (situation), facing (concern), we decided for (option) and against (other options), to achieve (quality goal), accepting (downside)." The sources I read word the connectors slightly differently ("and against" or "and neglected"), so treat it as a pattern, not a hard standard. The Y-statement suits simple decisions. For complex ones, the sources suggest fuller formats such as MADR or Nygard's version.

Takeaway: an ADR is not a complicated tool. The core is four questions: what situation are we in, what do we choose, what do we drop, and what do we accept losing.

3. Why teams lose decisions, and why AI makes it more visible#

3.1 Data from the Stack Overflow 2026 survey#

The results page of the Stack Overflow Developer Survey 2026 has a section on context-related difficulties. According to the summary I read, about 60.9% of respondents say important context lives in people's heads, 56.5% say information is outdated, 53.4% say it is spread across too many tools and 63.2% say it is incomplete. I read a summary and did not cross-check the tables, the original questions may be broader than technical decisions, and the respondents are people who chose to take the survey. So read this as a general signal that team knowledge tends to be scattered, not as evidence about ADRs.

3.2 With AI agents: it sees the code, not the reason#

Picture an agent as a colleague who is very fast but joined today: it reads all the code in seconds, but does not remember last week's meeting. When it sees a roundabout piece of code, the reasonable reflex is to simplify it. If the roundabout shape is the result of a decision with a reason, the only thing that can stop it is the reason written where it looks.

A limit to state plainly: I found no study that measures how much putting ADRs in context reduces how often agents reverse decisions. This is a practical argument, connected to the post A handbook for AI: AGENTS.md, where we put what the agent needs to know in the place it reads.

3.3 A calculation you can redo (illustration, skippable)#

This is an illustration with made-up numbers, not measurements. Suppose a team of six records two decisions a month, each ADR is 300 words, and technical reading speed is 150 words per minute (an assumption). In a year the team has 24 ADRs.

adr, words, wpm = 24, 300, 150
read_all = adr * words / wpm          # 7200 words, so 48 minutes
index = adr * 12 / wpm                # 24 titles, 12 words per line, so 288 words, almost 2 minutes

asked, explain = 5, 15                # a newcomer re-asks 5 decisions, a senior explains 15 minutes each
without = asked * explain             # 75 minutes of the senior's time
with_adr = asked * words / wpm        # 5 ADRs x 2 minutes = 10 minutes of the newcomer's time
print(read_all, round(index, 2), without, with_adr)
# 48.0 1.92 75 10.0

newcomers = 4
print(newcomers * without, newcomers * with_adr)
# 300 40.0  (minutes per year)

Reading the result: reading all 24 ADRs takes about 48 minutes, reading just the list of titles takes almost 2 minutes, so nobody has to load everything. If each newcomer re-asks 5 decisions and a senior explains each for 15 minutes, that is 75 minutes of senior time per person, or 300 minutes a year with 4 newcomers. With ADRs, the newcomer reads the 5 records in about 10 minutes each, so 40 minutes a year.

To be clear: 300 words, 150 words per minute, 5 questions and 15 minutes are my assumptions. ADRs also take time to write and maintain, which is not counted here. Replace them with your own team's numbers.

4. The one-page template#

Picture a submission form: few fields, every field required, small enough to fill in ten minutes. The template below builds on Nygard's version and adds two fields borrowed in spirit from MADR (a condition to revisit and a way to confirm), so the decision has an exit and a check.

# ADR 0007: Send reminders through a queue

Status: accepted (proposed / accepted / superseded by ADR 00NN)
Date: 2026-03-12
Decided by: a name or a role

## Context
The situation and the constraints, stated as fact, no judgment. Who is waiting for what?

## Decision
We will (one or two full sentences, active voice)

## Options rejected
- Option A: why it was not chosen.
- Option B: why it was not chosen.

## Consequences
- Gain: the good things.
- Loss: what we accept to give up.
- Neutral: what only changes, neither good nor bad.

## When to revisit
A concrete condition that makes us reopen the decision.

## How to confirm
An automated check or an observable sign that the decision is still followed.

Three rules to stay on one page:

1. One ADR, one decision. If the title needs the word "and", it may be two decisions.
2. Full sentences in context and decision. Bullets are for short lists such as options and consequences.
3. Long detail goes elsewhere. Comparison tables, charts, experiment results: link to them, do not paste them in.

5. A worked example: a filled-in ADR#

This is an illustration, made up for this post, and does not mention a real company or product. The situation: a small app helps service shops book appointments and sends reminder emails. Every number in the example is invented.

# ADR 0007: Send reminders through a queue, not while handling the request

Status: accepted
Date: 2026-03-12

## Context
When a customer books, the app sends a confirmation email while handling the request.
The email service sometimes takes several seconds to respond and once failed for an hour.
During that time the booking page hangs, and some customers have clicked book twice.

## Decision
We will put email sending into a job queue. The booking request only writes the
appointment to the database and replies at once. A separate worker sends the email
and retries up to three times.

## Options rejected
- Send directly in the request: the simplest, but the page hangs when the service is slow.
- Scan the table every minute with a scheduled job: no queue needed, but emails are
  up to a minute late and the job scans a large table.

## Consequences
- Gain: the booking page responds fast, and an email failure no longer breaks booking.
- Loss: one more component to monitor, and emails may arrive a few seconds late.
- Neutral: the logs gain one more source to read when debugging.

## When to revisit
If for six months fewer than 50 emails a day are sent, or the email provider
offers a built-in queue.

## How to confirm
A test that makes sure the booking request does not call the email service directly.

The Y-statement version of the same decision: "In the context of booking appointments, facing an email service that is slow and fails often, we decided for putting the sending into a queue and against sending directly or scanning the table on a schedule, to keep the booking page fast, accepting one more component to monitor."

The full version is about 230 words, within the one-page target of about 300 words. The point to notice: the "Options rejected" section is what keeps people and AI from drifting back to the simplest option, because the simplest option is the one everyone thinks of first.

6. Write ADRs so that an AI agent can read and use them#

Picture leaving a note for the next person on call: the note only helps if it is stuck where that person will look.

1. Keep the ADRs in the repository, for example in a docs/adr folder, next to the code and its change history.
2. Have an index file that lists each ADR's title, status and path, one line each. The agent reads the index and opens the relevant record, so it does not load everything.
3. Point to it from the agent's instruction file. In AGENTS.md or the tool's instruction file, add a few lines like the block below.
4. Put the status on the first line, in fixed words (proposed, accepted, superseded), so both people and machines can filter.
5. Let a person write the first two parts. Context and decision are filled in by a person. AI can suggest rejected options or draft, but a reason that sounds plausible is not necessarily the real reason. A person confirms the reasons, in the spirit of the post Make AI Output Prove Itself.

## Existing decisions
- Before changing an approach already covered in docs/adr, read the index
  docs/adr/README.md and open the relevant ADR.
- If your change goes against an accepted ADR, stop and ask,
  do not edit it yourself.
- When you report done, say which ADRs you read.

Three lines, short enough for a person and an agent to read in full. The last line borrows the fourth way from the evidence post: separate what was read from what was assumed.

7. The life of an ADR#

The key point: do not delete the old record. Readers need to see both what we once decided and why we changed.

8. When not to write an ADR#

This is my own rule of thumb, not a standard from Nygard or MADR. Picture the kitchen bin: not everything needs its own notebook.

• Do not write one when the decision can be reversed in about an hour of work (renaming a variable, choosing a file name).
• Do not write one when the reason is already in the tool's standard documentation and anyone can look it up.
• Do not write one just to complete a procedure. An ADR nobody reads costs effort and helps nobody.
• Do write one when a newcomer is likely to ask "why is it like this?", when reversing the decision takes more than half a day, or when we just rejected an option that everyone will think of first.

9. Where ADRs fail too#

Picture the recipe book: some are thick and nobody opens them, some have been wrong for years.

9.1 Illustration: a morning with and without an ADR#

This is an illustration, not a real case, and it names no person or product. It reuses the example from section 5.

Without an ADR:

• 09:00. A dev asks the agent: "Tidy up the booking code, remove redundant parts."
• 09:20. The agent sees the email queue as an extra layer and changes it to a direct call. All existing tests pass.
• 11:00. The dev reads the diff, finds it reasonable and merges.
• 15:30. The email service is slow. The booking page hangs and a customer books twice.

With an ADR and AGENTS.md pointing to it:

• 09:00. The same request.
• 09:10. The agent reads the ADR index, opens ADR 0007 and says: "This queue is an accepted decision. I will not change it. Do you want to open a new ADR to replace it?"
• 09:20. The dev answers: "Keep it. Only tidy the other parts."
• 09:50. The agent submits the change and notes "read ADR 0007, did not touch the queue".

What to do in both cases: stop at the point where the agent wants to remove something that looks redundant and ask whether it had a reason. This is an illustration, so it proves nothing. The signal from the Stack Overflow survey (section 3) and Nygard's account only show that missing context is real and that a short record may help.

10. The numbers and documents used#

Read it correctly: the sources above support recording short reasons. I did not find a controlled study that measures whether ADRs reduce errors or help AI agents avoid reversing decisions. The AI parts of this post are a synthesis of practice.

11. Habits to apply from tomorrow#

11.1 Eight questions before finalizing an ADR#

1. Is this one decision, or two glued together?
2. Would a reader a year from now understand the context without asking anyone?
3. Does the decision start with "We will" and use full sentences?
4. Are the rejected options and the reasons for rejecting them written down?
5. Do the consequences include the losses and neutral effects, not just the gains?
6. Is there a concrete condition to revisit and a way to confirm?
7. Are the status and date on the first line, and is the index updated?
8. Does AGENTS.md or the README point to this ADR?

11.2 Three things to do right away#

1. Create a docs/adr folder and an empty index.
2. Pick one recent decision that newcomers keep asking about and write the first ADR in about 15 minutes.
3. Add the three lines from section 6 to your AI agent's instruction file.

12. A balanced view: what ADRs do and do not do#

What they do: keep the reasons, help newcomers find answers themselves, give an agent a place to check before editing, and force the team to name the loss in each decision.

What they do not do:

• An ADR does not guarantee the decision is right. It only records what we thought at the time.
• An ADR that is not kept current can do more harm than no ADR, because it states the wrong thing as in force.
• An agent is not obliged to read ADRs, and I have not seen a test that measures how well it follows them. Check by asking the agent to report which ADRs it read.
• ADRs take effort to write and maintain. For a very small team or a short project, one sentence in the README may be enough.
• A reason drafted by AI can sound fluent without being true. A person needs to confirm it.

Do not oversell it: an ADR does not replace communication in the team. Do not dismiss it: the cost of a lost decision usually arrives at the end, when someone reverses it because nobody knew it once had a reason.

13. Conclusion#

Code shows what we did, an ADR shows why. One short page, written at decision time, including the options we dropped and what we accept to lose, with a path from where people and AI will read, is enough to keep many decisions from being reversed just because nobody knew they had a reason. Start with a decision that newcomers keep asking about.

References#

The numbers come from public sources as of October 2026. Some figures were read through a summary, so open the original sources to check before quoting.

• Michael Nygard, Documenting Architecture Decisions (Cognitect, 15 November 2011)
• Architectural Decision Records, adr.github.io and MADR (version 4.0.0)
• Olaf Zimmermann, Y-Statements (the one-sentence ADR pattern)
• Stack Overflow, 2026 Developer Survey (read through a summary, tables not cross-checked)

For anyone who uses an AI agent to write code or do multi-step work, beginners included. No knowledge of how models work is needed. Every term comes with an everyday picture, and the calculation is marked so you can skip it if you do not need it.

Numbers come from public sources listed at the end. Examples labeled "illustration" are constructed and name no real person or company. How a model uses long input internally is not fully understood by anyone, so the pictures below are memory aids only. Command names come from the Claude Code documentation at the time of reading; other tools and later versions may differ.

Quick summary#

• An agent only "remembers" what sits inside its context window: your request, the files it read, command output, project instructions. The window has a limit, and everything you open takes up room.
• Several studies found models use long input less reliably than short input. Liu et al. (2023) found results are often best when the needed information is at the start or end and drop when it sits in the middle. A Chroma report (July 2025) tested 18 models and found reliability fell as input grew longer. The authors themselves say their tasks are simpler than real work.
• When a conversation is nearly full, the tool may compact it into a summary. The Claude Code documentation says the summary keeps your requests and key code, but details of early instructions can be lost.
• In this post's worked example, a long session fills about 91.5% of the window, and about 43% of that is finished, old work. Starting over with a short handoff brings usage down to about 8%. These are illustration numbers for you to recompute.
• Lasting rules belong in the project instruction file, not in the conversation. Switching to unrelated work: clear the conversation. Same task but long: compact with guidance.
• Starting over has a cost too: the agent has to reread files, and the handoff may be incomplete or wrong. The "when" part of this post is a rule of thumb, not a measured result.

Hard words, plain meaning#

1. A picture first: a desk of fixed size#

Picture working at a desk that is not very big. You spread out your boss's request, a few files, the results of some lookups, your notes. At first the desk is empty and you see everything. Two hours later the desk is full of paper and the sheet you need is under three others. You can still work, but slower and with more mistakes. If it gets completely full, you copy the important points onto a fresh sheet, clear the desk and carry on. The copy is sure to be missing a few details.

An agent works much the same way. The Claude Code documentation describes the context window as holding conversation history, file contents, command output, instruction files, automatic memory, loaded skills and system instructions. As you work, the window fills. This is only a picture to help you remember: inside the model it is certainly more complicated than a desk.

Takeaway: "the agent forgot" is often not laziness or stupidity but limited room, and what occupies the room is not always what you need. Managing the desk is your job.

2. What takes up room, and what is left after compaction#

2.1 What takes up room#

According to the Claude Code documentation, before you type anything some things are already on the desk: instruction files, automatic memory, the names of MCP tools and skill descriptions. After that every file read, every command output and every exchange adds more. The /context command shows usage by category with suggestions for reducing it.

2.2 What is left after compaction#

When nearly full, the tool compacts the conversation into a summary. The table below is a digest of the "What survives compaction" section of the documentation, in the version I read. Details may change between versions.

Two takeaways from the table: if a rule must survive, put it in the project-root instruction file, not in speech. And if you want control over what is kept, compact on purpose with guidance, for example /compact focus on the auth bug fix, instead of letting the tool guess what matters.

Takeaway: lasting rules belong on the wall poster, not in the pile of paper on the desk. See Writing AGENTS.md for how to write that poster.

3. Why long conversations make agents worse#

Picture reading a 300 page file and then being asked about a detail on page 150. You may manage, but it is much harder than when you were handed just one page. Research on language models shows something similar, though why nobody has explained completely.

3.1 Liu et al. (2023): information in the middle gets missed#

The authors (including Percy Liang) tested answering questions from many documents and retrieving a value from a list of keys and values. According to the arXiv abstract, performance is often highest when the relevant information is at the beginning or end of the input and drops significantly when the model must use information from the middle of a long input, even for models advertised as supporting long contexts. This is an experiment with 2023 models. The abstract page I read does not name the individual models, so newer models may behave differently.

3.2 Chroma (July 2025): 18 models, reliability drops as input grows#

Chroma's technical report (Hong, Troynikov, Huber) tested 18 models, including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3 families, on three kinds of tasks: an extended "needle in a haystack" test, question answering over long conversations (LongMemEval, 306 prompts averaging about 113,000 tokens) and repeating a string of words. The authors write that performance grows increasingly unreliable as input length grows. They also found distracting content makes results worse, even when there is a single distractor, and that models react differently: according to them, Claude models tended to abstain when things were ambiguous, while GPT models tended to answer wrongly more often with distractors present.

Read this alongside the authors' own words: their tasks are simpler than real work, so they expect the effect to be worse in practice, but it is also hard to separate the effect of length from task difficulty, and the report does not explain the mechanism. It is a company's technical report, and I have not seen an independent review.

3.3 Anthropic (September 2025): a finite "attention budget"#

Anthropic's post on context engineering (29 September 2025) uses the term context rot and describes context as a finite resource, where every added token uses up part of the model's "attention budget". It proposes a few directions: compact when nearly full, keep notes outside the window to read back later, and use subagents with their own desks. This is the technical view of a company that sells tools, not a controlled experiment.

Takeaway: the three sources point the same way: longer does not usually mean better. How much, and why, is uncertain, and it differs by model.

4. A calculation you can recheck (illustration, skippable)#

This is an illustration with constructed numbers, not measurements. Suppose a 200,000 token window. A session consists of: 12,000 tokens loaded at the start, 25 file reads of 3,000 tokens each, 40 command outputs of 1,500 tokens each and 60 exchanges of 600 tokens each.

window = 200_000
startup = 12_000
files, file_size = 25, 3_000
cmds, cmd_size = 40, 1_500
turns, turn_size = 60, 600

used = startup + files * file_size + cmds * cmd_size + turns * turn_size
print(used, round(used / window * 100, 1))
# 183000 91.5

# suppose half of the work is an old, finished task
old = 0.5 * (used - startup)
print(old, round(old / window * 100, 1))
# 85500.0 42.8

# start over with a 4,000 token handoff
fresh = startup + 4_000
print(fresh, round(fresh / window * 100, 1), (window - fresh) // file_size)
# 16000 8.0 61   (how many more 3,000 token files fit)

Reading the result: the long session uses about 91.5% of the window, and about 43% of it is old work. Only about 5 more files fit (17,000 divided by 3,000). After starting over with a 4,000 token handoff, usage is about 8% and about 61 more files fit.

Note clearly: all the token sizes above are assumptions, including the assumption that half the session is old work. Window size, the automatic compaction threshold and how tokens are counted depend on the model and the tool. The calculation does not say 8% gives better results than 91.5%, only how much room old work takes. Run /context in your real session to see real numbers.

5. When to do what: clear, compact, split or start over#

Picture the desk: sometimes you only need to push some paper aside, sometimes copy notes onto a fresh sheet, sometimes move to a different desk.

The "basis" column shows which parts are in the documentation and which are personal experience. For specific thresholds such as "two corrections" or "more than half the window" I have not seen a study that measures them, so treat them as a starting point to adjust yourself.

5.1 Signs it may be time to start over#

1. You switch to a task unrelated to most of what is on the desk.
2. The agent breaks a rule you stated and you have to remind it a second time.
3. The agent keeps editing the same spot or proposes again something already rejected.
4. You no longer remember what the conversation settled (the agent will find it harder too).
5. After an automatic compaction, quality visibly changes.
6. /context shows most of the room is command output and files from finished work.

6. A handoff template (copyable)#

Picture shift notes: the next person reads for a minute and can continue. Paste this block at the start of the new session, or save it as a file in the project to read again later.

## Handoff
Goal: <one sentence>
Done and checked: <what, with the command run and result>
In progress: <file, function, next step>
Decided and why: <decision, one line each>
Tried and failed: <approach, why dropped>
Do not: <boundaries, files not to touch>
Unchecked or assumed: <what the agent guessed rather than ran>
Command to rerun the check: <one command>

The "unchecked or assumed" part comes from Make AI Output Prove Itself: separate what was run from what was only guessed, so the new session does not treat a guess as fact. You can ask the agent for a draft and then read and fix it yourself, because a summary written by AI can also be incomplete or wrong.

A guided compaction command, for when you stay on the same task:

/compact focus on: current goal, decisions made, files being edited,
the check command, and what must not be touched

7. Illustration: one morning, two ways of working#

This is an illustration, naming no real person or product.

Without clearing the conversation:

• 09:00. A dev asks the agent to fix a rounding bug in invoices and says: "New function names must start with calc."
• 09:40. The agent has read 25 files and run many commands. The bug is fixed, tests pass.
• 10:30. In the same session, the dev asks for a button colour change on another page.
• 10:45. The agent adds a function named computeColor, against the rule given at 09:00. The dev reminds it and the agent fixes it, but later proposes again an approach that was rejected at 09:20.

With clearing and a handoff:

• 09:40. Fix done, the dev commits and saves a short handoff to the project's notes file. The calc rule is already in the root instruction file.
• 10:30. The dev runs /clear and opens the new task with a clear request.
• 10:45. The agent reads the instruction file and names the function correctly the first time.

What to do in both: lasting rules go in the instruction file, finished work gets committed and then the desk is cleared. This is an illustration, so it proves nothing. The evidence is the research in section 3, and it has limits too: older models, tasks simpler than real work. See also Seven Mistakes When Writing Requests for Agents.

8. Where following this still breaks#

9. Numbers and documents used#

Read it correctly: all three research sources hint at "longer is often less reliable". I did not find a controlled study that measures directly how much "clearing the conversation" or "writing a handoff" improves results in real work. The thresholds in section 5 are rules of thumb.

10. Habits to apply from tomorrow#

10.1 Eight questions mid-session#

1. Is the current work still related to most of what is on the desk?
2. Are the required rules already in the root instruction file, or only spoken?
3. What does /context show taking up the most room?
4. Has finished work been committed and checked?
5. Do I need the agent to read many files only for lookup, and should a subagent do it?
6. If I compact, what do I want kept (have I written it as guidance)?
7. If I start over, does my handoff have an "unchecked or assumed" part?
8. Am I starting over because of a clear sign, or out of habit?

10.2 Three things to do now#

1. Run /context in your current session and note the three biggest items.
2. Move the three rules you keep repeating into a short block in the root instruction file.
3. Save the handoff template from section 6 as a file and try it on one task this week.

11. A balanced view: what starting over does and does not do#

It does: reduce how much old work takes up room, avoid repeating what was already rejected, make you write down what was decided (useful for later readers too) and give important rules a durable home in the instruction file.

It does not:

• It does not guarantee better results. A new session loses details the handoff did not record, and the agent has to reread files, costing effort and tokens.
• I have no measurements from real work for the "when" thresholds. Try them and compare.
• Tools differ in how they compact, reload and set thresholds, and this changes between versions. The documentation you read today may not be right in a few months.
• A summary written by AI can also be wrong.

Do not overstate: this is not a cure for every case of agent mistakes, many come from vague requests rather than a full window. Do not dismiss it: a conversation that runs on with nobody looking at the desk is where small errors pile up, and clearing is cheap compared with guessing what the agent forgot.

12. Conclusion#

An agent does not have unlimited memory: it works on a desk of fixed size, and research suggests a cluttered desk often means less reliable results. Put lasting rules in the instruction file, compact with guidance when you stay on the same task, clear when you change tasks, and when you start over bring a short handoff that you have read yourself.

References#

Numbers and descriptions come from public sources as of October 2026. Command names and tool behavior change by version, so open the originals to check before quoting.

• Liu et al., Lost in the Middle: How Language Models Use Long Contexts (arXiv, July 2023, version 3 on 20 November 2023)
• Hong, Troynikov, Huber (Chroma), Context Rot: How Increasing Input Tokens Impacts LLM Performance (14 July 2025)
• Anthropic, Effective context engineering for AI agents (29 September 2025)
• Anthropic, Claude Code documentation: How Claude Code works and Explore the context window

For anyone who uses AI to write code, look things up or hand work to an AI agent, including beginners. No background is needed. Every term comes with an everyday example, and the calculation part is marked so you can skip it if you do not need it.

The numbers come from public sources, listed at the end. Examples labeled "illustration" are made up, and they do not name any real person or company. Nobody fully understands how AI works inside, so the pictures below are only aids for remembering. The five ways in this article are a synthesis of practice drawn from the documents and studies cited, not five techniques that have each been tested in a controlled experiment.

Quick summary#

• Most people who use AI are already used to checking its work. The 2026 Stack Overflow survey found that 48% of respondents trust AI only when they can easily verify the result, and only 6.6% trust it for important decisions.
• Checking by "reading carefully and thinking it through again" is the most expensive way. If the AI hands in evidence you can open, checking moves from "rebuild the whole argument" to "look at one specific thing".
• The five ways to ask for evidence: sources you can open, a runnable check with its raw output, small, separate changes, a sort into ran, only read and assumed, and a way for someone else to rerun it.
• Each has its own weak spot: sources can be invented, a test can be written by the AI to pass, output can be cut off. The article shows how to catch each one.
• In this article's worked example, checking a summary with 8 claims takes about 48 minutes without evidence and about 18 minutes with it. These are illustration numbers so you can recompute with your own team.
• Evidence is not a guarantee. It makes checking cheaper, but a person who understands enough still has to look at it.

Hard words, plain meaning#

1. Picture it first: the repair slip#

Picture taking your motorbike to a shop. The mechanic says, "All done, runs great." You have no way to check except to ride it and hope. Now picture the mechanic handing you a slip: front brake pads replaced, brakes tested at 30 km/h on the slope behind the shop, stopping distance normal, brake fluid system not inspected. Same mechanic, but now you know what they did, what they tried and what they have not touched.

AI works the same way. "Done, all tests pass" gives you almost nothing to check. It only gives you something to believe. The Claude Code documentation describes this exactly: an agent stops when the work looks done, and if it has nothing to run, then "looks done" is the only thing it has, so you become the test suite. The documentation also recommends asking for concrete evidence such as the command that was run and its output.

Takeaway: saying "done" costs nothing. Evidence is what makes checking cheap. Your job is not to decide whether to trust, but to design the request so the result carries its own way of being checked.

2. Why it matters: users are doing the checking#

2.1 Two numbers from the 2026 Stack Overflow survey#

According to the results page of the Stack Overflow Developer Survey 2026 (more than 30,000 responses, per one news report), about 48% of respondents said they trust AI when they can easily verify the result, and 6.6% trust AI for many tasks, including important decisions. The survey also records that 69.9% ask an AI agent when they face an unfamiliar task. So people use AI a lot, but trust it with conditions.

Two cautions. First, I read a summary rather than checking each table, and one number on the page ("over 87% have some level of trust, up from 31%") sounds out of line with the 6.6% figure, so I do not use it. Second, on citing sources, the sources I read give 78.7% (one news report) and 93% (a summary of the survey page) for the share who consider citing sources important when judging an AI answer. The two numbers may differ because of how they were counted or how the question was asked, so read it as "most people, roughly 80% to 93%".

Takeaway: users have said what they need: results that are easy to check. What is missing is a way to make them easy to check.

2.2 A real case: court cases that did not exist (Mata v. Avianca, 2023)#

In a lawsuit in a federal court in New York, the plaintiff's lawyer used ChatGPT to draft a legal filing. The filing cited cases with airline names, quotations and docket numbers all filled in, but six of them did not exist. On June 22, 2023 the court fined two lawyers and their law firm a total of 5,000 USD. More detail is in the article Why not to trust AI blindly.

What stands out is this: the filing had "sources", but nobody opened them. Evidence is only useful when someone opens it.

Takeaway: asking for sources is step one, and opening the sources is step two. Without step two, step one is decoration.

2.3 Why AI so often answers with confidence: an argument worth knowing#

Kalai, Nachum, Vempala and Zhang (OpenAI and Georgia Tech, September 4, 2025) argue that the way models are trained and scored often rewards guessing over saying "I am not sure". Their example: asked for someone's birthday, a guess has about a 1 in 365 chance of being right, while answering "I don't know" always scores 0, so guessing pays. In the accompanying post, OpenAI reports a comparison on the SimpleQA benchmark: one model declined to answer 52% of the questions and was wrong on 26%, while an older model declined 1% and was wrong on 75%.

Read this as the authors' own argument and the company's own measurement; I did not find independent verification in my search. It gives a practical hint: if a system tends to answer even when unsure, then ask it to state what it is unsure about, instead of waiting for it to say so by itself.

Takeaway: "I am not sure" often does not show up on its own. Ask for it in the request.

3. Five ways to ask for evidence#

To make this easy to picture, think of each way as one box on the repair slip: where it read, what it tried, what it changed, what it has not done, how I can try it again myself.

Way 1. Ask for sources you can open#

Picture it as: a book with footnotes. A footnote is only worth something if you can flip to that page.

A sample line to put in the request:

For every claim about code or documentation, attach a place I can check:
file path and line number, or a link to the official documentation.
If a claim has no source, write "no source".

How to read the result: pick one or two of the most important claims, open the source and see whether it really says that. A link that opens is not enough, the content has to match.

Weak spots: an invented source (like the court cases above), a real source that says something else, an outdated source. How to catch them: open it, and look at the date.

Way 2. Ask for a runnable check, with its raw output#

Picture it as: squeezing the brake to test it. The result shows "stops" or "does not stop", and nobody has to comment.

A sample line:

Before reporting done, run the following check and paste the command
and its output word for word (do not summarize): npm test
If any test fails, fix it and run it again.
State how many tests ran.

The Claude Code documentation lists things that can serve as the check: a test file, the exit code of a build, a linter, a script that compares output with a sample result, or a screenshot placed next to the design. What matters is that the check returns pass or fail and the AI can read the result.

Weak spots: a test the AI wrote to fit its own code (the same mistake sits in both places), the AI editing the test to match the code, and only part of the tests running. How to catch them: read the diff of the test folder, compare the number of tests run with the number that exist, and break it on purpose: change one line to be wrong yourself and see whether the test turns red. Better ways to write tests are in the article Testing AI-written code.

Way 3. Ask for small changes, one part at a time#

Picture it as: handing in homework one question at a time instead of the whole notebook. The grader can finish during a break.

A sample line:

Work on one thing at a time. Each round, make one change that can be
described in one sentence, at most about 100 changed lines. After each
part, say what the part is for and where I look to see it working.

Google's public code review guide says small changes are reviewed faster and more thoroughly, because a reviewer can find five minutes several times instead of blocking out a 30 minute stretch, and large changes make it easy to miss important points. They give a rough guide: about 100 lines is usually reasonable and 1,000 lines is usually too large. This is a rule of thumb for human review, not a hard number, and it can vary with the number of files.

Weak spots: splitting into small pieces that depend on each other and are hard to follow. How to catch it: make the AI state the order and the reason for the split. When a change really has to be large (a mass rename), ask the AI to point out which part is mechanical and which part has logic.

Way 4. Ask for a sort: ran, only read, or assumed#

Picture it as: a doctor's record saying "examined", "looked at the scan", and "guess based on symptoms". The three are not allowed to mix.

A sample line:

At the end of the answer, sort the results into three groups:
(a) ran and saw the result, with the command;
(b) read the code but did not run it;
(c) assumed or guessed.
If you are unsure about something, put it in (c), do not leave it out.

The reason: as in section 2.3, a system may tend to give a complete-sounding answer, while "I have not run this" does not arrive by itself. A three-group table gives uncertainty a place to live.

Weak spots: the AI files under (a) something it only read. How to catch it: every item in (a) must have the command and the raw output right next to it. An item in (a) with no command counts as (b).

Way 5. Ask for a way for others to rerun it, or for an independent reader#

Picture it as: a recipe. If you repeat the steps exactly and get the same dish, that dish is more believable than the cook's praise.

A sample line:

Give me one command (or a few steps) so I can see the result again
myself on a clean machine. State the versions and environment
variables that are needed.

And if the work matters, ask a fresh session that has not seen the earlier reasoning to reread the change against the original request. The Claude Code documentation describes this with a review command that runs in a new sub-agent that never saw the conversation, so it is not swept along by the reasoning that produced the code. Take it as a reasonable practice, not a guarantee, because another AI can make the same kind of mistake.

Weak spots: the command works on the AI's machine and not on yours (a missing environment variable, a different version). How to catch it: really run it once on a clean machine or in CI. And for important work, have a person read it, not only a machine.

Putting it into one short block#

If you want to use this over the long term, paste the block below into AGENTS.md or the tool's instruction file so every session reads it:

## When reporting that a task is done
- Paste the check command you ran and its raw output, with the number of tests run.
- Give a file and line number for every claim about code.
- One change per purpose, about 100 lines or fewer.
- Sort results into: ran / only read / assumed.
- Give me one command to rerun this on a clean machine.

Five lines, short enough for both people and AI to read all of it. Keeping it short is deliberate: a long list can get skipped.

4. A calculation you can recheck yourself (illustration, can be skipped)#

This is an illustration with made-up numbers, not measurements. Suppose the AI hands in a summary of a change with 8 claims (for example "fixed the date format bug", "did not touch the payment function"). You need to confirm each claim before approving.

• Without evidence: you rebuild each claim yourself, say 6 minutes each.
• With evidence: you only open what is pointed to, say 1 minute each. For the 2 riskiest claims, you check deeper for another 5 minutes each.

claims, rebuild, glance, deep, risky = 8, 6, 1, 5, 2
without = claims * rebuild              # 48 minutes
with_proof = claims * glance + risky * deep   # 8 + 10 = 18 minutes
print(without, with_proof, without - with_proof)
# 48 18 30

per_month = 2 * 20                      # 2 summaries a day, 20 days
print(without * per_month / 60, with_proof * per_month / 60)
# 32.0 12.0   (hours per month)

Reading the result: 48 minutes drops to 18 minutes per summary, about 38% of the original. With two summaries a day over 20 working days, that is about 32 hours versus 12 hours a month for one reviewer.

To be clear: 6 minutes, 1 minute and 5 minutes are numbers I assumed, not measured. Real work can differ a lot, and asking for evidence also costs the AI extra time to run and write. Time yourself a few times with your own team and swap in your numbers.

5. Where asking for evidence also fails#

Picture the repair slip again: a mechanic can still write "tested the brakes" without testing them.

5.1 Illustration: one afternoon with and without evidence#

This is an illustration that does not refer to any real person or product.

Version without asking for evidence:

• 14:00. A dev asks the agent: "Fix the bug where December 31 shows as next year in the report."
• 14:20. The agent replies: "Fixed, tests pass." The dev skims the change, finds it reasonable and merges it into the main branch.
• 16:30. A customer reports that the year-end report is still wrong with a different date format. The agent fixed only one of the two places.

Version with asking for evidence:

• 14:00. The same request, with the five-line block above.
• 14:20. The agent hands in: ran npm test, 12 tests pass, with one new test for December 31 in the dd/MM format. Group (b): read the second date format function but did not run it. Group (c): assumed every report uses the dd/MM format.
• 14:25. The dev looks at groups (b) and (c) and adds a December 31 case for the second format. The test turns red. The agent fixes the remaining place.
• 14:40. Merged with two new tests.

The action in both versions: stop "done" at groups (b) and (c). The place where the AI honestly says "not run" tells the dev where to look. This is an illustration, so it proves nothing. The evidence lies in the studies: Perry and colleagues (Stanford, 2022) found that users of an AI assistant wrote less secure code in most of the tasks but were more confident that their code was secure, so skimming and approving gives real cause for worry. See also the article Reviewing AI-written code.

6. Numbers and documents used#

Reading it correctly: the sources above support making results easy to check. I did not find a controlled study that directly measures how many errors asking for evidence removes. The five ways above are a synthesis of practice.

7. Level of evidence by risk#

You do not need all five things for every task. Picture shopping: for vegetables you look, for a watch you want the warranty card.

8. Habits to apply from tomorrow#

8.1 Eight questions before accepting a result#

1. What way of checking does this result carry: a source, a check, or only words?
2. Have I opened at least one important source, and does it say what is claimed?
3. Are the check command and its output word for word, or only a summary?
4. How many tests ran out of the total that exist?
5. Who wrote the new test, and does it turn red when I break one line on purpose?
6. Is the change small enough for me to read all of it?
7. Which parts did the AI mark "only read" or "assumed", and what will I do about them?
8. Can I rerun it on a clean machine, or have I only taken the AI's word for it?

8.2 Three things to do right away#

1. Paste the five-line block into the instruction file of the tool you use.
2. Pick one small task today and time the checking before and after asking for evidence.
3. Ask a colleague to read one "ran, only read, assumed" report and tell you whether it is usable.

9. A balanced view: what evidence can and cannot do#

What it can do: make checking cheaper and faster, force the AI to say what it has not run, leave a trail for others to check again, and help spot gaps between the request and the change.

What it cannot do:

• Evidence is not a guarantee of correctness. A test can be wrong, a source can be invented, and a neat table can create too much of a sense of safety. The study by Perry and colleagues suggests users can become more complacent when they have an AI assistant.
• Asking for evidence costs extra time to run and write, so for low-risk work it is too much.
• I did not find a controlled experiment measuring the effect of each way separately. Measure it with your own team.
• Another AI rereading the work can make the same kind of mistake. For important work, a person is needed.

Do not exaggerate: this is not a formula that makes AI stop being wrong. Do not downplay it either: the cost of accepting "done" with nothing to check usually shows up at the end, when the error has already reached the customer.

10. Conclusion#

An AI's "done" sounds like an answer, but it is really a promise. Evidence turns the promise into something you can check: sources you can open, a check you can run, small changes, three groups of ran, only read and assumed, and a command to rerun it. Put these requests into the very prompt you give the AI, and always be the one who opens the evidence.

References#

The numbers come from public sources as of October 2026. Some figures are reported differently by different outlets, so open the original source to check before quoting.

• Stack Overflow, 2026 Developer Survey (read through a summary, each table not cross-checked)
• Anthropic, Best practices for Claude Code (section "Give Claude a check to run")
• Kalai, Nachum, Vempala, Zhang, Why Language Models Hallucinate (9/2025) and OpenAI's post
• Google Engineering Practices, Small CLs
• Mata v. Avianca, Inc.: Wikipedia and LawNext on the sanctions
• Pearce et al., Asleep at the Keyboard? (2021) and Perry et al., Do Users Write More Insecure Code with AI Assistants? (Stanford, 2022)

For students, newcomers and anyone looking for a first job in software. No statistics background needed. Every hard word has a short explanation.

This is a reflection: I weigh evidence and say what is unknown, it is not a research report. I read the PDFs of two papers (Stanford, Harvard) and the original pages of SignalFire, Ramp, Yale and Indeed (through FRED). I read the rest through summaries or news coverage and I say so each time. Open the original before quoting any figure.

Quick summary#

• Many articles say AI is narrowing the door into the profession, many others say nothing visible yet or the reverse. Most are right about what they measure. The trouble is that they measure different things.
• What the large studies share: no job loss across the whole economy, but employment of 22 to 25 year olds in AI-exposed jobs is weakening (Stanford, Harvard). The Stanford update (data to June 2026) gives 19%, and the authors themselves call it an early descriptive indicator, not causal proof.
• The same group from the same source still gives several numbers: 6%, 13%, 19%, 20%. They differ in baseline and method, not because someone is lying.
• Some explanations have nothing to do with AI: interest rates rising from 2022, the US tax rule on research and development costs (Section 174), and over-hiring in 2021 to 2022. Researchers tried to strip them out, but nobody has stripped them out completely.
• Before believing a figure, ask six questions: who was counted, what was counted, compared with when, where the data comes from, what else changed at the same time, how sure it is.
• Whichever side is right, a newcomer can still measure their own market, widen the doors, build real proof of skill, and re-check the numbers every quarter.

Hard words, plain meaning#

1. Picture it: five people weighing the same crowd#

Picture it as an everyday scene. This is my own illustration, not real data.

Five people stand in front of the same crowd at a train station gate and each tries to answer: "Is this crowd getting smaller?"

• The first person counts only children. He sees fewer.
• The second counts everyone. She sees almost no change.
• The third stands at the entrance and counts newcomers. She sees a sharp drop.
• The fourth reads the notice board saying "more people needed". He sees the board emptying.
• The fifth asks each person: "Was it hard to get in?" She hears many say yes.

All five are honest and all five are right about what they measured. If you only hear "the crowd is shrinking" you cannot tell who said what. The story of entry-level jobs in software is the same, with one more twist: people also pick a different starting year and a different place to stand.

2. The side saying "the door is narrowing"#

I sort the studies by what they measure. The last column is how sure I am that I read the study correctly, not how sure I am of its conclusion.

Two things to take from the table:

1. The large studies do not say AI is erasing jobs across the economy. The Stanford paper itself says it finds no widespread job displacement in the economy as a whole. They describe a narrow group: young people in occupations where AI can replace part of the tasks.
2. The same group and source still give several numbers. For 22 to 25 year olds in highly exposed jobs, the ADP Research page gives a fall of about 6% (late 2022 to Jul 2025, against themselves), the Stanford paper gives 13% and later 19% (against another group), and software developers alone are about 20% (against a peak). Section 4 recomputes these comparisons so you can see why they differ.

3. The side saying "not yet" or "the opposite"#

A few cautions so you do not misread:

• Ramp says "entry-level headcount up 12%", and that figure is not misreported: it is the growth in the number of entry-level people at heavy AI spenders over two years. But Ramp itself notes these firms were already larger, more engineering-heavy, more venture-backed and faster-growing beforehand. The result is a correlation among those firms, says nothing about the whole economy, and the paper is not final.
• Harvard and Ramp really do disagree about newcomers, even though both look at firms using AI. According to a note in the Harvard paper, both use Revelio résumé data. The difference is how "firm using AI" is defined: one means posting a role to integrate AI, the other means spending money on AI tools. Same data vendor, different definition, different result.
• The New York Fed by-major unemployment figures are very noisy because of small samples. The 2023 data also sits before most of the period the studies above cover. Later updates are reported by the press at about 7.0% and 7.8%, but I could not open the original table to confirm, so I do not use them as the main figure.

4. Why one market gives different numbers#

Picture it as a toy town. The numbers below are an illustration I built and computed by hand. They do not measure a real market.

The toy town has 100 positions for newcomers. In 2022: 40 positions are in occupations where AI can replace part of the work (group A), and 60 are in other occupations (group B). In 2025, group A has 32 and group B has grown to 66.

Same data, three answers: "down 20%", "27% lower", "almost unchanged". Nobody is wrong. That is also how the Stanford paper gets its 19%: employment in the two most exposed groups of occupations fell about 11% from Nov 2022 to Jun 2026, while the three least exposed groups rose about 10%, and 0.89 / 1.10 - 1 gives about 19% below where it would be had it kept pace. If you only look at "down 11%" you get a different number from the same data.

Besides the method, four more things make figures differ:

Change the baseline, change the story: real job-posting data#

Indeed Hiring Lab's series of software development job postings in the US (index, Feb 2020 = 100, seasonally adjusted) is published on FRED. I recomputed the comparisons from three points in the series.

One series, three sentences: "postings fell by two thirds", "still about one fifth below pre-pandemic", "up 32% from the bottom". And note that in 2022 postings were about 2.3 times the pre-pandemic level. Part of the "decline" is coming back from an unusual peak, not everything is new.

5. Explanations that have nothing to do with AI#

It can be thought of as a market that went through several changes at once, with AI only one of them.

The Stanford paper also states its own weak points: the gap shrinks when education is controlled for, some divergent trends already appeared before generative AI, and it is more pronounced in ADP data than in national surveys. They stress that these are early descriptive indicators, not causal estimates. The literature review in the Harvard paper reports that some other studies find no systematic difference between more and less exposed occupations, for example Chandar (2025) and the Financial Times (2025) on CPS data, and a Swedish study finding a smaller fall of about 5.5%. I read these through Harvard's review, not through each original paper.

6. Six questions before believing a figure#

Picture it as six flashlights. You shine them one by one on the number, and where it stays dark, do not rush to believe it.

Try it. Illustration, a headline I made up, not a real article: "Jobs for new devs fall 20%". Six questions: new devs in which country? People employed or hires? Down from which year, a peak or before the pandemic? Payroll data or postings? Were interest rates, tax and over-hiring considered? Is there a comparison group? If the article cannot answer three or more, treat the number as a rumor with a digit, not yet as data.

7. Vietnam and Japan: what I found#

• Vietnam. I found no government data specifically on new-graduate developers, and I could not open any public report from the large job platforms. The closest is a ManpowerGroup survey for Q4 2026, as reported by Tuổi Trẻ (9 Sep 2026): of 285 employers surveyed, 57% said they had increased hiring of entry-level workers compared with 2025, and 77% in the information sector. Read it with care: it is a self-reported employer survey by a staffing company, the article does not say whether the 285 are all in Vietnam, does not define "entry-level" and does not separate out developers. Confidence: low to medium.
• Japan. I found no primary data on entry-level developers in Japan tied to AI. Articles often cite a forecast from the Ministry of Economy, Trade and Industry (METI) that Japan could be short of roughly 160,000 to 790,000 IT workers by 2030 depending on the scenario. I only read it through summaries because the original page refused access, and it is a forecast of the total made before generative AI spread, so it does not answer the question about newcomers and AI. Confidence: low. Open the original before quoting.

Plainly: most figures in this post are American. Vietnam and Japan may differ, and I have no evidence to say how.

8. What a newcomer can do whichever side is right#

Picture it as not knowing whether tomorrow is rain or sun. If you carry both an umbrella and sunglasses, you do not need to guess right. The suggestions below are my own reasoning, not tested on many people.

One more thing: a national average is not a verdict on one person. A single rejection usually has specific reasons, such as budget or a better-fitting candidate, not a statistic. If you want the long view, Dev career through the eras puts this in the history of the profession, and When AI writes code faster, is programming still a creative craft? covers the work that is hard to replace.

9. What is not known#

• Cause. Nobody has separated AI from interest rates, tax, over-hiring and the post-pandemic correction. The Stanford and Harvard authors each say so in their own way.
• Slower hiring or no hiring. If it is only later hiring, newcomers still get in after a few years. If a rung of the ladder has gone, that is a different story. Current data cannot tell them apart.
• Software specifically. The best figure for developers aged 22 to 25 is the ADP 20% figure and small surveys. The CPS has very few people in this group per month.
• Outside the US. I have no reliable evidence for Vietnam or Japan.
• Figures change by version. The Stanford number moved from 13% to 16% to 19% as data was added and the method changed. This post will age, so check the date.
• My own limits. I read two papers directly as PDFs and the original pages named above, and the rest through summaries. I have no raw data to re-check, so my calculations only rework published numbers.

10. A balanced view#

• This post does not tell you to worry or not to worry. The data shows one narrow area (young people in jobs AI can partly replace) weakening, and does not show an economy-wide problem.
• "Not seen yet" is not "not there". The Yale Budget Lab says it has not seen disruption, not that none will come. In the summary I read, they think AI is likely to leave a mark later.
• "Seen" is not "caused by AI". Every figure about newcomers above is a correlation.
• What can be measured is easy to turn into a story. "AI took your job" is neater than "several things changed at once and nobody has separated them". Neater is not truer.

11. Conclusion#

When two articles give opposite numbers, they are usually weighing two different things on two different scales. Ask six questions: who, what, since when, which data, what else changed, how sure. For a newcomer, doing what helps in every scenario matters more than guessing the right side: measure your own market, widen the doors, build real proof of skill, and re-check every quarter.

Start this week: open a spreadsheet, record 20 postings where you want to apply, and do not change your plan because of one headline.

References#

Open the original before quoting any figure. Where I only read a summary or press coverage, it is stated above.

• Brynjolfsson, Chandar, Chen: Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of AI (Stanford Digital Economy Lab; the version I read is dated August 2026, data to June 2026)
• ADP Research: Yes, AI is affecting employment, here's the data (the 6% and 20% figures for the youngest software developers, Jul 2025)
• Hosseini, Lichtinger: Generative AI as Seniority-Biased Technological Change (SSRN 5425555; version of 6 May 2026)
• SignalFire: State of Tech Talent Report 2025
• Ramp Economics Lab: Companies hire more after AI adoption (30 Jun 2026)
• Federal Reserve Bank of New York: The Labor Market for Recent College Graduates
• Economic Innovation Group: A viral chart on recent graduate unemployment is misleading (re-reading the 6.1% and 7.5% by-major figures, ACS 2023)
• Yale Budget Lab: Evaluating the Impact of AI on the Labor Market: Current State of Affairs and Fortune: coverage of the Feb 2026 update and "AI washing"
• FRED: Software Development Job Postings on Indeed in the United States (Indeed Hiring Lab)
• Baker Tilly: Domestic research expenditures, Section 174 under the 2025 tax act (an accounting firm's summary, not the statute)
• Tuổi Trẻ News: Vietnam among region's strongest hiring markets, yet employers struggle to find skilled workers (9 Sep 2026, ManpowerGroup survey)
• Related posts in this blog: Fresher in the AI era: a 12-month plan, Interviews in the AI era: a two-week plan, Who is accountable when AI is wrong?, Dev career through the eras

For anyone learning a new library or language, especially students and freshers. No background in research or statistics is needed, and every technical term has a short explanation.

The evidence on this topic is small and recent, mostly small, short studies. The four-week plan is my suggestion and nobody has tested it. Treat it as an experiment you run on yourself.

Quick summary#

• How you use AI may matter more than whether you use it. In a small Anthropic experiment, the group that handed the work to the AI understood less, while the group that used AI to ask and understand kept its score.
• Faster does not always mean learning more. The AI group was only a few minutes faster, and that difference was not enough to conclude anything.
• A feeling is not a measurement. In another study (METR), people using AI took 19% longer yet believed they were faster.
• A four-week plan: AI-on sessions, AI-off sessions and a 10-point self-test every Friday.
• Four rules: predict before you ask, ask for explanation and not only code, rewrite from memory, keep AI-off sessions.
• Sometimes letting AI go first is fine. The article says when.

Hard words, plain meaning#

1. Picture it: watching someone cook and following a GPS#

Picture two ways to learn cooking. In the first, you stand and watch a chef finish a whole dish. You get to eat it, but the next day you may not be able to cook it again. In the second, you cook it yourself while a helper stands next to you, points out the mistake and explains why. The dish may come out slower, but your hands remember.

Directions work the same way. Following a GPS gets you there fast, but switch it off and you are lost. Learning the map is slower at first, and then you can find your way even with no signal.

To picture it, think of AI as both the chef and the GPS: it can do things for you, which is both its strength and its risk. The question of this article is: when should you let it do the work, and when should you switch it off so your own hands remember?

One general note: nobody fully understands the inside of AI, and research on learning with AI is only a few years old. The pictures in this article are only aids.

2. What the evidence says#

The figures come from Anthropic's research page and the arXiv paper. The two round slightly differently (one gives 50% and 67%, the other a gap of 4.15 points out of 27), so I write "about". The AI group finished a few minutes faster, but the summaries give slightly different numbers and none is statistically significant.

Six ways of using AI in Anthropic's experiment#

The researchers reviewed screen recordings and sorted the AI users into six patterns:

The first three learned less, the last three kept learning. What the last three share is that you still do your own thinking: asking to understand, or handling errors yourself. But each group has only 2 to 7 people and this is a descriptive analysis, so read it as a hint about usage, not a law.

Why "predict first" and "rewrite from memory" make sense#

Teaching guidance from MIT's Teaching and Learning Lab says that actively retrieving information improves long-term retention compared with re-reading, citing Roediger and Karpicke (2006). The same page says that guessing wrong and then seeing an explained answer improves learning, and suggests telling learners that such effort is a "desirable difficulty" (Bjork 1994). A University of Arizona guide adds that questions asking learners to predict or compare help more than questions that only ask for definitions.

This is evidence from classrooms in general. Nobody has tested this exact plan with AI. I use it as a reasonable basis for designing the rules, not as proof that the plan works.

3. Illustration: the same task, two ways to learn#

This is an illustration with made-up numbers, not real people.

A learner is writing a data-loading step with a new library.

Way A, hand it all over:

Way B, predict first, then ask why:

Way B is slower by 25 minus 10, which is 15 minutes. In return, at 11:30 the learner can make the change alone. The times here are made up for illustration, not measurements.

What supports this example is the evidence in section 2, not the example itself: the delegation groups scored lower in Anthropic's experiment.

4. Four rules#

1. Predict before you ask. Write one line: "I think the result will be X". Then run it or ask, and compare. A wrong guess is fine, since the gap is exactly what you are about to learn.
2. Ask for explanation, not just code. Ask why this piece is written this way, when it would fail, and whether there is another way. In Anthropic's study this is close to the group that asked for code with explanations.
3. Rewrite it from memory. Once you understand, close the AI tab and the docs and retype it. Mark where you got stuck, reopen, then type it again.
4. Keep AI-off sessions. Each week has a block with only official docs and the machine's error messages. That is when you check whether your hands still remember.

Two side rules: when you receive code from AI, review it like a stranger's code, and remember that AI can be wrong in a very confident voice.

5. The four-week plan#

I assume 4 sessions of 60 minutes a week (Monday to Thursday) and a 30-minute self-test on Friday (15 minutes for the test, 15 minutes to review). Pick one library or language so you can measure.

The arithmetic#

• 4 sessions a week, 60 minutes each, over 4 weeks: 4 times 4 times 60 equals 960 minutes of study.
• Fridays: 4 times 30 equals 120 minutes. Whole plan: 960 plus 120 equals 1,080 minutes, which is 18 hours.
• AI-off minutes: weeks 1 and 4 have 180 each, weeks 2 and 3 have 120 each, total 600. Add 120 minutes of Fridays (no AI) to get 720 minutes.
• AI-off share: 720 divided by 1,080 equals 66.7%. AI-on minutes are 360.

This ratio is my choice, and no study says what the right number is. If you are under deadline pressure, keep at least the week 1 sessions and the Friday test.

6. The Friday self-test#

No AI, no notes, 15 minutes, 10 points:

Then open the docs, mark it again and write the misses in your notebook. The misses are next week's homework.

A made-up example of one learner: week 1 scores 4 out of 10 (40%), week 2 scores 5 (50%), week 3 scores 7 (70%), week 4 scores 8 (80%). From week 1 to week 4 that is 4 more points, or 40 percentage points. These numbers are invented, and a self-marked test has error. Watch your own trend, not other people.

One way to compare yourself: each week pick two concepts you learned with AI on and two you learned with AI off, and see which you remember better on Friday. Four items is a tiny sample, so use it only to learn about yourself.

7. When letting AI go first is fine#

Anthropic notes that an earlier study of theirs found AI raised productivity on tasks people had already mastered, so the two results may not conflict.

8. A balanced view#

• The experiment lasted only about an hour. It is unknown whether the gap narrows or stays after months.
• Small sample, one library, one quiz. The authors state these limits too, and another experiment might give a different result.
• Run by an AI vendor. The results are somewhat unfavorable to using AI, but you should still read the original and compare it with independent sources.
• The tool in the experiment was a chat. Agentic tools that write whole projects may behave differently, and the authors say so.
• No comparison with learning from a person. It is unclear whether AI or a good mentor is better.
• The PNAS math study used school students. The results may not carry over to adults learning to code.
• The Microsoft and CMU survey is self-reported and only a correlation.
• This plan has not been tried on many people. If it feels too slow or too strict, adjust the AI-off ratio, as long as the Friday test stays so you can measure.

9. Conclusion#

The question is not "should I use AI to learn", but how to use it so my own hands still remember. The evidence so far, though thin, points one way: people who keep thinking, asking why and retyping keep their learning. Predict first, ask for explanation, rewrite from memory, keep AI-off sessions, and measure with the Friday test.

Start today: pick one library, write your first prediction in a notebook, and book your first Friday test.

If you lack the basics, see the 12-month plan for freshers. If an interview is coming, see the two-week plan. To hear two people talk about this, see the dialogue between a fresher and a senior.

References#

Open the originals before quoting, because figures may be updated.

• Shen and Tamkin, "How AI Impacts Skill Formation", arXiv 2601.20245 (January 2026)
• Anthropic, research page "How AI assistance impacts the formation of coding skills"
• Bastani and colleagues, "Generative AI without guardrails can harm learning", PNAS 2025
• Wharton, summary of the Bastani and colleagues study
• METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity" (July 2025)
• Lee and colleagues, "The Impact of Generative AI on Critical Thinking", CHI 2025 (authors' PDF)
• MIT Teaching + Learning Lab, "Help Students Retain, Organize and Integrate Knowledge"
• Roediger and Karpicke, "Test-enhanced learning", Psychological Science (2006)
• University of Arizona, "L2L Strategy: Retrieval Practice"
• Related posts on this blog: A Fresher in the AI Era: a 12-month plan, Interviewing in the AI Era: a two-week plan, A fresher and a senior in dialogue, Reviewing AI-written code, Why you should not trust AI blindly

For anyone who uses AI to write code or hands tasks to an AI agent, beginners included. No security background needed. Every term comes with an everyday picture, and the calculation is marked so you can skip it.

Numbers come from public sources listed at the end. Examples marked "illustration" are constructed and name no real person or company. Why models invent package names is not fully understood by anyone, so the pictures below are memory aids, not a description of the machine.

Quick summary#

• When writing code, AI sometimes suggests a library that sounds perfectly plausible but does not exist. A large study at USENIX Security 2025 measured this at an average of at least 5.2% for commercial models and 21.7% for open-source models, in 2024 experiments.
• If the same fake name keeps being suggested, an attacker can register it first on a public registry such as npm or PyPI and put malware inside. This attack is called slopsquatting.
• In that study, when 500 prompts that had produced a fake name were rerun, about 43% of the fake names showed up in all 10 runs. Invented names are not fully random, enough for an attacker to guess ahead.
• Real experiments exist: the empty package huggingface-cli got over 30,000 downloads, and the name react-codeshift spread to more than 200 repositories. Researchers registered both, and neither contained malware.
• The risk is real, but how big it is remains unclear: I did not find a confirmed slopsquatting attack that caused major damage. What worries more is an agent that runs install commands with nobody watching.
• Defense needs no expensive tool: pin versions with a lockfile, glance at a new package before installing, turn off install scripts, make agents ask for approval and run them in isolation.

Hard words, plain meaning#

1. The picture first: the clerk and the copycat shop#

Picture asking a very eager shop assistant: "Do you have a phone case for this model?" She answers at once: "Yes, the Zorbo Pro, we stock it." No product has that name. She is not lying. She is simply good at producing names that sound right.

Now picture a scammer who notices she gives that same name to many customers. He opens a shop next door, hangs a sign saying "Zorbo Pro" and sells something harmful. The next customer she sends there walks straight into his shop.

That is slopsquatting. The assistant is the AI, the product is a library name, the shop is a registry like npm, and npm install is stepping through the door. Unlike a mistyped name, the user did nothing wrong. They followed a suggestion.

The term is credited to Seth Larson, security developer-in-residence at the Python Software Foundation, in April 2025, according to Wikipedia and to Andrew Nesbitt, who helped it spread. I could not find Larson's own original post to check, so treat this as reported, not quoted.

Takeaway: the danger is not that AI is wrong. It is that the wrong answer can be predicted and someone is ready to occupy it.

2. Two real experiments and one easily confused incident#

2.1 The empty huggingface-cli package (Lasso Security, 2023 to 2024)#

Bar Lanyado, a researcher at Lasso Security, noticed AI models repeatedly suggesting an install command for a PyPI package called huggingface-cli. It does not exist, and the correct install is huggingface_hub[cli]. He registered an empty package under that name to see whether anyone would install it.

According to press coverage, the package received over 30,000 downloads in about three months. Outlets report different figures, from about 15,000 to 35,000, so read it as tens of thousands. The Register also reported that the fake name appeared in the install instructions of an Alibaba research repository, which was corrected after he reported it.

A caution: downloads do not prove anyone was harmed. It is unclear how many came from AI-generated code, how many from people copying instructions, and how many from automated scanners.

Takeaway: a repeated invented name gets installed by real people, and it can even creep into other people's documentation.

2.2 react-codeshift spreads to over 200 repositories (Aikido Security, 2025 to 2026)#

According to Aikido Security, a commit on 17 October 2025 to a public repository added 47 LLM-generated "skills" (instruction files for agents). At least two of them contained the command npx react-codeshift. The name looks like a blend of two real packages, jscodeshift and react-codemod. The package react-codeshift had never existed.

Those files were copied and forked widely. According to Aikido, over 200 repositories referenced the command in December 2025, and by 14 January 2026, when researcher Charlie Eriksen registered the name, the count was 237 or more. A GitHub code search then returned about 215 matching files. Read it as "over 200", since counting methods differ. Right after registration the package got about 74 downloads on day one, then one to four a day. The author suggests this trickle hints at real agents trying the command. That is their inference, not a verified result. The package was kept as a safe placeholder and contained no malware.

Takeaway: an AI-made mistake does not stay put. It gets copied, forked and translated until it becomes a "standard" that other agents read and follow.

2.3 The May 2026 npm worm: dangerous, but not slopsquatting#

A Cloud Security Alliance (CSA) research note dated 17 May 2026 describes an npm worm linked to a group called TeamPCP: 373 malicious package versions across 169 npm packages and two PyPI packages. The note says the worm also wrote configuration into the files of AI coding tools, and that some versions carried valid-looking provenance. This is a secondary note relaying other people's reporting, so open the original before quoting. CSA itself says some of the "first" claims belong to the researchers it cites.

The point to keep: this is about real packages being taken over, not names invented by AI. It belongs here for two reasons. First, it shows registries are real targets and an agent that runs install commands is an easy link to exploit. Second, it shows provenance is not an absolute guarantee. The habits in section 6 work against both kinds of incident.

Takeaway: do not fixate on the word slopsquatting. The job is to control what gets installed on your machine, whether the name was invented by AI or a real package was poisoned.

3. The mechanism, step by step#

Picture a chain of four links. One thing first: why models invent package names is not understood in full. A common way to think about it is that the model keeps trying to write a name that looks like a real library, and real names are often assembled from familiar words such as react, codemod and cli. That is a picture, not a proven conclusion.

1. You ask. You (or an agent) need a feature, say reading a CSV file, and ask AI to write the code with an install command.
2. The AI suggests a name. Sometimes that name is not on the registry. According to the study, only about 13.4% of fake names were one or two characters away from a real name, so this is mostly not a typo problem.
3. An attacker registers it. Public registries let anyone claim a free name quickly. If a fake name repeats, an attacker can ask popular models many times to see which names recur and claim those.
4. The install succeeds. At first the command fails with "not found", which is harmless. Once the name is taken, the same command works, and install scripts can run on the installer's machine or CI server.

3.1 A calculation you can recompute (illustration, optional)#

This is an illustration with made-up usage, using two real numbers from the study as anchors: 5.2% (commercial models) and 43% (fake names repeated in all 10 runs). Suppose a team of 10 developers each asks AI for install commands 5 times per working day, over 20 working days.

devs, per_day, days = 10, 5, 20
suggestions = devs * per_day * days          # 1000
fake_rate = 0.052                            # low anchor, commercial models
repeat_rate = 0.43                           # fake names repeated in all 10 runs
fake = suggestions * fake_rate               # 52
predictable = fake * repeat_rate             # 22.36
check_minutes = fake * 2                     # 2 minutes to vet each package
print(suggestions, round(fake), round(predictable), check_minutes, check_minutes / devs)
# 1000 52 22 104.0 10.4

Reading it: out of 1,000 suggestions, about 52 are fake names, and roughly 22 of those are the repeating kind an attacker could predict. If one were claimed and nobody checked, a single npm install is enough. On the other side, vetting each new package in 2 minutes costs about 104 minutes a month for the whole team, around 10 minutes per person. Checking is far cheaper than one wrong install.

Note: the 43% was measured on 500 prompts that had produced a fake name before, not on all prompts, and 5.2% is an average per suggested package name, not per user. Your team's numbers may differ a lot, so treat this as a method, not a forecast.

4. Where it slips in daily work#

Picture a building site: a good builder can still pick up the wrong material if nobody checks the store.

4.1 Illustration: a morning with an agent#

This is an illustration with no real person or product. The package name is made up.

• 10:00. A developer asks an agent: "Merge the CSV files in this folder and clean the date column."
• 10:03. The agent writes code using a package csv-tidy-helper-x and runs the install. It is in auto-approve mode, so the command runs.
• 10:04. The name never existed before, but someone registered it last week. The install succeeds and the install script runs.
• 11:30. The code works, the developer opens a PR. The lockfile gains 14 lines and nobody reads them.

What to do: make the agent stop at the install command and ask. The developer looks the name up: created last week, 3 downloads, no source repository. The signs are clear enough to refuse. And if the install had run inside a sandbox, the damage would have been limited. Section 6 lists these layers.

An illustration is not evidence. Evidence is in the studies below. Separately, a Stanford study (Perry et al., 2022) found that people using AI assistants wrote less secure code while being more confident it was secure, so worrying about unchecked install commands has a basis. See also Reviewing AI-Written Code and Why You Should Not Trust AI Blindly.

5. Data from studies and incidents#

How to read it: the table shows two different things. There is good evidence that AI invents package names, and that invented names can be registered and installed by people. There is no good evidence yet about how much real damage this causes.

6. Habits to apply tomorrow#

Picture a kitchen with a receiving routine: a new ingredient needs a receipt and someone's signature, not just anyone dropping things in the pantry.

6.1 Eight steps#

1. Pin versions with a lockfile and use npm ci instead of npm install on CI and clean machines. According to npm documentation, npm ci installs exactly what the lockfile says and fails if the lockfile does not match package.json.
2. Vet a new package before installing: creation date, download count, a real and matching source repository, the maintainer, the version history. A package created last week with a name that fits your need perfectly is a sign to stop.
3. Turn off install scripts by default with --ignore-scripts or ignore-scripts=true in .npmrc. Enable them only for packages you have decided to trust.
4. Use an allowlist or a private registry mirror: only approved packages can be installed, and unknown names are simply not found.
5. Check provenance with npm audit signatures when a package has it. Remember it tells you the origin, not that the package is clean.
6. Require a human to approve every install command an agent wants to run. Do not enable auto-approve for installs or for running unfamiliar code.
7. Run agents in a sandbox or devcontainer, and do not put real keys, deploy tokens or production write access inside.
8. Read the lockfile diff in every PR: is there a new package, what is it called, which registry does it come from. This sits in the checklist of Reviewing AI-Written Code.

6.2 Four commands worth remembering#

# 1. Install exactly what the lockfile says, without running install scripts
npm ci --ignore-scripts

# 2. Look at a package before installing: creation date, maintainer, repository
npm view some-package time.created time.modified maintainers repository.url
curl -s https://api.npmjs.org/downloads/point/last-week/some-package

# 3. Check signatures and provenance of what is installed
npm audit signatures

# 4. After an agent's change: any new resolved lines in the lockfile?
git diff package-lock.json | grep '^+.*"resolved"'

For Python, the similar command is pip install --require-hashes -r requirements.txt, but it needs a requirements file with hashes for every package. Syntax differs between tools, so check the documentation of the one you use.

6.3 Ten questions before you install#

1. Does this package name exist on the registry, and did I check on the registry page or from memory?
2. When was it created? Under a few weeks old deserves suspicion.
3. How many downloads per week, compared with what a package like this should have?
4. Is there a public source repository that matches the package name?
5. Who maintains it, and do they have a history?
6. Does the package run a script at install time?
7. Is there a more popular package that does the same job?
8. Is this my install command, or an unapproved one from an agent?
9. If this package were malicious, what could it reach on this machine?
10. After installing, how many lines did the lockfile gain, and have I read them?

7. A balanced view: what is known and what is not#

Fairly solid: models do invent package names, the measured rate ranges from a few percent to over twenty percent in the experiments, fake names can repeat, invented names can be registered, and real people do install them.

Unclear:

• The invention rate of newer models and tools. Many tools now search the web or check package names, which may lower the rate. I have not seen independent measurements per tool.
• Real-world damage. I did not find a confirmed slopsquatting attack that caused major damage, but not having found one does not mean none has happened.
• The experiments were run by researchers who registered harmless packages. A real attacker would not announce it.

Do not exaggerate. The bigger supply chain incidents, like the May 2026 npm worm, come from real packages being taken over. If you defend only against slopsquatting you miss most of the risk. The habits in section 6 cover both.

Do not dismiss it either. Prevention is cheap (about 10 minutes per person per month in the example above), while an agent that installs unchecked packages can reach access keys and source code. The OWASP Top 10 for AI Apps places supply chain risk on a wider map. If your agent calls tools through MCP, read What Is MCP? too, because every tool is also something you are installing.

8. Conclusion#

A name invented by AI is harmless until someone registers it and a machine installs it on its own. Most of the risk sits in "nobody looked". Give install commands a checkpoint: pin versions, look up new packages, turn off scripts, make agents ask for approval and read the lockfile.

References#

Figures come from public sources as of October 2026. Some numbers are reported differently by different outlets, and the secondary notes are for orientation only, so open the originals before quoting.

• Spracklen et al., We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs (USENIX Security 2025)
• Lasso Security, AI Package Hallucinations and The Register (the huggingface-cli experiment)
• Aikido Security, post on react-codeshift and hallucinated npx commands in skill files (2026)
• Origin of the term (reported, original post not found): Wikipedia, Slopsquatting and Andrew Nesbitt, Slopsquatting meets dependency confusion
• Cloud Security Alliance, Mini Shai-Hulud: npm Worm Targets AI Developer Supply Chain (secondary note, 17 May 2026)
• npm Docs, Generating provenance statements and the npm ci command
• Pearce et al., Asleep at the Keyboard? (2021) and Perry et al., Do Users Write More Insecure Code with AI Assistants? (Stanford, 2022)

For programmers, product people and team leads bringing AI into their work. Hard terms get a short explanation.

The author is not a lawyer and this is not legal advice. The two cases are retold from public sources (a law firm's summary, news, an open encyclopedia), not from the original rulings. The "responsibility map" and the five things at the end are my opinion, not rules.

Quick summary#

• Two real cases show that "the AI did it, not me" helps nobody.
• The person who uses AI output and the party who puts AI in front of customers both have something to check.
• Responsibility does not find an owner on its own. Put a name on it before an incident.
• Five things a team can do now: one owner, checkpoints where risk is high, logs, a path to a person, blameless write-ups.

Hard words, plain meaning#

1. Case one: an airline's chatbot got it wrong#

According to a summary by the law firm Manatt, a passenger named Moffatt asked the chatbot on Air Canada's site about a bereavement fare. The chatbot said he could buy a regular ticket and apply for a refund of the difference within 90 days. But the airline's real policy did not allow refunds after travel. When he applied, the airline refused.

The airline argued, per that summary, that the chatbot was a separate legal entity and the airline was not responsible for its words. The civil resolution tribunal in British Columbia (Moffatt v. Air Canada, 2024 BCCRT 149) rejected the argument, finding that the airline owed a duty of care and had made a negligent misrepresentation. The airline had to refund the difference, a figure given as 650.88 Canadian dollars.

What is worth thinking about: the sum is small, but the principle is not. A customer has no reason to know which page of a website is "right" and which is "the chatbot talking nonsense".

2. Case two: lawyers filed case law an AI invented#

In Mata v. Avianca (a federal court in New York, 2023), the plaintiff's lawyers filed a brief citing several cases. When the other side could not find them, it turned out ChatGPT had invented them and the lawyers had not checked. On 22 June 2023 the judge fined them 5,000 dollars in total and remarked that they had "abandoned their responsibilities" (per Wikipedia, Bloomberg Law and Above the Law).

What I take from this case: a tool does not sign, the user does. The duty to verify does not disappear because a draft was written by a machine.

3. Two cases, one pattern#

Both lacked a named person holding the duty to verify at the last step.

4. A responsibility map (my opinion)#

When AI is wrong there are usually four parties, each with a part:

I do not know what the "correct" share is before the law in any given place. This table is only a way of dividing work, so a team does not have everyone assuming someone else has it.

5. Five things a team can do now#

1. Name one owner for each AI feature. A name, not "the team". That person knows what the feature does, how it is measured and how to switch it off.
2. Put human checkpoints where risk is high. Money, health, legal matters, personal data, anything irreversible. Low-risk places (suggesting titles) can be looser. See Reviewing AI-written code and OWASP Top 10 for AI apps for checking and limiting permissions.
3. Keep logs so you can trace what happened. When a complaint arrives you need to know what the AI said, to whom, when and from what source.
4. Offer a path to a person, and tell customers. When the AI is unsure, or a customer says "I want a person", there must be a way. Say clearly when someone is talking to a machine, and do not let the machine's words sound like a commitment when they are not.
5. Write up incidents without blaming individuals. Ask "how did the system let this happen", not "who did it". The stories of GitLab 2017 and Amazon S3 2017 are examples of telling an incident with a focus on mechanism. Why not to trust AI blindly covers why verification is needed.

6. Common traps#

• "The AI did it, not me." The two cases above show this argument is weak.
• Rubber-stamp review. The checker clicks approve hundreds of times a day without reading. A name on paper is not real checking.
• Blaming the individual at the end of the chain. The person who pressed the last button is often just the last link in a system without controls.
• Pushing everything to the vendor. A contract may limit their liability. Do not assume, read it.
• Logging without protecting. Logs contain what customers said and must be kept as carefully as any personal data.
• Naming an owner but not giving them the power to switch the feature off.

7. A balanced view#

• Only two cases, through secondary sources. I read about them through a summary, news and an open encyclopedia, not the original rulings. Details may be missing.
• Law differs by place. A decision of a tribunal in British Columbia does not automatically apply elsewhere.
• Checking everything is unrealistic. That is why the article speaks of high risk. Where risk is low, little checking is fine.
• The responsibility map is a way of dividing work, not of dividing fault before the law.
• I may be looking at a skewed sample, because both cases are ones the press noticed. Many times AI is wrong and nobody sues.

8. Conclusion#

An AI tool can draft quickly, but it does not sign and it is not fined. Responsibility still falls on the people and the organization that put the result forward. What to do is name responsibility before an incident, put checks where mistakes are costly, and learn from incidents without hunting for someone to blame.

A small step today: pick an AI feature you run and ask "if it tells a customer something wrong tomorrow morning, who finds out first, and who decides to switch it off?" If you cannot answer yet, that is the first job.

References#

• Manatt: AI Gone Wild, Airline Has to Honor a Refund Policy (secondary, a law firm's summary)
• Wikipedia: Mata v. Avianca, Inc. (secondary)
• Related posts in this blog: Reviewing AI-written code, OWASP Top 10 for AI apps, Why not to trust AI blindly, The day an engineer deleted the wrong data: GitLab 2017, One wrong command, four hours: Amazon S3 2017

For product people, team leads and programmers bringing AI into a product. Every technical term gets a short explanation.

The three-layer split and the metrics come from practitioners' guides (blogs and tool documentation), which are secondary sources. All the thresholds and figures in this article are illustrations, not standards. The shop "Hong" example is rebuilt.

Quick summary#

• Usage counts are not success. Measure the outcome of the work.
• Measure three layers at once: quality, cost, trust. A single layer creates blind spots.
• The right cost is the cost per successful outcome, reviewer effort included.
• A high acceptance rate can mean over-trust. Set it beside the pass rate from evals.
• Use a one-page board and decide in advance what happens when a metric worsens.

Hard words, plain meaning#

1. Why not measure one number#

It is easy to start with what is easy to count: number of calls, number of users. But an AI feature can be used heavily and still be bad: answering wrongly, costing a lot, or being quietly ignored by users.

Practitioners' guides (which I read via summaries) agree on two points: counting calls is shallow, what matters is work completed; and metrics should be grouped in layers, because focusing on one creates blind spots. A common example: a model that is 95% right is still useless if users do not trust it.

So I split three layers:

This post connects to Writing evals for an AI feature (the quality layer) and Do not measure productivity by lines of code (why activity metrics mislead).

2. The quality layer#

Quality is not a single "accuracy" number. Draw it from error analysis: group failures into categories and count them, as in the post on evals.

Suggested metrics:

• Pass rate per failure category. For example "misread the intent", "invented information", "tone too stiff". One number per category.
• Count of severe failures. An invented price is very different from a tone slip. Count failures with real consequences separately.
• Overall pass rate, but not as the only number.

Report an "all pass" figure, then drill down into each category. The evals guide also advises grading pass or fail rather than on a scale.

How to track: rerun the evals whenever you change the model, the prompt or the data. If you change the model without rerunning, you do not know how quality moved.

3. The cost layer#

3.1. Cost per call and per successful outcome#

Cost per call is the easy figure. But a feature may call the model cheaply while needing lots of human fixing, or need many calls to get one usable result. So the cost worth comparing is:

Cost per successful outcome = (model cost + infrastructure + effort of people who review or fix) ÷ number of successful outcomes

An illustration (rebuilt numbers): a customer assistant for the shop "Hong", one month.

Looking only at model cost, 20 million over 10,000 calls is 2,000 dong a call, and you would miss the staff effort and the 1,000 failed cases. That is why "cost per successful outcome" reveals more.

3.2. Latency#

Measure latency at p95 (and p99), not just the average. The average hides the slowest cases, which is when users leave. For a chat feature, time to the first word strongly shapes the feeling of speed.

4. The trust layer#

This is the layer most often skipped. Look at user behavior:

Watch the trend, not only the absolute level. Some guides note that a sudden jump in the edit rate signals a quality drop, not personalization.

Context: the Stack Overflow 2025 survey (more than 49,000 respondents) found 46% of users do not trust AI's accuracy and only 3% trust it highly. Your product's users may feel similarly. Trust has to be earned, and has to be measured.

5. The trap: a high acceptance rate can be over-trust#

This is the point I find most important in this article. If users accept 90% of outputs but evals show only 75% are actually right, the 15-point gap is users over-trusting, not a good feature.

How to read it: set the acceptance rate beside the pass rate from evals.

I have no data on what these rates typically are. This is a way of reading the metrics, not a measurement result.

6. A one-page board#

One board of nine metrics, three layers, recorded weekly.

Pair them: every speed or cost metric goes with a quality metric. Lower cost with a falling pass rate is not progress.

7. Decide in advance when a metric worsens#

A board without action rules is decoration. An illustration, with thresholds rebuilt:

Specific thresholds depend on the feature's risk. A feature answering about money needs stricter thresholds than one suggesting titles.

8. Common traps#

• Measuring only usage. Heavy use does not mean good.
• Only thumbs up or down. Few people press it and it skews to the extremes. Combine with behavior (edits, overrides).
• Forgetting reviewer effort in cost. Model cost may be cheap while total cost is not.
• Averages hide the tail. Look at p95 and the worst cases.
• Changing the model without rerunning evals.
• Turning a metric into a target (Goodhart's law): if you reward acceptance rate, people will make it look good without improving quality.

9. A balanced view#

• Secondary sources. The three-layer split comes from practitioners' guides (blogs, tracking-tool documentation), not independent research.
• No standard for thresholds. Every number here is an illustration. I do not know what a "normal" acceptance or edit rate is for each kind of feature.
• Measuring takes effort. For a small experiment, a trimmed one-page board (quality, cost, one trust metric) is enough.
• Metrics do not replace listening to users. Read a few real exchanges each week.
• The shop "Hong" example is rebuilt, with figures I made up to illustrate the formula.

10. Conclusion#

A good AI feature is not the one called most, but the one that gives right results, at a reasonable cost, and is trusted appropriately. Measure three layers at once, compute cost per successful outcome, and read the acceptance rate beside the pass rate.

A small step today: pick an AI feature you run, compute its cost per successful outcome (remember to add the effort of fixing), and compare it with the figure you normally use.

References#

• Antoine Buteau: AI product metrics, economics and support burden (secondary, a practitioner's guide)
• PostHog: Product metrics to track for LLM apps (secondary)
• Hamel Husain and Shreya Shankar: LLM Evals, Everything You Need to Know
• Stack Overflow Developer Survey 2025, AI section
• Related posts in this blog: Writing evals for an AI feature, Do not measure productivity by lines of code, Vibe coding a prototype in one day

This is a true story, but not my own. I retell it from the post-event summary AWS itself published (linked at the end, which I read directly). The summary does not name the person who ran the command, so I do not name them and do not guess what they thought.

This post differs from the GitLab 2017 one in one way: it is not a story about broken backups but about a tool that lets one mistyped input remove far too much. Times in the post are Pacific time (PST), as in the summary.

Quick summary#

• On 28 February 2017, at 9:37 AM, an authorized member of the S3 team ran a command to remove a small number of servers from a billing-related subsystem. One mistyped input caused more servers than intended to be removed.
• The removed servers supported two of S3's core systems: the index and placement. Both had to be fully restarted.
• S3 in US-EAST-1 returned to normal only at 1:54 PM, about four hours seventeen minutes later.
• Worth noting: the command did what it was built to do; nothing stopped it from removing too much. Even AWS's own service health dashboard depended on S3.
• AWS changed the tool to remove more slowly and never remove below a minimum.

Hard words, plain meaning#

1. An ordinary operational task#

That morning the S3 team was debugging a slow billing process in US-EAST-1. An authorized team member followed an existing playbook: remove a small number of servers from a billing-related subsystem to see whether it would help.

There was nothing odd about it. Removing servers from a subsystem is everyday work for an operator, and there was a playbook for it. That is an important point: this was an allowed action, following a procedure, by a person with the right to do it.

2. 9:37 AM: one mistyped input#

According to the summary, the command ran at 9:37, and one of its inputs was entered incorrectly. The result: more servers than intended were removed.

The summary does not say which input, how it was wrong or why. I do not guess. What can be said with confidence is that the tool accepted the value and carried it out, with no checkpoint asking "how many will this remove, and is that safe".

The removed servers belonged not to the billing system but to two core systems:

• The index: holds the metadata and location information for every object in the region. Needed for get, list, put and delete requests.
• Placement: allocates storage for new objects on put requests. Depends on the index working properly.

Removing that much capacity meant both had to be fully restarted.

3. The restart: something not done for years#

This is the detail I find most worth thinking about in the summary. AWS wrote that it had not completely restarted these systems in its larger regions for many years. In that time S3 had grown enormously, and the safety checks that validate metadata integrity, needed for a safe restart, made it take longer than expected.

In other words: the ability to recover had never been tested at today's scale. It echoes GitLab's lesson: a backup never test-restored is unknown, and a restart procedure never run at real size does not have a known duration.

4. The timeline#

All times Pacific, 28 February 2017, region US-EAST-1.

From 9:37 to 1:54 PM is about 4 hours 17 minutes. The summary does not give the durations of each restart on its own.

5. Who was affected, and an ironic detail about the dashboard#

According to the summary, affected services included the S3 console, launching new virtual machines (EC2), EBS volumes needing data from snapshots stored in S3, and AWS Lambda. Some services built up a backlog during the disruption and needed extra time to recover.

The ironic detail: the admin console of AWS's service health dashboard also depended on S3. Until 11:37 it could not update individual service status, so AWS had to use its Twitter account and banner text to communicate.

A general lesson (my inference): the tool that reports incidents should not live on the very thing that is broken.

6. What turned one wrong command into a big incident#

From the summary, I see five layers that together turned one mistyped input into four hours of disruption. This reading is my own.

The common thread: each layer is reasonable on its own, and only when they meet do they become an incident.

7. What AWS fixed#

According to the summary:

• Remove more slowly. The capacity-removal tool was changed to remove at a lower pace.
• A minimum. A safeguard was added to block any removal that would push a subsystem below its minimum required capacity level.
• Audit other tools. AWS is checking its other operational tools for similar safety checks.
• Reduce blast radius and shorten recovery. Partition subsystems into independent cells. Partitioning of the index subsystem, planned for later in the year, was moved up to start immediately.
• A dashboard running across multiple regions.

Overall, AWS did not focus on forbidding anyone from mistyping, but on making a mistyped input unable to cause great harm.

8. Compared with the GitLab story#

The two share the "one wrong command" motif, but the lessons sit in different places:

Together: make the wrong command harder to type, and make its consequence smaller.

9. What I take from it: questions for a tool that can remove#

This part is my suggestion, not AWS's conclusion. If you write or use a tool that can remove, delete or shut down, ask:

1. Is there a minimum? Does the tool block any action that would push the system below a safe level?
2. Is there a rate limit? Removing slowly gives the operator time to notice a mistake.
3. Is there a preview? A list-only mode: "this will remove N machines" before removing.
4. What is the blast radius? At most how much can one wrong command break?
5. Has recovery been tested at real scale? When did you last restart the whole system?
6. Does the incident-reporting tool live independently?
7. Can a second person see the dangerous action?

10. When AI can run tools#

This story happened in 2017, but the motif is very modern: a tool with the power to remove takes a wrong input and nothing stops it. With an AI agent, the wrong input may come from a steered prompt, and the agent does not "feel something is off".

That is excessive agency (LLM06) in the OWASP Top 10 for AI apps: excess functionality, excess permissions, excess autonomy. And the mitigations there match AWS's lessons: limit functions, least privilege, require approval for high-impact actions. The post What is MCP talks about fuses, and S3's minimum is exactly a fuse.

11. A balanced view#

• This rests on one main source. I rely on AWS's own summary, which is written from AWS's point of view.
• The summary omits many details. No name, no specific mistyped value, no separate durations for each restart, so I do not guess.
• The comparison and the "five layers" table are my reading.
• The incident is nearly ten years old. The systems have changed a lot, but the causes (no minimum, a large blast radius, untested recovery, a tool depending on the broken thing) still feel very familiar.
• I do not know how the person who ran the command felt, so the post does not describe it.

12. Conclusion#

One mistyped input, four hours, and part of the internet slowed down. But what AWS chose to fix was not "tell operators not to mistype". They fixed the tool so a wrong command does not remove too much, and the system so the consequence is smaller. That is the way of thinking most worth learning.

A small step today: find a tool you use that can remove or delete, and ask whether it has a minimum and a preview mode. If not, add it to your to-do list.

References#

• AWS: Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region
• Related posts in this blog: The day an engineer deleted the wrong data: the GitLab 2017 story, OWASP Top 10 for AI apps, What is MCP? Explained with a power socket

For people who write or review code with AI involved, including beginners. Every technical term gets a short explanation.

The code example and its numbers are rebuilt to be easy to picture. The "100% coverage, 4% mutation score" figure comes from a summary of a paper (I have not read the full text). The tools named are only examples; I did not run them. The practices are my suggestions.

Quick summary#

• When AI writes both the code and the tests, a test easily confirms the code, not the requirement.
• A green test is not enough. High coverage is not enough either: a suite can cover 100% and catch very few bugs.
• The right question is not "is the test green" but "if the code were wrong, would the test turn red?"
• Seven ways: write from the spec, see red first, break the code yourself, mutation testing, property-based, boundary tables, a real run.
• Start with the cheapest: break the code by hand and watch the tests.

Hard words, plain meaning#