Does AI Really Make You Faster? A Two-Week Experiment You Can Run on Yourself

For anyone who uses AI to write code or to work day to day, beginners included. No statistics background needed. Every term comes with an everyday picture, and the calculation parts are marked so you can skip them.

Numbers come from public sources listed at the end, and I opened the originals for the METR figures used here. Examples marked "illustration" are numbers I made up, not anyone's real measurements. Nobody fully understands how AI works inside, and this post says nothing about that: it only teaches how to measure outcomes from the outside. The two studies cited are other people's, on specific tools and dates, not conclusions about you.

---

Quick summary#

• Feeling "AI makes me faster" and actually being faster can differ. In METR's 2025 randomized trial, 16 experienced developers did 246 real tasks. Tasks with AI took about 19% longer, although the developers had predicted a 24% speedup and afterwards still believed in about 20%.
• METR's early-2026 follow-up (57 developers, over 800 tasks) leaned toward AI helping, but METR itself calls the data "very weak evidence" about the size of the gain, because many developers refused to work without AI.
• You can measure yourself in two weeks: about ten tasks of similar size, randomly assigned with or without AI, timed including rework, then compare medians.
• Three things most easily spoil the measurement: choosing which tasks get AI, forgetting rework time, and peeking at results and changing course.
• Ten tasks prove nothing. They give you a measured hint instead of just a feeling.
• The post includes a CSV template, a Python script that runs, a worked illustration with answers and a checklist. Do not use the result to judge colleagues.

---

Hard words, plain meaning#

---

After this post you can explain#

1. Why feeling faster can differ from measured speed, and why you need a stopwatch.
2. Why you must randomize and include rework time for the comparison to be roughly fair.
3. Why ten tasks are only a hint, and how to read the result without fooling yourself.

---

1. Picture it first: new running shoes#

Picture buying a new pair of running shoes. After the run your feet feel light and you think you were faster. The watch says you were two minutes slower than usual, because it was hotter today and the route was hillier. The feeling is real, the watch is real, but they measure different things.

To learn whether the shoes help, a careful runner does many runs, picks at random which ones get the new pair, times every run and compares. They also count not just "running time" but "how sore my feet were afterwards".

AI is the same. The feeling of "less typing, less thinking" is real. But the time from receiving a task to really being done and not sent back is a different quantity. This post teaches how to start that second stopwatch.

Takeaway: to find out whether something helps you, do not ask your feelings. Run a small, fair comparison.

---

2. What others have measured#

2.1 METR's 2025 trial#

METR is an independent research organization. It ran a randomized trial: 16 experienced open-source developers (about 5 years of prior experience with their own projects) did 246 real tasks such as bug fixes, features and refactors, on large codebases (on average over 22,000 stars and over a million lines of code). Each task was randomly assigned "AI allowed" or "AI not allowed". The AI tools were mainly Cursor Pro with Claude 3.5 and 3.7 Sonnet. The period was February to June 2025, each task averaged about two hours, and developers were paid 150 USD an hour.

Result: tasks with AI took 19% longer. Before the work, the developers predicted AI would make them 24% faster. After finishing, they still believed AI had made them about 20% faster. Economics and machine learning experts who were asked predicted roughly 39% and 38% speedups.

METR lists its own limits: only 16 people so it may not generalize, possible sampling bias (people who expected large gains may not have signed up), learning effects beyond about 50 hours of tool use cannot be ruled out, part of the time was self-reported, and it covers only work on familiar codebases. The METR page now also labels these results as out of date.

Takeaway: 19% does not mean "AI slows you down". It is one experiment on one group, tools and date. What is worth learning is the gap between feeling and measurement.

2.2 The early-2026 follow-up#

On 24 February 2026 METR posted "We are Changing our Developer Productivity Experiment Design". They reran with 57 developers (10 returning and 47 new), 143 repositories and over 800 tasks. According to the post, returning developers were about 18% faster (confidence interval from 38% faster to 9% slower) and new developers about 4% faster (from 15% faster to 9% slower). Both intervals include zero.

More important: METR says the data is "very weak evidence" about the size of the gain, because more and more developers declined to take part since they did not want to work without AI (this likely biases the speedup downward), the pay cut from 150 to 50 USD an hour may have added selection effects, and timing became unreliable when developers ran several agents at once. METR still considers it likely that developers are more sped up by AI than in early 2025, and is redesigning the experiment.

Takeaway: even people who measure for a living have to change their method when tools and habits change. Someone else's number does not replace yours.

---

3. Designing the two-week experiment#

Picture a small kitchen experiment: the same dish cooked two ways, tasted blind.

3.1 Eight steps#

1. Pick about ten tasks of similar size. For example each expected to take 30 to 90 minutes. Ten tasks fit two work weeks, roughly one a day.
2. Write your prediction first. Record "I think AI will make me this many percent faster". METR used this step and it is the one most often skipped.
3. Define "done" before you start. For example: tests pass, you re-read the code, you ran it. The same definition for both groups.
4. Randomize. Flip a coin or use the script below. About half the tasks get AI, half do not. Never switch groups after seeing the task.
5. Time from start to "done" as defined in step 3. Record real start and end times, not estimates.
6. Record rework too. If a task is sent back, or you realize something is missing, add the fixing time to that same task.
7. Do not peek at results midway. Peeking makes you want to change your method or choose tasks.
8. Analyze once you have ten tasks. Compare the medians of both groups, then compare with the prediction from step 2.

3.2 CSV template#

One task per row. minutes is the time until done by your definition, rework_minutes is the fixing time afterwards:

task_id,arm,minutes,rework_minutes,done_checks
T1,no_ai,70,5,pass
T2,ai,35,20,pass

Also keep elsewhere: a description of the task, your prediction, your impression afterwards (how many percent faster or slower) and odd notes. Write the impression before you open the table of numbers.

3.3 Script for randomizing and analysis (Python 3, nothing to install)#

The plan command randomizes tasks. The analyze command computes medians, the difference, an exact permutation test and a 95% bootstrap interval:

#!/usr/bin/env python3
"""Self-experiment helper: randomize which tasks use AI, then compare the times.

  python3 ai_experiment.py plan 10 [seed]    : prints a CSV with the arm of each task
  python3 ai_experiment.py analyze data.csv [--no-rework] : medians, difference, exact permutation test, bootstrap interval

Needs only the Python standard library.
"""
import csv, itertools, random, statistics, sys


def plan(n, seed=None):
    rng = random.Random(seed)
    arms = ["ai"] * (n // 2) + ["no_ai"] * (n - n // 2)
    rng.shuffle(arms)
    print("task_id,arm")
    for i, a in enumerate(arms, 1):
        print(f"T{i},{a}")


def load(path, rework=True):
    rows = list(csv.DictReader(open(path)))
    cost = lambda r: float(r["minutes"]) + (float(r["rework_minutes"]) if rework else 0)
    ai = [cost(r) for r in rows if r["arm"] == "ai"]
    no = [cost(r) for r in rows if r["arm"] == "no_ai"]
    return ai, no


def diff(a, b):
    return statistics.median(a) - statistics.median(b)


def analyze(path, rework=True, seed=1, boots=10000):
    ai, no = load(path, rework)
    obs = diff(ai, no)
    # exact permutation test: every way of choosing which tasks would have been "ai"
    pooled = ai + no
    n, k = len(pooled), len(ai)
    count = total = 0
    for idx in itertools.combinations(range(n), k):
        s = set(idx)
        a = [pooled[i] for i in s]
        b = [pooled[i] for i in range(n) if i not in s]
        total += 1
        if abs(diff(a, b)) >= abs(obs):
            count += 1
    # bootstrap interval for the difference of medians
    rng = random.Random(seed)
    ds = sorted(
        diff([rng.choice(ai) for _ in ai], [rng.choice(no) for _ in no]) for _ in range(boots)
    )
    lo, hi = ds[int(0.025 * boots)], ds[int(0.975 * boots) - 1]
    print(f"tasks with ai: {len(ai)}, without ai: {len(no)}")
    print(f"median minutes with ai:    {statistics.median(ai):.1f}")
    print(f"median minutes without ai: {statistics.median(no):.1f}")
    print(f"difference (ai - no_ai):   {obs:+.1f} min ({obs / statistics.median(no) * 100:+.0f}%)")
    print(f"exact permutation p-value: {count}/{total} = {count / total:.3f}")
    print(f"bootstrap 95% interval:    {lo:+.1f} to {hi:+.1f} min")


if __name__ == "__main__":
    cmd = sys.argv[1]
    if cmd == "plan":
        plan(int(sys.argv[2]), int(sys.argv[3]) if len(sys.argv) > 3 else None)
    elif cmd == "analyze":
        analyze(sys.argv[2], rework="--no-rework" not in sys.argv)

In a real run, do not pass a seed to plan, so that you cannot predict it either. Save the output the moment it runs and never edit it afterwards.

---

4. A worked example (illustration, skippable)#

This is an illustration with numbers I made up, not anyone's real measurements. I ran the planner with seed 7 (only so the example is repeatable) and then invented a time for each task:

$ python3 ai_experiment.py plan 10 7
task_id,arm
T1,no_ai
T2,ai
T3,ai
T4,ai
T5,no_ai
T6,ai
T7,no_ai
T8,no_ai
T9,ai
T10,no_ai

After two weeks, the table sample.csv (invented minutes) looks like this:

4.1 Running the analysis, two ways#

Real output of the script on the table above, counting rework:

$ python3 ai_experiment.py analyze sample.csv
tasks with ai: 5, without ai: 5
median minutes with ai:    55.0
median minutes without ai: 60.0
difference (ai - no_ai):   -5.0 min (-8%)
exact permutation p-value: 252/252 = 1.000
bootstrap 95% interval:    -45.0 to +25.0 min

And forgetting rework (the --no-rework flag):

$ python3 ai_experiment.py analyze sample.csv --no-rework
tasks with ai: 5, without ai: 5
median minutes with ai:    40.0
median minutes without ai: 60.0
difference (ai - no_ai):   -20.0 min (-33%)
exact permutation p-value: 48/252 = 0.190
bootstrap 95% interval:    -50.0 to +10.0 min

4.2 Answers and how to read them#

The honest reading: in this example, AI looks slightly ahead, but ten tasks cannot confirm it, and the initial feeling is bigger than the measurement. More important: if you forget to add rework time, you will believe the prettier 33%.

Note: these numbers are invented. Real work can give a completely different result, including the opposite direction.

---

5. Where the measurement breaks#

Picture timing a run but forgetting to start the watch again after the red light.

5.1 Illustration: one developer's week#

This is an invented scenario with no real person or product.

• Monday, 09:00. The developer runs plan. The first task lands in the AI group. They write the prediction: 30% faster.
• 09:05. Start the timer, ask the AI to write a data-cleaning function.
• 10:20. The AI says it is done, tests pass. The developer stops the clock: 75 minutes.
• 10:45. On re-reading, they find the empty-date case is missing. Fixing takes another 25 minutes. They enter it in the rework column rather than skipping it.
• Friday, 17:30. They promise themselves not to look at the totals until the end of next week, even though a few nice numbers are already visible.
• Next Friday, 16:00. All ten tasks are done. They run the analysis, compare it with the prediction and write the conclusion: AI is slightly ahead on tasks with a clear pattern, not ahead on tasks that need business understanding.

What to do: add rework time to the same task, even when it felt "almost done". This is an illustration, so it proves nothing. The evidence is in the research: the METR trial above is one example of feeling and measurement differing.

---

6. Numbers and documents used#

Reading it correctly: the sources show that feeling and measurement can diverge and that measuring AI productivity is hard. They do not say AI is slow or fast for you.

---

7. Checklist: eight questions before trusting your number#

1. Did I write my prediction before starting?
2. Was which tasks get AI decided by a coin or script, and did I save it?
3. Are the tasks of similar size?
4. Is "done" defined the same way for both groups?
5. Did I add all rework time to the right task?
6. Did I peek at results midway and change my method?
7. How wide is the interval, and am I concluding more than it allows?
8. Am I using this only to adjust my own habits, not to judge anyone?

---

8. A balanced view: what it measures, what it does not#

It measures: a hint about which tasks AI seems to help with and which it does not, and the gap between your feeling and the measurement.

It does not measure:

• Ten tasks prove nothing. The interval is wide and the difficulty of individual tasks can skew the result.
• Time is not everything. AI may help you learn, tire you less or make you willing to take harder work, none of which the stopwatch captures. And faster does not mean you understand the code you hand in. See Learning an Unfamiliar Library with AI Without Losing the Skill.
• Today's result may not hold for the tools three months from now.
• Do not use the result to evaluate colleagues. The sample is small, everyone does different work, and a single number is easy to misuse. Read Do Not Measure Productivity by Lines of Code before putting any metric in front of a team.
• If you want to read numbers about people entering the field with care, see Entry-Level Developers: The Numbers Contradict Each Other, So Which Ones Do You Believe?.

Do not overstate it: a personal experiment is not rigorous science. Do not dismiss it: it still beats trusting a feeling, and the cost is a time log for two weeks. If you want to check the quality of what AI hands you, read Make AI Output Prove Itself too.

---

9. Conclusion#

Feeling faster is a feeling. The time from receiving a task to really being done and not sent back is a measurement. The two can differ, as the METR trial showed for one specific group. Run a small experiment: ten tasks, randomized, timed with rework included, compare medians, and read the result with exactly the confidence it deserves.

---

References#

Numbers come from public sources as of October 2026. I opened the METR pages and the arXiv paper and checked the METR figures against them. Open the originals to check before quoting.

• METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (arXiv 2507.09089) and METR blog post, 10 July 2025
• METR, We are Changing our Developer Productivity Experiment Design (24 February 2026)
• Python Software Foundation, documentation of the statistics and itertools modules (for the median and combinations)