---
title: "Secrets in Your Code: Where API Keys Leak and How to Rotate One in 15 Minutes"
url: https://www.stacklogs.io.vn/en/engineering/api-keys-in-code-rotate-in-15-minutes/
language: en
author: "Nhu Duy Thin (Thinnd)"
published: 2026-10-31
updated: 2026-10-31
section: "Engineering"
tags: ["secrets", "API keys", "git", "key rotation", "AI agent", "security", "checklist"]
summary: "Deleting an API key from a file and committing again does not make it disappear: git history keeps the old version, and anyone who copied the repo earlier still has it. When a key leaks, revoke or rotate it first, then clean the history, then read the usage logs. To prevent it, use environment variables, .gitignore, a pre-commit hook and the platform's secret scanning."
license: "Cite with the author name and the canonical URL above."
---

# Secrets in Your Code: Where API Keys Leak and How to Rotate One in 15 Minutes

> For anyone who writes code or asks an AI to write it, beginners included. No security background needed. Every term comes with an everyday picture, and the sections where you run commands are marked so you can skip them for now.
>
> Every key in this post is a **fake key**, including the example key pair published in AWS's own documentation. Numbers come from public sources listed at the end. Examples marked "illustration" are reconstructed and name no real person or company. This is a practical guide, not security advice for your particular system.

---

## Quick summary

- API keys, passwords and access tokens are **secrets**: whoever holds one can use it, like a house key.
- **Deleting a key from a file is not enough.** Git remembers every old version. In this post's experiment, after the key was removed in the second commit, `git log -S` still found both commits and `git show` printed the key again.
- **The right order when a key leaks: rotate or revoke first, clean history second, then read the usage logs.** GitHub's documentation says the same.
- **Rewriting history does not recall existing copies.** In the experiment, a clone made before the rewrite still held the key.
- You can block leaks early with environment variables, `.gitignore`, a 30-line pre-commit hook and the platform's scanning. Each layer has gaps, so you need several.
- Working with AI agents adds three habits: do not paste real keys into the conversation, keep `.env` files out of the agent's reach, and read the diff before committing.

---

## Hard words, plain meaning

| Term | Plain meaning |
|---|---|
| **Secret** | Anything that works for whoever holds it: API keys, passwords, access tokens, database connection strings, private keys. Like a house key. |
| **Commit** | One save in git, with a name and a message. Each commit keeps every file as it was at that moment. |
| **History** | All the commits so far. Deleting a line in a new commit does not delete it from old commits. |
| **Rotate** | Create a new key, move everything to it, then disable the old one. Like changing the lock and handing out new keys. |
| **Revoke** | Disable a key right now so it opens nothing. |
| **Environment variable** | A value set outside the code that the program reads when it runs. Like keeping the key in a separate box instead of engraving it on the door. |
| **Pre-commit hook** | A script git runs just before saving a commit, which can refuse the save. |
| **Secret scanning / push protection** | Features of a platform (such as GitHub) that scan pushed code, or block a push when it looks like it contains a secret. |

---

## After this post you can explain

1. Why deleting a key from a file does not make it disappear, and how to check with three git commands.
2. The right order when a key leaks (rotate first, clean later) and why rewriting history is not enough.
3. Three layers of defense (keep it out of code, block it at commit, scan after push) and where each one falls short.

---

## 1. A picture first: the key under the doormat

Imagine you keep your house key under the doormat. The next day you realize that is a bad idea and move the key to a drawer. But during that one day a neighbor walking by **had a copy made**. Putting the key away does not make the copy disappear. The only sure fix is to **change the lock**.

An API key in code works the same way. When you commit it, git has already stored a copy. Someone may have cloned the repo. An automated scanner may have read it from a public repo. Deleting the line in a later commit is like putting the key in the drawer: good, but it does not deal with the copy.

**Takeaway:** once a secret has left its hiding place, treat it as exposed and **change the lock**, not just put it back.

---

## 2. What counts as a secret, and where it leaks

| Kind | Examples | Where it often leaks |
|---|---|---|
| API keys | Cloud keys, payment keys, AI service keys | Hard-coded in code, in committed config files |
| Passwords | Database passwords, service accounts | Connection strings in code or README examples |
| Access tokens | Short or long-lived login tokens | Program logs, screenshots |
| Private keys | SSH keys, signing keys | A file added to the repo by mistake |
| Connection strings | Database address plus password | Config files, printed environment variables |

Common places leaks happen:

- **Committed straight into git**, then deleted in a later commit (section 3 runs this for real).
- **A `.env` file committed** because it was never added to `.gitignore`, or was committed before it was.
- **Logs and screenshots** containing a key, then sent to a teammate or posted in a forum.
- **Pasted into a chat with an AI** to "show it the error". That text may be stored and shown again in the conversation history.
- **Code that runs in the browser** (a client-side bundle): anything sent down to a user's browser can be read, so never put a secret key there.

**Takeaway:** secrets leak through more than one door, so prevention needs layers.

---

## 3. A real experiment: delete it, it is still there

> Every command and output in this section was run for real in a throwaway repo, with Git 2.56.0 and Python 3.14.8, using fake keys only. Commit hashes differ on every run. Timings are from my machine on a 2-commit repo, so they only give you the order of magnitude.

### 3.1 Commit a fake key, then delete it in the next commit

```bash
git init -q -b main repo
cd repo

# config.py holds the example key pair from the AWS documentation (fake keys)
#   AWS_ACCESS_KEY_ID = "AKIAIOSFODNN7EXAMPLE"
#   AWS_SECRET_ACCESS_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
git add .
git commit -m "add config"             # about 0.03 s

# the next commit switches to environment variables
git add .
git commit -m "remove secret from config"

git log --oneline
```

```text
2bfea15 remove secret from config
a0d01a2 add config
```

### 3.2 It looks clean

```bash
grep -n AKIA config.py          # no match
git grep AKIA                   # no match, it only searches current files
```

Both commands print nothing. The current file really is clean.

### 3.3 But the history still has it

```bash
git log --oneline -S AKIAIOSFODNN7EXAMPLE
```

```text
2bfea15 remove secret from config
a0d01a2 add config
```

The `-S` option finds commits that changed how many times a string appears, so it shows both the commit that added it and the commit that removed it. Next, search every commit and print the old file:

```bash
git grep -n AKIA $(git rev-list --all)      # about 0.02 s
git show a0d01a2:config.py
```

```text
a0d01a2473ddcb923dbfd69164c1df8a11de379b:config.py:3:AWS_ACCESS_KEY_ID = "AKIAIOSFODNN7EXAMPLE"

import os

AWS_ACCESS_KEY_ID = "AKIAIOSFODNN7EXAMPLE"
AWS_SECRET_ACCESS_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
REGION = "ap-southeast-1"
```

Three commands and under a second, and anyone holding this repo can read both keys again.

**Takeaway:** a clean file does not mean a clean repo. You have to search the history.

---

## 4. The right order when a key leaks

Imagine someone tells you "a copy of your house key was made". You change the lock first, and only then go find out who has been inside.

1. **Revoke or rotate the key now.** GitHub's secret scanning documentation recommends rotating "immediately" and says removing a secret from git history is slow and often unnecessary once the key has been revoked.
2. **Read the usage logs** of the key on the service: who used it, from where, when, during the exposure window.
3. **Clean the history** (section 6), if needed, so the next push does not carry the key.
4. **Write it down** and put guardrails in place so it does not happen again (section 5).

### 4.1 Rotating an AWS key: the documented procedure

The AWS documentation on updating access keys describes a rotation that does not interrupt applications:

1. While the first access key is still active, create a second one (now two keys are active).
2. Update every application and tool to use the new key.
3. Check the "last used" information of the old key to see whether anything still uses it. The documentation suggests you can wait several days.
4. Do not delete the old key right away. Set it to inactive first.
5. Use only the new key to confirm applications still work. Anything still using the old key will break, and you can reactivate the old key temporarily.
6. After waiting a while, delete the old key.

That procedure is written for **routine** rotation, with time to wait. For a key that has **already leaked**, disabling the old key immediately is my own inference from GitHub's "revoke first" principle, not something the AWS page says. The cost is that anything still using the old key breaks at once, so have the new key ready.

### 4.2 What fits in 15 minutes, and what does not

This is a target I propose, not a measurement. Real timing depends on whether you already have access and know where the key is used.

| Minutes | Task | Feasible in 15 minutes? |
|---|---|---|
| 0 to 2 | Disable the leaked key on the service | Yes, if you have permission |
| 2 to 5 | Create a new key and store it in the secret store | Yes |
| 5 to 10 | Update every place that uses it (environment variables, servers, CI) | Yes if there are few, hard if you do not remember them all |
| 10 to 13 | Check the application still works with the new key | Yes |
| 13 to 15 | Tell the people involved, note the time | Yes |
| Afterwards | Read usage logs, assess damage | No, usually hours |
| Afterwards | Clean history and ask everyone to pull again | No, and it does not replace rotating |
| Afterwards | Rotate other keys this one could reach | No, you need to list them first |

**Takeaway:** 15 minutes is enough to **close the door**. **Checking the damage** is a job for the hours after.

---

## 5. Block it early: several layers, each with gaps

Imagine a house with three layers: never carry the key out, someone reminds you at the door, and a camera on the street.

### 5.1 Layer one: keep secrets out of code

Put secrets in environment variables or a secret store, and let the config file only say "read from this variable". In the experiment, the second commit did exactly that:

```python
AWS_ACCESS_KEY_ID = os.environ["AWS_ACCESS_KEY_ID"]
```

If you use a `.env` file locally, add it to `.gitignore` **before creating the file**. A real test (I used the name `dev.env` instead of `.env`, because my run environment does not let me create a file named `.env`; it works the same way):

```bash
git status --short
```

```text
?? dev.env
?? tools/
```

```bash
echo "dev.env" > .gitignore
git status --short
git check-ignore -v dev.env
```

```text
?? .gitignore
?? tools/
.gitignore:1:dev.env    dev.env
```

`dev.env` disappears from the list, and `check-ignore` tells you which line of `.gitignore` is blocking it. But `.gitignore` **only affects files git is not already tracking**. I tried it with a file that was committed earlier:

```bash
git add -f old.env
git commit -m "oops: commit old.env"
echo "old.env" >> .gitignore
git ls-files old.env
```

```text
old.env
```

Even after adding it to `.gitignore`, git still tracks the file. To stop tracking it you need `git rm --cached old.env`, and the key in history stays, as in section 3.

### 5.2 Layer two: block at commit with a pre-commit hook

Here is a tiny scanner (about 30 lines of Python) that reads the lines **about to be added** and looks for three shapes: AWS-style keys, the header of a private key, and lines assigning a string to a name like `api_key`, `secret`, `token` or `password`:

```python
PATTERNS = {
    "aws access key id": re.compile(r"AKIA[0-9A-Z]{16}"),
    "private key header": re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"),
    "key or password assigned to a string": re.compile(
        r"""(?i)(api[_-]?key|secret|token|passwd|password)\w*\s*[=:]\s*['"][^'"\s]{8,}['"]"""
    ),
}
# reads: git diff --cached -U0, checks lines starting with "+", exits with code 1 on a match
```

Hook it up in `.git/hooks/pre-commit`:

```sh
#!/bin/sh
exec python3 tools/scan_secrets.py
```

Trying to commit a file with a fake key:

```text
Blocked: this commit adds something that looks like a secret.
  deploy.py: key or password assigned to a string
Use an environment variable, then rotate the key if it was ever real.
exit code: 1
```

The commit is refused (the whole attempt took about 0.14 seconds). But this scanner has two weak spots, and I tested both:

- **A miss:** when a key is split into two strings (`part1 = "AKIA"`, `part2 = "IOSFODNN7EXAMPLE"`, then joined), the commit **goes through** with exit code 0.
- **A false alarm:** a line `password = "change-me-please"` in a sample document is **blocked**, even though it is just a placeholder.

**Takeaway:** a scanner that matches text patterns catches common mistakes, not deliberate or clever ones, and sometimes cries wolf. Do not treat it as a guarantee.

The hook lives in `.git/hooks`, which does not travel with the repo. To share it with a team you need a way to distribute hooks (for example a hook manager in the project), and anyone can still skip it with `--no-verify`.

### 5.3 Layer three: scan after the push, and block at the push

Platforms offer features for this. According to the GitHub documentation pages I read (through a page summarizer):

- **Secret scanning** automatically detects exposed credentials such as API keys and passwords, and scans "your entire Git history on all branches", plus the text of issues, pull requests, discussions and wikis.
- **Push protection** blocks a push that contains a secret before it reaches the repository. It recognizes provider patterns, generic patterns (such as private keys and database connection strings) and custom patterns. At the repository level it is **off by default**, and anyone with write access can bypass it by giving a reason. At the user level, the documentation says it is on by default to stop you pushing secrets to public repositories on GitHub.com.

I read these pages through a summarizer and did not check every word, so open the original pages (listed under references) and look at the settings of your own account. Independent tools such as gitleaks, trufflehog and git-filter-repo exist too, but **not run here**: I did not install them in my test environment.

---

## 6. Cleaning history: what it does and does not do

After you have rotated the key, you may want the repo to stop containing it. The tool recommended today is `git filter-repo`, but I **could not run it** because my test machine does not have it. So I demonstrate with `git filter-branch`, which ships with Git (the Git documentation page says using it "is not recommended" and suggests git filter-repo, but it runs and is enough to illustrate):

```bash
git clone -q repo clone-before       # take a copy BEFORE rewriting
cd repo
git filter-branch -f --tree-filter "python3 redact.py" -- --all   # about 0.8 s, 5 commits
```

Here `redact.py` replaces the two fake key strings with `REDACTED` in every commit. Check:

```bash
git log --oneline -S AKIAIOSFODNN7EXAMPLE main
```

It prints nothing: the `main` branch is clean. But the key still survives in other places:

```bash
git for-each-ref --format='%(refname)'
```

```text
refs/heads/main
refs/original/refs/heads/main
```

The tool leaves a backup ref `refs/original/refs/heads/main`, and `git log -S AKIAIOSFODNN7EXAMPLE --all` still sees the two old commits. To clean up you must delete the backup ref, expire the reflog and prune objects nothing points to:

```bash
git for-each-ref --format='%(refname)' refs/original | xargs -n1 git update-ref -d
git reflog expire --expire=now --all
git gc -q --prune=now                          # about 0.1 s
git log --oneline -S AKIAIOSFODNN7EXAMPLE --all --reflog   # prints nothing
git cat-file -t a0d01a2        # fatal: Not a valid object name
```

Now this repo is clean for good. And **the copy taken before the rewrite?**

```bash
cd ../clone-before
git log --oneline -S AKIAIOSFODNN7EXAMPLE
git cat-file -t a0d01a2
```

```text
2bfea15 remove secret from config
a0d01a2 add config
commit
```

That copy **still has the key**. This is why cleaning history cannot replace rotating the key: anyone who pulled or forked the repo earlier keeps the old version, and a public repo may have been copied or archived by others.

One more warning: rewriting history changes the commit hashes, so teammates must pull again and may hit conflicts. Warn them first. The post [Git for people who use agents](https://www.stacklogs.io.vn/en/engineering/git-for-people-who-use-agents/) explains why force pushing needs care.

---

## 7. Working with AI agents: six more habits

Imagine hiring a contractor: you hand over a key that expires and opens one room, not the master key.

1. **Do not paste real keys into the conversation.** If the AI needs to see an error, replace the key with a placeholder such as `<API_KEY>`.
2. **Keep `.env` files out of paths the agent can read**, or use your tool's setting that blocks reads. An agent that can read the whole project folder can read the secrets in it.
3. **Use narrow, short-lived keys** for anything an agent does: just enough for the task, expiring soon.
4. **Read the diff before committing.** An agent may hard-code a connection string so it "works right away". One `git add` is enough to put it into history. See [Reviewing AI-written code](https://www.stacklogs.io.vn/en/engineering/reviewing-ai-written-code/).
5. **Do not let agents push to the main branch** or run installs and high-privilege commands unattended. See also [OWASP Top 10 for AI apps](https://www.stacklogs.io.vn/en/engineering/owasp-top-10-for-ai-apps-everyday-examples/).
6. **Put the rule into team policy.** The post [A one-page AI use policy](https://www.stacklogs.io.vn/en/product-thinking/one-page-ai-use-policy-for-small-teams/) has a "forbidden" column for this kind of data.

### 7.1 Illustration: a morning asking an agent to connect a database

This is an illustration, naming no real person or product. Names and values are invented.

**Without those habits:**

- **10:00.** A dev asks the agent: "Connect the database for me, here is the connection string" and pastes the real one into the conversation.
- **10:10.** The agent writes `config.py`, hard-coding the connection string so it "works right away", and commits.
- **10:20.** The dev skims it, sees it run, and pushes.
- **10:45.** The platform sends an alert that a secret was detected in the repo. The dev deletes the line and pushes again, thinking it is done.

**With those habits:**

- **10:00.** The dev puts the connection string in an environment variable on their machine and tells the agent: "read it from `DATABASE_URL`, do not write the value into code".
- **10:10.** The agent writes `config.py` reading the environment variable. The pre-commit hook finds nothing odd.
- **10:20.** The dev reads the diff, sees only a variable name, and pushes.

If you landed in the first version, what to do at **10:45** is: **rotate the database password immediately**, then read the access logs, and only then think about cleaning history. Deleting the line in a later commit solves nothing: history already holds the old string, and the conversation with the AI already contains it.

---

## 8. Where people go wrong

| Mistake | Why it fails | What to do |
|---|---|---|
| Delete the key in a new commit and call it done | History and copies remain | Rotate or revoke the key first |
| Commit a key, force push over it and call it clean | Anyone who pulled or forked keeps the old version, and the platform may have kept what was pushed | Treat the key as leaked, rotate |
| Rotate one key but forget another in the same leak | The file had two keys and the attacker uses the other | List every secret in the leaked files and history, rotate all |
| Add a file to `.gitignore` after committing it | Git keeps tracking committed files | `git rm --cached`, and rotate if it was ever pushed |
| Trust the scanner completely | Patterns miss split or encoded keys and flag placeholders | Keep secrets out of code, use layers |
| Rotate the key but not the places that use it | The application suddenly breaks | Create the new key first, update, then disable the old one |
| Skip the usage logs after a leak | You do not know if it was abused | Check who used the key during the exposure window |
| Paste a real key into a chat with AI | The text may stay in the conversation history | Use placeholders |

---

## 9. Numbers and documents used

| Source | What it says | How to read it | Caveat |
|---|---|---|---|
| GitHub Docs, About secret scanning | Scans the whole git history on all branches; recommends rotating immediately, removing from history is slow and often unnecessary once revoked | Basis for "rotate first, clean later" | Read through a page summarizer |
| GitHub Docs, About push protection | Blocks pushes containing secrets; off by default at repository level; bypassable with a reason | A guardrail, not a guarantee | Read through a page summarizer |
| AWS Documentation, Update access keys | Create a second key, update applications, deactivate the old key, then delete | Rotation without downtime | Written for routine rotation, not specifically for leaked keys |
| Escape, The State of Security of Vibe Coded Apps | More than 2,000 vulnerabilities, more than 400 exposed secrets and 175 instances of personal data in apps built with vibe-coding platforms | Fast-built apps often expose secrets | One security vendor scanning its own sample. The number of apps differs between pages I saw (about 1.4 thousand and 5.6 thousand), so I do not quote an app count, and the page gives no date |
| The experiment in this post | A key can still be read from history after deletion; a 30-line scanner misses and false-alarms; a copy made before a rewrite keeps the key | Shows the mechanism, not how often it happens in the wild | One machine, repos of 2 to 5 commits, fake keys |

**A number I did not use:** I tried to check a figure of "41% of 100 apps had exposed secrets or API keys" attributed to a VibeWrench study (March 2026). I could not find a source that confirms it. VibeWrench's product page only says they scanned 100 apps and found 318 vulnerabilities, with no rate of exposed keys. So this post **does not use** that number.

**Reading it correctly:** there is good evidence for the mechanism (git remembers history, copies persist) because you can rerun it yourself. There is no trustworthy figure for how often keys leak in general, and I found no study measuring the effect of pre-commit hooks on their own.

---

## 10. An exercise with answers (30 minutes)

**Task.** Create a throwaway repo and use **fake keys** (like the AWS example pair above, never real ones), then:

1. Commit a file with a fake key, then remove the key in the next commit.
2. Use `git log -S` and `git show` to prove the key is still there.
3. Write a five-step order of actions as if it were a real key.
4. Install a scanning pre-commit hook, then try one commit with a directly assigned key and one with a key split into two strings.
5. Take a copy of the repo, rewrite history with `git filter-branch`, then check both repos.

**Answers.**

1. After two commits, `git log --oneline` shows two lines.
2. `git log --oneline -S <key>` prints both commits. `git show <first commit>:<file>` prints the key.
3. A reasonable order: (1) disable the old key on the service, (2) create a new key and update where it is used, (3) check the application, (4) read usage logs, (5) clean history if needed and tell teammates. Grading point: rotation comes **before** cleaning history.
4. The commit with the directly assigned key is blocked (exit code 1). The one with the split key is **not** blocked, because the scanner only looks for whole patterns.
5. The rewritten repo is clean after deleting the backup ref, expiring the reflog and running `gc`; the copy taken before still has the key. Ideas to improve the scanner: scan whole files instead of only added lines, flag strings by randomness, or use a dedicated tool.

---

## 11. A checklist before you push

1. Is there any real secret in this change (key, password, connection string, private key)?
2. Do secrets live in environment variables or a secret store, not in code?
3. Are `.env` and similar files in `.gitignore` **before the file was created**?
4. Did I read the diff and look for odd long strings before committing?
5. Is a scanning hook running locally, and is platform scanning on?
6. Are the keys I use narrow in scope and short-lived?
7. Have I pasted a real key into a conversation with AI? If so, rotate it.
8. If a key leaked right now, do I know where to rotate it, how long it takes and who to tell?

---

## 12. A balanced view: what we know, what we do not

**Fairly solid:** git keeps history and can reprint a deleted key; rewriting history does not recall existing copies; scanners that match text patterns can miss things and raise false alarms; rotating the key is the root step. All four were run for real in this post.

**Unclear or limited:**

- How often keys leak in the wild and how much damage follows. Available numbers mostly come from security vendors scanning their own samples, and one number I could not confirm was dropped.
- The real effectiveness of each defensive layer. I found no study measuring pre-commit hooks or push protection on their own.
- The dedicated tools (git-filter-repo, gitleaks, trufflehog) I did not run. Read their documentation before using them.
- Platforms differ in features and defaults. I read the GitHub pages through a summarizer and they may have changed.

**Do not overstate:** no single layer is enough. **Do not play it down:** the cost of prevention is a few minutes of setup, while a leaked key can let someone else reach your data or resources.

---

## 13. Conclusion

Once an API key has gone into git, treat it as out of your hands. Deleting the line in a later commit is putting the key in a drawer while the copy is still out there. Rotate the key first, read the logs, and only then clean history. To keep it from happening again, keep secrets out of code, block at commit, scan after the push, and with AI agents hand over narrow, short-lived keys and always read the diff before committing.

---

## References

Numbers come from public sources as of October 2026. I read the GitHub and AWS pages through a page summarizer, so open the originals to check before quoting.

- GitHub Docs, [About secret scanning](https://docs.github.com/en/code-security/secret-scanning/about-secret-scanning)
- GitHub Docs, [About push protection](https://docs.github.com/en/code-security/secret-scanning/introduction/about-push-protection)
- AWS Documentation, [Update access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id-credentials-access-keys-update.html) (IAM User Guide)
- Escape, [The State of Security of Vibe Coded Apps](https://escape.tech/state-of-security-of-vibe-coded-apps) (undated, done by a security vendor on its own)
- Git, [git-filter-branch](https://git-scm.com/docs/git-filter-branch) and [git-log (the -S option)](https://git-scm.com/docs/git-log)