Six months of silent secret scanning

A coding agent on our main repo committed a file called .env.bak. It sat on origin/main and origin/production for sixteen days before one of us spotted it in a git status and asked the room "wait, is that the dev env file we don't commit?"

It was. It had 24 env vars in it.

Here's how it landed silently:

1. The agent, in a session resolving a non-trivial merge, wrote .env.bak to the working tree alongside the conflict resolution and included it in the merge commit. The local pre-commit gitleaks hook ran and found nothing.
2. The commit was pushed. Our PR-time CI runs the same pre-commit hooks via pre-commit/action, so gitleaks ran again against the same config. Still nothing.
3. Sixteen days of normal PRs landed through the same hook. Nothing flagged.
4. We noticed.

How bad was it?

Less bad than the first five minutes of panic suggested. Of 24 vars, most were the placeholder values every dev keeps in their local .env and that aren't valid anywhere beyond their laptop. A handful were live staging credentials. We rotated them within the hour and confirmed they had not been used.

Roughly two hours of work end to end. No customer impact. We moved on.

Then we looked at why our scanner had not caught it.

Why the scanner stayed silent

We had a .gitleaks.toml in the repo, a pre-commit hook that ran gitleaks on every commit, and a CI step that re-ran the same hook on every PR. None of them caught anything.

The .gitleaks.toml looked like this:

[allowlist]
  description = "Allowlist for known false positives"
  regexes = []
  commits = []
  paths = []
  stopwords = [
    # 47 entries of historical fingerprints
  ]

In gitleaks v8, a custom config file does not load the default ruleset unless it explicitly sets [extend] useDefault = true. Without that line, gitleaks loads zero rules and reports "no leaks found" against every input. Our config had been doing that since it was first committed in November.

The pre-commit hook had been a silent no-op for nearly six months. The 47 fingerprints stuffed into stopwords were in the wrong field too. stopwords is a substring check against the matched secret value, not a way to suppress findings by commit + path + rule + line. They were suppressing nothing because there was nothing to suppress.

That bothered us more than the .env.bak itself. The leak was a single commit. The silent scanner is what made that single commit dangerous.

What we rebuilt

Three things, in order.

1. Make the local config actually scan

First fix was the simplest. The repo's .gitleaks.toml got [extend] useDefault = true, a meaningful disabledRules entry for one well-understood false positive (asana-client-secret, which matched on the substring "asana" inside identifiers like hasAnalysis...), and a real .gitleaksignore file with the reviewed historical fingerprints instead of fake stopwords entries.

We also added a CI step that parses .gitleaks.toml as TOML and rejects PRs that put broad [allowlist] blocks back into it, or that add new rules to disabledRules without explicit org-level review. The build now rejects the broken shape; nobody has to remember to check it by hand.

2. Move the workflow from per-repo to org-level

In-repo guardrails drift. When we ran the fixed scanner against a sibling repo we found the same broken [allowlist]-only .gitleaks.toml, with its own pre-commit hook doing nothing for the same reason. The pattern was contagious.

The right place for "must run a secret scan on every PR" is the GitHub org's workflow ruleset. We moved the workflow to our org-level .github repo and enrolled it the same way we enroll our supply-chain check. The workflow runs on every pull_request and merge_group event in every repo in the org. Individual repo maintainers cannot disable it. Only .github owners can change it.

The workflow does not trust each repo's .gitleaks.toml. The base ruleset, the disables, and the allowlist regexes are baked into the workflow itself. Repos still own their .gitleaksignore for per-finding suppressions (gitleaks loads it from the working directory regardless of --config), but the ruleset and what counts as "disable a rule" are owned centrally. A broken local .gitleaks.toml cannot disarm CI anymore.

3. Monthly full-history sweep

PR-time scanning catches new leaks. It does not catch a leak that landed a year ago on a branch nobody is scanning anymore. We needed a periodic full-history pass on every repo.

We built a matrix workflow that runs at 06:00 UTC on the 1st of each month (and on-demand via workflow_dispatch, optionally narrowed to a comma-separated list of repos for ad-hoc runs). On each run:

1. A list-repos job enumerates every active, non-archived, non-fork repo in the org via gh repo list.
2. One matrix job per repo clones the full history and runs gitleaks git . --max-target-megabytes 3 against it. Up to ten run in parallel. fail-fast: false, so one repo's failure does not block the others.
3. Findings are uploaded as a per-repo JSON artifact, rendered as a markdown table in the run's GitHub step summary, and posted to a security Slack channel via the same Slack app our deploy pipeline uses. Clean repos are Slack-silent. Alerts that fire on every run are alerts no one reads.

Cross-repo scanning needs credentials the default GITHUB_TOKEN does not have. We registered a private GitHub App with contents:read, metadata:read, and actions:read across the org, installed it everywhere, and let the workflow mint a short-lived installation token per job via actions/create-github-app-token. No long-lived PAT to rotate.

The gitleaks config, the same [extend] useDefault = true plus disables plus allowlist regexes, lives in a composite action at .github/actions/gitleaks-config. Both workflows (the per-PR scan and the monthly sweep) uses: that action. One file to edit when we need to add a new rule.

What the first real sweep found

We ran it manually against the org once the workflow landed. It found one immediate problem: 13 hits in our public docs repo, all curl-auth-header and curl-auth-user from install-instruction examples. Looking closer, every match was the same shape: a comp_<32 hex> token. These are Konvu company ingestion tokens, the public identifier we hand customers to wire up their workloads. They are not secrets and they appear in every customer's install snippet.

We added one allowlist regex (comp_[0-9a-f]{32}) to the shared gitleaks config and the 13 hits went away. The next monthly sweep will tell us what is actually in our repos.

Skeletons you can lift

If your org has the same shape as ours (a .github repo plus a workflow ruleset), the files below are most of what we run. Drop them in, replace the <YOUR_…> placeholders, and enroll the per-PR workflow via your org ruleset the same way you would any required-workflow rule.

Shared config (.github/actions/gitleaks-config/action.yml)

name: Org gitleaks config
description: Writes the org-wide gitleaks TOML to a temp file and outputs the path.
outputs:
  config-path:
    description: Absolute path to the generated gitleaks config TOML.
    value: ${{ steps.write.outputs.path }}
runs:
  using: composite
  steps:
    - id: write
      shell: bash
      run: |
        set -euo pipefail
        path="$RUNNER_TEMP/gitleaks.org.toml"
        cat > "$path" <<'EOF'
        [extend]
        useDefault = true
        EOF
        echo "path=$path" >> "$GITHUB_OUTPUT"

Per-PR scan (.github/workflows/gitleaks.yml)

name: Secret Scan (gitleaks)

on:
  # Rulesets require pull_request/merge_group triggers (not workflow_call).
  workflow_call:
  pull_request:
  merge_group:

permissions:
  contents: read

jobs:
  scan:
    name: secret scan
    runs-on: ubuntu-latest
    env:
      GITLEAKS_VERSION: 8.30.1
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Install gitleaks
        run: |
          set -euo pipefail
          curl -sSL "https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz" \
            | tar -xz gitleaks
          sudo install -m 0755 gitleaks /usr/local/bin/gitleaks

      - name: Write org gitleaks config
        id: cfg
        uses: <YOUR_ORG>/.github/.github/actions/gitleaks-config@main

      - name: Scan PR diff
        if: github.event_name == 'pull_request'
        run: |
          gitleaks git \
            --config "${{ steps.cfg.outputs.config-path }}" \
            --redact \
            --no-banner \
            --max-target-megabytes 3 \
            --log-opts="${{ github.event.pull_request.base.sha }}..${{ github.event.pull_request.head.sha }}"

      - name: Scan merge group commits
        if: github.event_name == 'merge_group'
        run: |
          gitleaks git \
            --config "${{ steps.cfg.outputs.config-path }}" \
            --redact \
            --no-banner \
            --max-target-megabytes 3 \
            --log-opts="${{ github.event.merge_group.base_sha }}..${{ github.event.merge_group.head_sha }}"

Enroll the per-PR scan org-wide (ruleset)

The workflow above lives in your .github repo but does nothing until you enroll it as a required workflow via an org ruleset. Go to Org → Settings → Rules → New ruleset → Import a ruleset and paste the JSON below.

{
  "name": "Secret Scan (gitleaks)",
  "target": "branch",
  "source_type": "Organization",
  "source": "<YOUR_ORG>",
  "enforcement": "evaluate",
  "conditions": {
    "ref_name":        { "exclude": [], "include": ["~DEFAULT_BRANCH"] },
    "repository_name": { "exclude": [], "include": ["~ALL"] }
  },
  "rules": [
    {
      "type": "workflows",
      "parameters": {
        "do_not_enforce_on_create": false,
        "workflows": [
          {
            "repository_id": <YOUR_DOT_GITHUB_REPO_ID>,
            "path": ".github/workflows/gitleaks.yml",
            "ref": "main"
          }
        ]
      }
    }
  ],
  "bypass_actors": [
    { "actor_id": null, "actor_type": "OrganizationAdmin", "bypass_mode": "always" }
  ]
}

Fill in <YOUR_ORG> and grab the numeric <YOUR_DOT_GITHUB_REPO_ID> with gh api repos/<YOUR_ORG>/.github --jq .id.

Two knobs worth thinking about before you hit save:

• "enforcement": "evaluate" is the rollout mode you want first. The workflow runs against every PR in the org and results show up on the PR page, but failures don't block merges. After a sprint of clean evaluate runs, flip to "active" to actually gate.
• Narrow the blast radius before you widen it. Instead of ~ALL in repository_name.include, list ["my-test-repo", "my-other-test-repo"] for the first day. Once a couple of PRs land cleanly, widen back to ~ALL.

Monthly sweep (.github/workflows/gitleaks-monthly-sweep.yml)

Needs a GitHub App registered in your org (contents:read + metadata:read) installed on all repos, plus a Slack bot token. Secrets: INTERNAL_APP_ID, INTERNAL_APP_PRIVATE_KEY, SLACK_BOT_TOKEN.

name: Monthly Org Gitleaks Sweep

on:
  schedule:
    - cron: "0 6 1 * *"   # 06:00 UTC on the 1st of each month
  workflow_dispatch:
    inputs:
      repos:
        description: "Comma-separated repo names (empty = all active non-archived non-fork repos)"
        required: false
        type: string

permissions:
  contents: read

jobs:
  list-repos:
    runs-on: ubuntu-latest
    outputs:
      repos: ${{ steps.list.outputs.repos }}
      count: ${{ steps.list.outputs.count }}
    steps:
      - name: Mint installation token
        id: app-token
        uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0
        with:
          app-id: ${{ secrets.INTERNAL_APP_ID }}
          private-key: ${{ secrets.INTERNAL_APP_PRIVATE_KEY }}
          owner: <YOUR_ORG>

      - id: list
        env:
          GH_TOKEN: ${{ steps.app-token.outputs.token }}
          REPOS_INPUT: ${{ inputs.repos }}
        run: |
          set -euo pipefail
          if [ -n "${REPOS_INPUT:-}" ]; then
            jq -cn --arg csv "$REPOS_INPUT" '$csv | split(",") | map(gsub("^\\s+|\\s+$"; "")) | map(select(length > 0))' > repos.json
          else
            gh repo list <YOUR_ORG> --limit 200 --no-archived --json name,isFork \
              --jq '[.[] | select(.isFork | not) | .name] | sort' > repos.json
          fi
          {
            echo "repos=$(cat repos.json)"
            echo "count=$(jq 'length' repos.json)"
          } >> "$GITHUB_OUTPUT"

  scan:
    name: scan ${{ matrix.repo }}
    needs: list-repos
    if: needs.list-repos.outputs.count != '0'
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      max-parallel: 10
      matrix:
        repo: ${{ fromJson(needs.list-repos.outputs.repos) }}
    env:
      GITLEAKS_VERSION: 8.30.1
    steps:
      - name: Mint installation token
        id: app-token
        uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0
        with:
          app-id: ${{ secrets.INTERNAL_APP_ID }}
          private-key: ${{ secrets.INTERNAL_APP_PRIVATE_KEY }}
          owner: <YOUR_ORG>
          repositories: ${{ matrix.repo }}

      - uses: actions/checkout@v4
        with:
          repository: <YOUR_ORG>/${{ matrix.repo }}
          token: ${{ steps.app-token.outputs.token }}
          fetch-depth: 0
          persist-credentials: false

      - name: Install gitleaks
        run: |
          set -euo pipefail
          curl -sSL "https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz" \
            | tar -xz gitleaks
          sudo install -m 0755 gitleaks /usr/local/bin/gitleaks

      - name: Write org gitleaks config
        id: cfg
        uses: <YOUR_ORG>/.github/.github/actions/gitleaks-config@main

      - name: Full-history scan
        id: scan
        run: |
          set +e
          gitleaks git . \
            --config "${{ steps.cfg.outputs.config-path }}" \
            --redact --no-banner --max-target-megabytes 3 \
            --report-format json --report-path "$RUNNER_TEMP/findings.json"
          set -e
          [ -s "$RUNNER_TEMP/findings.json" ] || echo "[]" > "$RUNNER_TEMP/findings.json"
          count=$(jq 'length' "$RUNNER_TEMP/findings.json")
          echo "count=$count" >> "$GITHUB_OUTPUT"

      - name: Upload findings
        if: steps.scan.outputs.count != '0'
        uses: actions/upload-artifact@v4
        with:
          name: findings-${{ matrix.repo }}
          path: ${{ runner.temp }}/findings.json
          retention-days: 30

      - name: Build Slack payload
        if: steps.scan.outputs.count != '0'
        env:
          REPO: ${{ matrix.repo }}
          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
        run: |
          jq --arg repo "$REPO" --arg run_url "$RUN_URL" --arg channel "<YOUR_SLACK_CHANNEL_ID>" '
            {
              channel: $channel,
              unfurl_links: false,
              text: (
                ":rotating_light: *gitleaks findings in `\($repo)`* (\(length))\n"
                + ([.[0:5][] | "• `\(.RuleID)` in `\(.File)` (line \(.StartLine // 0))"] | join("\n"))
                + (if length > 5 then "\n…and \(length - 5) more" else "" end)
                + "\n<\($run_url)|workflow run>"
              )
            }' "$RUNNER_TEMP/findings.json" > "$RUNNER_TEMP/slack-payload.json"

      - name: Notify Slack
        if: steps.scan.outputs.count != '0'
        uses: slackapi/slack-github-action@91efab103c0de0a537f72a35f6b8cda0ee76bf0a # v2.1.1
        with:
          method: chat.postMessage
          token: ${{ secrets.SLACK_BOT_TOKEN }}
          errors: true
          payload-file-path: ${{ runner.temp }}/slack-payload.json

      - name: Write job summary
        run: |
          {
            count=${{ steps.scan.outputs.count }}
            if [ "$count" -eq 0 ]; then
              echo "### ✅ ${{ matrix.repo }} — no findings"
            else
              echo "### 🚨 ${{ matrix.repo }} — $count finding(s)"
              echo
              echo "| Rule | File | Line |"
              echo "|------|------|------|"
              jq -r '.[0:20][] | "| `\(.RuleID)` | `\(.File)` | \(.StartLine // 0) |"' "$RUNNER_TEMP/findings.json"
            fi
          } >> "$GITHUB_STEP_SUMMARY"

      - name: Fail job on findings
        if: steps.scan.outputs.count != '0'
        run: exit 1

Takeaways

A silent scanner is worse than no scanner.

Ours was silent for nearly six months. It ran on every commit. The dashboard was green. A real secret-bearing file sat in origin/main for sixteen days before a human (not the tool) noticed.

Agents are part of the model now. This commit came from a coding agent working through a non-trivial merge. It did what a junior engineer might do under pressure. It preserved a file it had been editing. The answer is not "no agents." It is what we already do for humans: scan everything before it ships, rotate fast when something gets through.

Most "leaks" turn out to be local dev. Of the 24 env vars in our .env.bak, most were local-dev placeholders. That is typical for these incidents: the first read looks alarming, then a careful diff against your actually-deployed secrets narrows it to a small list. Start by assuming anything in a tracked .env file is compromised, then check whether prod and local share values. Sometimes they do. Then you rotate.

If you operate a multi-repo GitHub org and your secret scanning is per-repo, you have the same blast radius we had. Move it up a level. Test that the scanner actually flags something it should. Rotate as if history is permanently public, because it is.

Most of what Konvu does is sort scanner findings, picking the few real alerts out of the many. This was that problem turned around: a scanner with no findings because it had no rules. Either way, you have to check that the tool is actually doing the work.