Secrets Management: Stop Hardcoding API Keys

Every breach postmortem has a familiar chapter: somebody found a credential. In a repo, in a Docker layer, in a CI log, in a screenshot pasted to Slack. Secrets management is not hygiene theater. It is the difference between a contained incident and an attacker holding the keys to your production database.

Why hardcoded secrets never stay secret

A secret committed to code is not a secret. It is a time bomb with public visibility settings.

Here is where they actually leak:

• Git history. Deleting the key in a new commit does nothing. The old blob lives in the object store, in every clone, in every fork, and in your CI runner caches. git log -p finds it in seconds.
• Docker image layers. COPY .env . followed by RUN rm .env still ships the secret. Layers are additive. Anyone with pull access can run docker history and extract it.
• CI logs. A stray env | sort in a debug step, a framework that prints config on boot, a test runner that dumps the environment on failure. Logs get retained, forwarded, and indexed.
• Client bundles. Frontend builds inline anything prefixed for public exposure. We regularly see server keys shipped to the browser because someone renamed a variable to make the build pass.
• Third party sync. Editor plugins, backup tools, and AI coding assistants that index your workspace will happily index .env too.

Automated scanners watch the public GitHub event stream around the clock. In our experience, an exposed cloud key pushed to a public repo gets probed within minutes, not days. Cryptominers on your AWS bill are usually how teams find out.

The threat model, stated plainly

You are defending against three things:

1. Exfiltration: an attacker reads the secret from somewhere it was written down.
2. Replay: an attacker who has the secret uses it, from anywhere, for as long as it remains valid.
3. Blast radius: one leaked credential unlocks far more than it should.

Every control below attacks one of these. Encryption at rest attacks exfiltration. Short TTLs attack replay. Scoped, per service credentials attack blast radius. If a tool does not clearly map to one of the three, it is decoration.

The maturity ladder

Do not jump straight to running a Vault cluster. Climb deliberately.

Level 0: .env files, gitignored

The floor, not the goal. Acceptable for a solo prototype. The secrets are plaintext on every laptop, there is no audit trail, no rotation, and offboarding an engineer means hoping they delete the file.

Level 1: platform secret stores

Use the store your platform already gives you: AWS Secrets Manager or SSM Parameter Store, GCP Secret Manager, Azure Key Vault, or the encrypted secrets in Vercel, Fly.io, or GitHub Actions. Secrets are encrypted at rest, injected at runtime, and access is governed by IAM. For most teams under 20 engineers, this level, done properly, beats a badly operated Vault.

The key discipline at this level: the application reads secrets from the environment or the SDK at boot. It never reads them from a file in the repo.

Level 2: centralized secrets with access control and audit

One system of record for every secret across every environment. HashiCorp Vault, OpenBao (the open source fork), Infisical, or Doppler. What you gain over level 1:

• Uniformity: one API and one policy language across AWS, GCP, on prem, and CI.
• Audit: every read is logged with identity, path, and timestamp. When a key leaks, the audit log is how you scope the incident.
• Policy: the billing service can read billing/* and nothing else, enforced centrally.

Level 3: dynamic, short lived credentials

The endgame. No static secrets exist at all. Credentials are minted on demand, scoped to one identity, and expire in minutes or hours. A leaked credential at this level is a nuisance, not a breach.

Dynamic secrets: kill the static credential

Vault's database secrets engine is the clearest example of the pattern. Instead of one shared DB_PASSWORD living in twelve places, each service instance requests its own user at startup:

vault write database/roles/app-readonly \
  db_name=postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' \
    VALID UNTIL '{{expiration}}'; \
    GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" max_ttl="24h"

Every read of database/creds/app-readonly creates a fresh Postgres role that self destructs in an hour. The properties you get for free:

• Attribution. Slow query from v-k8s-app-readonly-x7Hf? You know exactly which pod issued it.
• Instant revocation. Revoke the lease, the credential dies now, not at the next deploy.
• Worthless leaks. A credential screenshotted in a debug session is expired before anyone can use it.

The same pattern exists for AWS STS credentials, GCP service account keys, SSH certificates, and PKI. The tradeoff is real: your database must tolerate role churn, connection poolers need re-authentication logic, and Vault becomes tier zero infrastructure. Which brings us to the failure modes.

The secret zero problem, and other failure modes

Teams adopt a vault and then trip over the same four things:

• Secret zero. The app needs a credential to talk to the vault. If that credential is a static token in an env var, you have relocated the problem, not solved it. The fix is platform identity: Kubernetes service account tokens, AWS IAM auth, GCP instance identity. The workload proves who it is with something the platform attests, not something a human pasted.
• The vault as a single point of failure. If Vault is down and your pods cannot boot, you have converted a security tool into an availability incident. Run it with real HA, cache leases client side, and set TTLs long enough to survive a short outage.
• Sprawl regression. Engineers under deadline copy secrets out of the vault and back into .env files "temporarily." Make the sanctioned path the easiest path: SDK integration, sidecar injection, or templated files at deploy time.
• Write only audit logs. An audit log nobody alerts on is a compliance artifact. Alert on reads from unusual identities, bulk reads, and root token usage.

CI/CD: stop storing cloud keys in your pipeline

CI is where static credentials go to get stolen. A long lived AWS_SECRET_ACCESS_KEY in your pipeline settings is readable by every maintainer, every compromised action, and every dependency with a postinstall script.

The modern answer is OIDC federation. The CI provider issues a signed, short lived identity token per job, and your cloud trusts that token directly. No stored key exists to steal:

permissions:
  id-token: write
  contents: read
steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789012:role/deploy-prod
      aws-region: eu-west-1

Scope the IAM trust policy to your exact org, repo, and branch. A job on main in your repo can deploy; a job on a fork cannot, cryptographically. GitHub Actions, GitLab CI, and CircleCI all support this against AWS, GCP, and Azure. If you are building out your pipeline security posture more broadly, our DevSecOps pipeline guide covers where secrets handling sits in the larger chain.

Catch leaks before they ship

Prevention beats response, and the tooling is free. Layer three nets:

1. Pre-commit scanning. Gitleaks runs in under a second on a staged diff:

repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.24.0
    hooks:
      - id: gitleaks

2. Push protection. GitHub secret scanning with push protection blocks the push server side when it matches a known credential pattern. Turn it on for every repo, private included. GitLab has an equivalent.

3. History and artifact scans. Run TruffleHog or Gitleaks across full git history, container images, and build artifacts on a schedule. Verified detections (the scanner actually calls the provider to check the key is live) cut the false positive noise dramatically.

None of this replaces designing APIs and services so that secrets are scoped tightly in the first place. Our post on API security best practices goes deeper on key scoping and rotation at the API boundary.

When a key leaks anyway: the first hour

Assume it will happen. The playbook, in order:

1. Revoke first, investigate second. The moment you confirm exposure, kill the credential. Do not wait for a maintenance window. Availability pain is recoverable; exfiltrated data is not.
2. Mint the replacement. Issue the new credential through your secret manager, scoped tighter than the old one. This is the moment to fix the over broad permissions you have been meaning to fix.
3. Scope the damage. Pull audit logs for the credential's entire exposure window, not just since discovery. In AWS that means CloudTrail queries for the access key ID. Look for unfamiliar IPs, regions, and API calls.
4. Purge, but treat it as burned. Rewrite history with git filter-repo and force push, but understand this is cleanup, not remediation. Clones and forks still have it. The credential is dead either way, that is the point of step 1.
5. Fix the path, not the person. The secret got committed because the safe path was harder than the unsafe one. Add the pre-commit hook, wire the secret manager, close the gap.

If you do not have this written down and rehearsed before you need it, start with our incident response playbook.

A decision framework

Match the tool to the team, not the conference talk:

• Solo or tiny team, single cloud: your cloud's secret manager plus OIDC in CI. An afternoon of work, most of the value.
• Growing team, one or two clouds, Kubernetes: cloud secret managers as the backend, External Secrets Operator syncing into the cluster, push protection on every repo.
• Multi cloud, compliance requirements, 24/7 workloads: Vault or OpenBao with platform identity auth, dynamic credentials for databases and cloud access, alerting on the audit log.
• Regulated or high value targets: everything above, plus short TTLs as policy, quarterly leak drills, and secrets rotation verified by automation rather than a calendar reminder.

Whatever level you choose, three rules are universal. No secret in git, ever. No secret shared between two services. Every secret must have an owner, a scope, and an expiry someone can state out loud.

How Innovation T can help

Innovation T designs and operates secrets infrastructure for teams across Europe and North Africa: Vault and OpenBao deployments with real HA, OIDC federation for CI/CD, dynamic database credentials, and leak detection wired into the pipeline rather than bolted on after an incident. We have migrated teams off hardcoded keys without a single production outage, and we leave behind runbooks your engineers actually use.

If your .env files are one laptop theft away from being an incident report, talk to us. See our services or get in touch for a secrets posture review.