Hardening Containers and Kubernetes: A Field Checklist

Most Kubernetes compromises do not start with a zero day. They start with a container running as root, a service account with cluster-admin, or a dashboard someone exposed to the internet in 2023 and forgot. Hardening is not a product you buy. It is a set of defaults you refuse to accept, and this is the checklist we run when we take over a cluster.

The Threat Model in One Paragraph

An attacker who lands inside a container wants three things: escalate inside the container, escape to the node, or move laterally through the cluster network and the Kubernetes API. Every control below exists to break one of those three paths. If you cannot say which path a control blocks, you probably do not need it yet. That framing keeps hardening work honest and stops teams from drowning in CIS benchmark line items that do not change real outcomes.

Start With the Image, Not the Cluster

The cheapest vulnerabilities to fix are the ones you never ship. Image hygiene is where hardening pays off first.

Shrink the Attack Surface

A default node:20 image ships hundreds of OS packages, a shell, a package manager, and typically hundreds of known CVEs on scan day. Every binary in the image is a tool for the attacker after compromise. Your options, in rough order of effort:

• Slim variants (-slim, -alpine): quick wins, but Alpine's musl libc occasionally breaks native dependencies and DNS behavior in subtle ways. Test before you commit.
• Distroless (Google's gcr.io/distroless images): no shell, no package manager. Debugging happens through ephemeral containers (kubectl debug), which is a workflow change your team must practice before an incident, not during one.
• Chainguard or Wolfi based images: rebuilt daily, typically near zero known CVEs, with SBOMs included. The tradeoff is a dependency on a vendor's build cadence and, for some images, licensing costs.

Build Multi-Stage, Run Non-Root

The build toolchain never belongs in the runtime image. Compilers, npm, git, and curl are exactly what a reverse shell needs.

FROM golang:1.23 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /app ./cmd/api

FROM gcr.io/distroless/static:nonroot
COPY --from=build /app /app
USER 65532:65532
ENTRYPOINT ["/app"]

That final image has one binary, no shell, and a non-root user. An attacker who exploits the app process lands in a room with no tools.

Scan, Sign, and Pin

Scanning without a policy is theater. Wire it into the pipeline with a real gate:

• Trivy or Grype in CI, failing the build on fixable critical and high CVEs. Allow documented, time-boxed exceptions, not silent ignores.
• Cosign to sign images at build time, with a cluster admission policy that rejects unsigned images. This closes the "someone pushed an image from a laptop" hole.
• Digest pinning for base images (FROM node@sha256:...) so builds are reproducible and a poisoned tag cannot slip in.
• SBOM generation with Syft, stored as an artifact, so when the next Log4j style event hits you answer "are we exposed" in minutes, not days.

This is a pipeline problem as much as a security problem. We covered the CI side in depth in our guide to building a DevSecOps pipeline.

Lock Down the Pod Spec

Kubernetes defaults are optimized for "it runs", not "it is safe". The pod securityContext is where you fix that, and it costs almost nothing at runtime.

securityContext:
  runAsNonRoot: true
  runAsUser: 10001
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  seccompProfile:
    type: RuntimeDefault
  capabilities:
    drop: ["ALL"]

What each line actually buys you:

• runAsNonRoot plus a fixed UID: container escape techniques that rely on root inside the container mostly stop working.
• allowPrivilegeEscalation: false: blocks setuid binaries from elevating, which kills a whole class of local privilege escalation.
• readOnlyRootFilesystem: true: malware cannot write payloads to disk. Mount an emptyDir at /tmp for apps that need scratch space.
• seccompProfile: RuntimeDefault: filters roughly 60 of the more exotic syscalls. Nearly every kernel level container escape of the past several years needed a syscall this profile blocks.
• drop: ["ALL"] capabilities: add back only what you can justify, and be suspicious of anything asking for NET_ADMIN or SYS_ADMIN. A pod with SYS_ADMIN or privileged: true is, for practical purposes, root on the node.

Enforce all of this with Pod Security Admission at the namespace level. Label every application namespace pod-security.kubernetes.io/enforce: restricted and treat exceptions as tickets with owners and expiry dates. In our experience the migration surfaces a handful of legacy workloads that genuinely need privileges (CNI agents, log collectors, storage drivers). Isolate those in dedicated namespaces rather than loosening the policy everywhere.

Contain the Blast Radius

Assume a pod will eventually be compromised. The question is what the attacker can reach next.

RBAC: Least Privilege or Bust

The most common finding in our cluster reviews: service accounts with permissions nobody can explain. The rules are simple and boring:

• Never bind cluster-admin to a workload. Ever. CI deployers included.
• One service account per workload, scoped to its namespace, with verbs it actually uses.
• Set automountServiceAccountToken: false by default. Most application pods never call the Kubernetes API, yet every one of them ships with a valid API credential mounted at a well known path. That token is the first thing an attacker reads.
• Audit periodically with kubectl auth can-i --list --as=system:serviceaccount:ns:name or tooling like rbac-tool. Wildcards in verbs or resources are findings, not conveniences.

Network Policies: Default Deny, Then Allow

Out of the box, every pod can talk to every other pod across every namespace. That flat network is why one compromised frontend becomes a database breach. Start every namespace with a default deny:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

Then allow specific flows: frontend to API on 8080, API to Postgres on 5432, everything to DNS. Egress policy matters as much as ingress, because it is what turns "attacker in a pod" into "attacker in a pod who cannot reach their C2 server or the cloud metadata endpoint". Blocking pod access to 169.254.169.254 unless the workload genuinely needs it has stopped real cloud credential theft paths. This is zero trust applied at the pod level, and the reasoning mirrors what we laid out in zero trust architecture explained.

The tradeoff: network policies are unforgiving to debug when a deploy breaks because someone forgot the DNS egress rule. Ship a tested policy library with your platform templates instead of asking every team to write their own.

Secrets: Stop Pretending Base64 Is Encryption

Kubernetes Secrets are base64 encoded, not encrypted, and anyone with read access to secrets in a namespace has the plaintext. Minimum bar:

1. Enable encryption at rest for etcd with a KMS provider (managed clusters: verify it, do not assume it).
2. Prefer mounted secret volumes over environment variables. Env vars leak into crash dumps, kubectl describe, and child processes.
3. For anything serious, pull from an external manager (Vault, AWS Secrets Manager, GCP Secret Manager) via External Secrets Operator or CSI driver, so rotation does not require redeploying the world.
4. Lock down RBAC on the Secret resource itself. get secrets cluster-wide is domain admin in disguise.

Harden the Control Plane and Nodes

On managed Kubernetes (EKS, GKE, AKS) the provider owns etcd and the API server binaries, but you still own the configuration that matters:

• Private API endpoint, or at minimum strict CIDR allowlists. Internet-exposed API servers get scanned within minutes of creation.
• Audit logging enabled and shipped somewhere searchable. Without audit logs, post-incident you are reconstructing attacker actions from memory.
• Node OS: use minimal, container-optimized images (Bottlerocket, COS, Flatcar), keep node pools on an automated upgrade cadence, and never allow SSH to nodes as routine practice.
• Version currency: Kubernetes minor versions fall out of support in about a year. Clusters more than two minors behind accumulate unpatched CVEs and a terrifying upgrade cliff. Upgrade little and often.

Detect at Runtime, Because Prevention Fails

Everything above reduces probability. Runtime detection handles the residual. Tools like Falco or Tetragon watch syscalls via eBPF and flag behavior that should never happen in a hardened workload: a shell spawning in a distroless container, an unexpected outbound connection, a process reading the service account token, writes to /etc/passwd.

The signal quality here is unusually good, precisely because hardening removed the noise. If your image has no shell, "shell executed" is not an anomaly score. It is an incident. Wire these alerts into a response path with owners and runbooks. If you do not have that muscle yet, start with our incident response playbook.

Failure Modes We Keep Seeing

• Hardening applied to new services while the legacy namespace keeps privileged: true "temporarily", for two years.
• A scanner in CI with the failure threshold set to none, generating reports nobody reads.
• Network policies deployed without an egress rule for DNS, breaking production, then reverted entirely instead of fixed.
• Admission policies in audit mode forever because nobody scheduled the enforcement cutover.
• Secrets "migrated to Vault" but the old Kubernetes Secrets never deleted, still readable, still valid.

The pattern behind all five: hardening treated as a project with an end date instead of a set of enforced defaults. Policy engines (Kyverno, or native ValidatingAdmissionPolicy with CEL) exist to make the secure path the only path.

The Field Checklist

Run this top to bottom. Each item is a yes or no, and "mostly" counts as no.

1. All runtime images are minimal (distroless, Chainguard, or slim) and built multi-stage.
2. CI fails builds on fixable critical and high CVEs, with time-boxed exceptions only.
3. Images are signed with Cosign and the cluster rejects unsigned images.
4. Base images are pinned by digest and SBOMs are generated per build.
5. Every workload runs non-root with allowPrivilegeEscalation: false, dropped capabilities, RuntimeDefault seccomp, and a read only root filesystem.
6. Pod Security Admission enforces restricted on all application namespaces.
7. No workload service account holds cluster-admin; token automount is off by default.
8. Every namespace has default deny network policies for ingress and egress, including metadata endpoint blocking.
9. etcd encryption at rest uses a KMS; secrets live in an external manager with rotation.
10. The API server is private or CIDR restricted, and audit logs ship to searchable storage.
11. Nodes run a container-optimized OS with automated patching, and the cluster is within two minor versions of current.
12. Runtime detection (Falco or Tetragon) is deployed with alerts routed to an owned response process.
13. Admission policies enforce all of the above so drift cannot happen silently.

Score twelve or more and you are ahead of the vast majority of clusters we assess. Score under eight and you should treat this as this quarter's priority, because attackers automate the discovery of exactly these gaps.

How Innovation T can help

Innovation T designs, builds, and hardens containerized platforms for companies across Europe, the Gulf, and North Africa. We run cluster security assessments against this exact checklist, then do the unglamorous work: migrating workloads to restricted pod security, writing the policy library, wiring image signing into CI, and leaving your team with enforced defaults instead of a PDF report.

If your cluster has grown faster than its guardrails, explore our cloud and security services or talk to our engineers. We will tell you honestly which of these thirteen items matter most for your environment, and which can wait.