Governance & Power

Secrets, Sealing, and the False Promise of Safety

Secrets are never a single object. They are a pipeline: creation, storage, distribution, use, and rotation—each step with its own exposure costs.

Return to Archive Governance Initiation

Text

Authored as doctrine; evaluated as systems craft.

Doctrine

Operators frequently treat Kubernetes Secrets as a security primitive. They are not. They are an API object with a distribution mechanism. Safety depends on encryption at rest, RBAC, audit, node compromise assumptions, and how workloads consume values.

Kubblai doctrine: the first secret to protect is your threat model. Without it, you ship theatre.

Define the threat model: cluster admin, namespace admin, node compromise, CI compromise.
Minimize secret material in the cluster; prefer workload identity where possible.
Treat rotation as mandatory, not optional.

Where secrets leak in real clusters

Secrets leak through logs, crash dumps, misconfigured sidecars, exec into pods, node-level access, and overbroad RBAC. They leak through backups and snapshots you forgot existed.

A sealed system is one that is audited, not merely encrypted.

Audit who can get/list/watch secrets (and via which groups).
Avoid env var injection for high-risk material when logs/metrics might capture it.
Review backup policies for etcd snapshots and persistent volumes.

Encryption boundaries

Encryption at rest for etcd is table stakes. It protects against storage theft, not against a compromised API path or an overprivileged user. In-transit encryption protects the network, not the cluster admin.

The boundary that matters is authority: who can ask the system for the secret, and where it is decrypted.

Enable encryption at rest with key rotation procedures.
Restrict secret access using least privilege RBAC and namespace boundaries.
Prefer external secret stores with short-lived credentials where feasible.

Rotation without illusion

Rotation is painful because consumption is sticky. Some workloads read secrets only at startup. Some cache credentials. Some use connection pools that survive past the rotation window.

Rotation doctrine includes operational windows, rollout plans, and verification steps.

Design workloads to reload secrets safely (SIGHUP, sidecar reloaders, or restart protocols).
Stage rotations with overlapping validity (old + new) where possible.
Verify with metrics: auth failures, connection churn, error rates.

The quiet replacement: workload identity

Many secrets exist because identity is hard. When you can use workload identity (OIDC, SPIFFE/SPIRE, cloud IAM federation), you reduce static secret material and improve auditability.

But identity systems also fail. Plan for outage posture and fallback.

Treat identity provider availability as an SLO.
Implement least privilege policies for token audiences and scopes.
Avoid long-lived tokens; prefer short-lived exchange flows.

Canonical Link

Canonical URL: /library/secrets-sealing-and-the-false-promise-of-safety