Doctrine / Theology

The Doctrine of Reconciliation

Reconciliation is not a feature; it is the constitutional law of Kubernetes. The cluster stays honest by continuously closing the gap between intent and reality.

Return to Library Governance Apply

Text

Authored as doctrine; evaluated as operations.

Doctrine

Reconciliation is the refusal to accept drift as fate. It is the act of comparing declared intent to observed reality and taking the smallest repeatable steps toward convergence—without theatrics, without panic.

Kubblai treats reconciliation as a moral stance for operators: you do not improvise the platform into health; you instrument it, encode intent, and let control loops do what control loops are meant to do.

Desired state must be explicit and versioned.
Controllers must be idempotent and retry-safe.
Status is a contract; conditions are the platform’s testimony.

Control loops in practice

A controller’s core is simple: watch → compare → act → record. The complexity is in the edges: eventual consistency, watch resyncs, leader election, conflict retries, and the fact that multiple actors can touch the same objects.

When reconciliation fails, it rarely fails loudly. It stalls. It thrashes. It makes progress in the wrong direction. Serious operations require you to read the shape of convergence, not just the current symptom.

Understand informer/watch semantics and what events you can miss.
Budget for API QPS/burst and controller backoff; starvation is a failure mode.
Prefer declarative fields and server-side apply where it reduces write conflicts.

Backpressure, rate limits, and the ethics of throughput

Control planes are finite. etcd write amplification, API server admission latency, and controller work queues all impose limits. Reconciliation must respect those limits or become the incident.

Kubblai doctrine is explicit: when the platform is unstable, the correct move is to reduce churn, not to increase it.

Use work queues with bounded concurrency and exponential backoff.
Avoid tight reconcile loops that continuously write status without meaning.
Design for load-shedding: pause non-essential controllers during systemic events.

Observability: proving convergence

You cannot operate reconciliation by intuition. You need metrics: queue depth, reconcile duration, error rates, conflict retries, and API latency. You need logs that connect decisions to object keys. You need events that don’t become noise.

The strongest operators can answer: “Is the system converging?” before they can explain every symptom.

Instrument controllers with structured logs keyed by namespace/name.
Expose SLO-aligned metrics: time-to-converge and percent reconciles succeeding.
Treat status conditions as an API: stable, documented, and parsable.

Field notes

Reconciliation loops are at their most dangerous during partial outages: API latency spikes, etcd compaction pressure, and failing webhooks. Your controller may ‘work’ while making everything else worse.

The operator’s discipline is to recognize systemic failure and stop contributing to it.

If admission is failing, stop applying changes. Fix admission first.
If etcd is unhappy, reduce write load (including status spam).
During an outage, prefer read-only diagnosis until the control plane stabilizes.

Canonical Link

Canonical URL: /library/doctrine-of-reconciliation