K8s Controllers

Kubernetes controllers operate in a perpetual “control loop,” much like a thermostat that never shuts off. You declare your desired state say, “I want three replicas of my web service running” and Kubernetes continuously measures the current state against that target. When a discrepancy appears (perhaps one Pod crashed), the controller springs into action, reconciling the two by creating, deleting, or rescheduling resources. This non-terminating loop ensures your cluster self-heals and stays aligned with your intent.

1# Pseudo-code: Kubernetes Control Loop
2while true:
3  current_state = query_cluster_state()
4  desired_state = read_manifest_specs()
5  if current_state != desired_state:
6    reconcile(current_state, desired_state)
7  sleep(check_interval)

Under the hood, built-in controllers live inside the kube-controller-manager and each watches a specific set of resources via the API server. For example, the Job Controller focuses on one-time, finite tasks: it reads a Job’s spec, spins up Pods, watches them to completion, and then marks the Job done.

1# What happens when you create a Job?
2apiVersion: batch/v1
3kind: Job
4metadata:
5  name: example-job
6spec:
7  completions: 1
8  template:
9    spec:
10      containers:
11      - name: task
12        image: busybox
13        command: ["echo", "Run migration"]
14      restartPolicy: Never
15  backoffLimit: 4

Not all controllers work solely through the API server. Some, like the Cluster Autoscaler, interact directly with cloud APIs. They still read the desired node count from your cluster spec but then bypass Kubernetes resources to add or remove VMs, finally reporting status back upstream.

1# Pseudo-code: Node Autoscaler Loop
2while true:
3    utilization = measure_node_utilization()
4    desired_count = calculate_nodes(utilization, target=75%)
5    if desired_count != current_node_count:
6        cloud_api.adjust_node_count(desired_count)
7        api_server.update_node_status(desired_count)
8    sleep(autoscale_interval)

Kubernetes favors many small controllers over a monolith. Each controller has a narrow focus (ReplicaSets, Jobs, Nodes, etc.) and identifies its resources via labels, preventing cross-talk or unintended interference. Independent loops mean that if one controller falters, the rest carry on unaffected—bolstering the system’s resilience.

Controllers come in two flavors:

  1. Built-in (in kube-controller-manager), handling ReplicaSets, Jobs, Nodes, etc., with built-in failover and leader election.
  2. External, running as Pods (or outside the cluster) to extend Kubernetes—common for CRD-based operators or custom workflows.

Even within the cluster, the loop repeats ad infinitum. Imagine you deploy a Deployment set to 3 replicas:

1# ReplicaSet controller’s reconciliation logic
2desired_replicas: 3
3current_replicas: count_pods(label=app=web)
4if current_replicas < desired_replicas:
5  create_pods(desired_replicas - current_replicas)
6elif current_replicas > desired_replicas:
7  delete_pods(current_replicas - desired_replicas)

When a node fails and one Pod disappears, the ReplicaSet controller detects only two running, compares to the desired three, and immediately spies a gap—instantly spawning a new Pod to restore equilibrium.