Home
Kubernetes
SRE

From Splunk to Kubernetes: What 8 Years of Alert Fatigue Taught Me About Self-Healing Systems

Eight years of SRE alert fatigue shaped how I think about failure detection. See what a failed Kubernetes rollout taught me about rollout safety.
Cristian Gomez Aranda

For eight years I worked as an SRE at a major financial institution. Legacy services, high-stakes transactions, multiple platforms running simultaneously. Monitoring coverage was extensive: Splunk, AppDynamics, and several other systems feeding dashboards around the clock. Thousands of alerts per day.

The tooling was sophisticated. The hard part was still detection, routing, and deciding what to do next.

The problem with detecting your own failures

The architecture used to manage that alert volume was layered. Each domain had its own distribution list with pre-filtered, high-priority alerts routed to the responsible team. The intent was good: reduce noise, surface what matters, and close the gap between failure and response.

At the scale of a large institution, across multiple shifts and dozens of services running simultaneously, alerts still got buried. An incident could start silently. By the time it was detected, triaged, and routed to the right specialist team, a service might have been failing for minutes. Sometimes longer.

Then the real work started.

In production at a bank, you do not simply restart a service. You open an incident, notify stakeholders, get specialists onto a bridge, and determine whether the problem came from infrastructure, configuration, application behavior, or a recent change.

If the issue came from a bad deployment, you might need an urgent change request with approvals before rolling it back. Paperwork, signatures, a second pair of eyes. All while clients were potentially affected.

Once approved, the team could open the internal Ansible automation platform, select the appropriate playbook, run it, watch the console output, return to the monitoring dashboards, verify recovery, and then close the incident.

The important distinction
The automation itself was not the limiting factor. Humans still had to detect the failure, interpret it, authorize the response, execute the recovery path, and verify the result.

Today, my rollout broke

I am currently deep in hands-on Kubernetes and Platform Engineering practice. During a drill, I pushed an image update using a tag I had not verified:

kubectl set image deployment api httpd=httpd:2.4.1 --namespace=drill-fri
kubectl rollout status deployment api --namespace=drill-fri

The rollout stalled. The image tag could not be pulled, and three of the new Pods entered ImagePullBackOff.

After the Deployment exceeded its progress deadline, kubectl rollout status returned an error:

error: deployment "api" exceeded its progress deadline
Important detail
Exceeding progressDeadlineSeconds does not automatically roll the Deployment back. Kubernetes reports the stalled progress through Deployment status, while the Deployment controller continues processing the Deployment.

I checked the Pods:

kubectl get pods --namespace=drill-fri
NAME                   READY   STATUS             RESTARTS   AGE
api-57ffdf6c9b-k77xl   0/1     ImagePullBackOff   0          20m
api-57ffdf6c9b-lq98m   0/1     ImagePullBackOff   0          20m
api-57ffdf6c9b-xxksp   0/1     ImagePullBackOff   0          20m
api-5f4768f975-9dtr2   1/1     Running            0          45m
api-5f4768f975-qgt98   1/1     Running            0          47m
api-5f4768f975-slxz9   1/1     Running            0          47m
api-5f4768f975-wcmfw   1/1     Running            0          47m

Four Pods from the previous ReplicaSet were still available. The failed update had not replaced the entire healthy ReplicaSet.

What the rollout protected
The Deployment preserved four available replicas while the new Pods failed to start. That demonstrates rollout availability protection. It does not, by itself, prove zero client impact because the workload was operating with fewer available replicas than the desired count of five.

What was happening under the hood

Kubernetes Deployments use RollingUpdate by default. Two parameters control how aggressively old replicas are removed and new replicas are created:

Parameter Default Meaning With 5 replicas
maxUnavailable 25% Maximum number of desired replicas that may be unavailable during the rollout 1, because percentage values are rounded down
maxSurge 25% Maximum number of extra Pods that may exist above the desired replica count 2, because percentage values are rounded up

With five desired replicas and the defaults, the rollout can temporarily create up to seven Pods in total, while the availability rules constrain how far the old ReplicaSet can be reduced.

The new Pods in this drill never became Ready because their containers could not start. Kubernetes could not pull httpd:2.4.1, so those Pods could not become usable replacements for the old replicas.

Running is not the same as application-ready
For a real application, rollout safety depends on Kubernetes having a meaningful signal for readiness. A container merely running is not sufficient proof that the application can serve traffic. A correctly designed readinessProbe should represent whether the Pod is ready to receive requests.

In this particular failure, the distinction was simple: an ImagePullBackOff Pod could not even start its container, so it could not become Ready.

What the progress deadline actually means

The Deployment's progressDeadlineSeconds defaults to 600 seconds.

If the Deployment does not make progress within that period, the Deployment controller surfaces a status condition equivalent to:

type: Progressing
status: "False"
reason: ProgressDeadlineExceeded

This is a reporting mechanism. It tells operators and higher-level automation that the rollout has stalled.

It is not an automatic rollback mechanism.

Operational model
A higher-level deployment system can react to ProgressDeadlineExceeded and automate remediation, but the Kubernetes Deployment controller itself does not automatically undo the rollout.

Diagnosing the failure

Once I saw the stalled rollout, the diagnostic sequence was straightforward.

  1. Check the state of the Pods.
  2. Identify which Pods belong to the new ReplicaSet.
  3. Describe a failing Pod.
  4. Read the Events section before changing anything.
  5. Determine whether the old ReplicaSet is still providing capacity.
# 1. Check Pod states
kubectl get pods --namespace=drill-fri

# 2. Describe one of the failing Pods
kubectl describe pod api-57ffdf6c9b-k77xl --namespace=drill-fri

The Events section exposed the cause:

Failed to pull image "httpd:2.4.1": ... docker.io/library/httpd:2.4.1: not found
Root cause
The new ReplicaSet referenced an image tag that could not be pulled. The kubelet repeatedly retried the image pull with backoff, leaving the new Pods unable to start and therefore unable to become Ready.

Recovery

For this drill, I chose to roll the Deployment back to the previous revision:

kubectl rollout undo deployment api --namespace=drill-fri
kubectl rollout status deployment api --namespace=drill-fri

The Deployment returned to a successful rollout state:

deployment "api" successfully rolled out
Verify the recovery
Do not stop at a successful rollout status. In a real system, verify the actual service path, application readiness, error rate, latency, and any other signal that represents user-visible health.

What eight years taught me

The difference between the operational model I knew and what I was observing in Kubernetes was not simply "manual versus automated."

We already had automation.

The more interesting difference was where operational policy lived.

In the older model, monitoring detected a problem after the workload had changed state. Humans then interpreted the alert and initiated recovery automation.

With a Kubernetes Deployment, rollout policy is encoded directly into the reconciliation process. The controller manages old and new ReplicaSets according to availability constraints, observes whether new Pods become available, and reports when the rollout stops making progress.

That does not eliminate monitoring, incident response, or human judgment. It moves an important class of failure handling closer to the deployment mechanism itself.

A stalled rollout is not necessarily the platform failing to act. Sometimes it is the platform refusing to replace more known-good capacity with replicas that have not proven they can become available.

Operational takeaway

When you see ImagePullBackOff during a rollout

  1. Inspect the Deployment and Pods.
    Determine which Pods belong to the old and new ReplicaSets and how many replicas remain available.
  2. Describe a failing Pod.
    Read Events. Common causes include an invalid image name or tag, registry connectivity problems, authentication failures, or a missing imagePullSecret.
  3. Inspect rollout status and Deployment conditions.
    A ProgressDeadlineExceeded condition tells you the rollout has stopped making progress; it does not mean Kubernetes automatically rolled it back.
  4. Choose the recovery action deliberately.
    Fix forward, change the image reference, or use kubectl rollout undo when reverting to the previous Deployment revision is the appropriate response.
  5. Verify the complete service path.
    Confirm application readiness and user-visible behavior, not only Pod status.

The mental model I am keeping

Deployment rollout mental model
New ReplicaSet created → new Pods attempt to start → Pods must become Ready/Available → old ReplicaSet is reduced within rollout constraints → stalled progress is surfaced in Deployment status.

Kubernetes did not magically repair the invalid image tag. It did something more specific and, operationally, very useful: it prevented the failed new replicas from becoming valid replacements for all of the existing capacity and gave me status information that pointed directly at the stalled rollout.

The cluster still needed an operator to understand the failure and choose the recovery action.

But the failure surfaced inside the deployment workflow itself, before the bad revision could completely replace the previous ReplicaSet.

That is the part that changed my mental model.

References

References:
Kubernetes Documentation · Deployments
Kubernetes Documentation · Images and ImagePullBackOff

Post a Comment