Lost requests during a rollout: why Kubernetes returns 502 while the pod is shutting down

A rollout can run through cleanly, every pod comes up, every pod reports Ready, no restarts, and yet a few hundred requests never find their way. The application logs show nothing about them, because those requests never reached the application. There is no configuration error behind this. When a pod is terminated, removing it from the network does not happen before the shutdown, it happens alongside it. This article demonstrates the failure on a Spring Boot service behind an NGINX ingress, then the fix with a preStop hook and graceful shutdown, and finally a load test that proves it is really gone.

Contents

• A release at ten past two
• What actually happens when a pod is terminated
• Why the application log stays empty
• Why the failure looks sporadic and hits the writes
• The reflex that does not help: tightening the readiness probe
• The preStop hook that does nothing
• The arithmetic that has to work out
• The application has to play along: graceful shutdown in Spring Boot
• The complete manifest
• What the preStop hook does not solve
• Produce the failure before it arrives on its own
• The patterns at a glance
• When you do not need a preStop hook
• FAQ
• Conclusion
• Sources

A release at ten past two

A Spring Boot service checkout in the namespace shop-prod, three replicas, an ingress with the NGINX controller in front. Deployment strategy RollingUpdate, the one that falls out of every Helm chart. A release is nothing more than a new image tag.

At 02:10 exactly such a release goes through. Kubernetes starts a new pod, waits until it reports Ready, and terminates an old one in exchange. Three times in a row. After barely two minutes the new version is everywhere, all pods are running, no container has restarted, no event looks suspicious.

Six hours later eleven reports of aborted orders sit in the support inbox, all from the window between 02:10 and 02:12.

The first search goes into the application logs of checkout, and they are clean. No exception, no error, no aborted transaction. As far as the application is concerned, those eleven orders never existed.

They are in the ingress controller’s log, with status 502 and the note connect() failed (111: Connection refused) while connecting to upstream. So the requests did arrive, just not where anyone looked for them.

What actually happens when a pod is terminated

The common assumption is a sequence: Kubernetes deregisters the pod, waits until no more traffic arrives, and then shuts it down. That would be logical. That is not what happens.

When a pod is deleted, the API server sets a deletion timestamp, and from that moment two processes start at the same time:

Time 0: pod is marked for deletion
   │
   ├─ Track A (on the node)
   │    kubelet runs the preStop hook
   │    then: SIGTERM to PID 1 inside the container
   │    application shuts down
   │
   └─ Track B (across the cluster)
        EndpointSlice controller removes the pod IP
        API server distributes the change
        kube-proxy rewrites its rules on every node
        ingress controller rewrites its upstream list

The Kubernetes documentation is explicit here: removal from the EndpointSlices happens asynchronously to the shutdown. Anyone relying on a sequence is relying on a race.

And track A usually wins that race. A SIGTERM is available on the node immediately. Track B has to travel through the API server, then to every single node, then through the config reload in the ingress controller. Depending on cluster size that is anywhere from a few hundred milliseconds to several seconds.

Important for understanding it: there are two different consumers that both depend on track B, but that send traffic along different paths.

• kube-proxy is responsible for traffic inside the cluster, that is, calls going through a service’s ClusterIP. It rewrites iptables or IPVS rules.
• The ingress controller is responsible for traffic from outside. The NGINX controller bypasses the ClusterIP entirely and keeps its own list of pod IPs as upstreams. kube-proxy is not involved in this path at all.

Both take their truth from the same EndpointSlices, both need their own time to apply it. That is why the problem affects calls from outside as well as calls between two services inside the cluster.

Why the application log stays empty

In the window where the application is already closing up and the upstream entry still exists, the request arrives at a port that no longer accepts. The operating system answers with a TCP reset, and the ingress controller translates that into a 502 for the client.

The application never noticed any of it. There was no request handler, no filter, no interceptor that could have logged anything. The connection setup failed before anything happened at the HTTP level.

That explains the second half of the problem. Not only does the failure occur, it is also invisible in the exact place you look first:

Anyone monitoring only the application side has not a single indicator for this failure. The error rate in Grafana stays at zero because the requests were never counted. This is the same blind spot as with missing time series: a value nobody measures looks exactly like a value that is fine.

Why the failure looks sporadic and hits the writes

At this point comes the objection that shows up in every discussion: NGINX tries the next upstream on a connection error. That is true, and it is the reason the problem often stays undetected for years.

The NGINX ingress controller sets proxy_next_upstream to error timeout by default. A refused connection setup is an error, so the request is handed to the next pod and the client notices nothing.

Except that this does not apply to all requests. NGINX does not retry non-idempotent requests on its own, so no POST, PATCH or LOCK. The reason is sound: for a request that changes something, NGINX cannot know whether the server already processed it before the connection broke. A retry could place a duplicate order. Anyone who wants that anyway has to write non_idempotent into the directive explicitly.

From this follows the failure pattern that arrives in the inbox:

• GET requests are retried silently and land on a healthy pod. Nobody complains.
• POST requests get through and reach the client as a 502.

That is why the problem feels sporadic, why it cannot be reproduced by reloading the page, and why it hits exactly the calls that move money. An order, a payment, a registration: all POST.

Break the 502s in the ingress log down by method and you see it immediately. The share of write requests is far above their share of total traffic.

The reflex that does not help: tightening the readiness probe

The first idea, once the picture is clear, is usually: then the readiness probe simply has to notice faster that the pod is going away.

That misses the point, for two reasons.

First, the readiness probe answers a different question. It decides whether a running pod should currently receive traffic, for instance while it is starting up or while a dependency is briefly unreachable. During termination Kubernetes treats the pod as not ready anyway, as soon as it is marked for deletion. Nobody waits for the result of the next check.

Second, even then it would not be enough. Suppose the probe ran every second and fired immediately: the path from that insight to the rewritten upstream list in the ingress controller is exactly the same as before. You would have sped up the trigger, not the distribution.

The readiness probe is still not worthless, it just solves the other half of the problem. During startup it makes sure a pod only receives traffic once it can answer. During shutdown a different mechanism is needed.

The preStop hook that does nothing

The mechanism is so simple that it looks wrong on first reading: the pod waits before it starts shutting down. During that time it does nothing special, it simply stays fully operational.

lifecycle:
  preStop:
    sleep:
      seconds: 10

What happens is exactly what is wanted: track A is slowed down artificially while track B keeps running at full speed. The kubelet runs the hook and only sends the SIGTERM afterwards. During those ten seconds the pod is still a fully functioning server. It accepts connections, answers requests, and does not even know it is supposed to leave.

In parallel the deregistration travels through the cluster, kube-proxy rewrites its rules, the ingress controller reloads its upstream list. By the time the SIGTERM finally arrives, nobody is sending anything to that pod IP any more.

On the availability of the sleep action, because this regularly gets mixed up:

On clusters before 1.30 the classic route remains:

lifecycle:
  preStop:
    exec:
      command: ["sleep", "10"]

It works the same way but requires a sleep binary in the image. Distroless or scratch images do not have one, and the hook fails silently. That is exactly what the native sleep action is for.

On the duration: ten seconds is a usable starting value for a normal cluster. The right value is the time an endpoint change needs in your setup until it has arrived everywhere. On a small cluster five is enough, with many nodes or an external load balancer with its own health check interval it can be thirty. A cloud load balancer that checks every ten seconds and needs two failures accounts for twenty seconds of delay on its own.

The arithmetic that has to work out

Here is the trap that breaks the whole fix again. The countdown for terminationGracePeriodSeconds does not start after the preStop hook, it starts together with it.

The Kubernetes documentation states it plainly: the grace period covers the total time of the preStop hook plus the regular shutdown of the container.

Which gives:

preStop wait  +  time to drain  <  terminationGracePeriodSeconds

The default for terminationGracePeriodSeconds is 30 seconds. Anyone adding a preStop hook of 10 seconds and leaving it at that has only 20 seconds left for the actual shutdown.

It gets unpleasant in combination with Spring Boot. There spring.lifecycle.timeout-per-shutdown-phase also defaults to 30 seconds. Add it up and you land at 40, which is over the grace period:

10s preStop  +  30s draining  =  40s  >  30s grace period

What happens then is particularly annoying: after 30 seconds the kubelet sends a SIGKILL. That cannot be caught, the process is gone instantly, and the requests still being served break off mid-response. So you have removed the 502s on connection setup and traded them for aborted responses on long-running requests.

The arithmetic therefore has to be set explicitly:

terminationGracePeriodSeconds: 60

With 10 seconds of preStop and 30 seconds of draining that leaves 20 seconds of headroom. The headroom is deliberately generous, because the sequence knows other delays too, such as terminating sidecars.

The application has to play along: graceful shutdown in Spring Boot

The preStop hook protects the requests that have not arrived yet. It does not protect the requests currently being processed. For that the application has to react to SIGTERM properly, and it does not do so on its own.

Spring Boot terminates immediately by default. Running requests are cut off. The switch for it:

server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=30s

With graceful Spring Boot first closes the acceptor on SIGTERM, so it stops accepting new connections, and lets the running requests finish. Only then does the context shut down. timeout-per-shutdown-phase is the upper bound for that.

One detail that likes to tip over: the SIGTERM has to actually reach the Java process. It goes to PID 1 inside the container. If you start your image through a shell wrapper, PID 1 is the shell, and a shell does not forward signals automatically.

# Wrong: the shell is PID 1 and swallows the SIGTERM
ENTRYPOINT java -jar /app.jar

# Right: the Java process is PID 1
ENTRYPOINT ["java", "-jar", "/app.jar"]

The first form is the shell form, Docker starts it through /bin/sh -c. The second is the exec form. If you really need a startup script, end it with exec java -jar /app.jar so the Java process replaces the script and takes over its PID.

You can check this in one line:

kubectl exec -it deploy/checkout -- ps -o pid,comm

If PID 1 is anything other than java, the signal never arrives, and the entire graceful shutdown configuration has no effect.

The complete manifest

All parts together, on one deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  namespace: shop-prod
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: checkout
          image: registry.example.com/checkout:1.4.2
          ports:
            - containerPort: 8080
          lifecycle:
            preStop:
              sleep:
                seconds: 10
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            periodSeconds: 10
            failureThreshold: 6
          startupProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            periodSeconds: 5
            failureThreshold: 30

Three settings in there deserve a justification.

maxUnavailable: 0 makes sure that during the rollout there are never fewer pods available than intended. Together with maxSurge: 1 Kubernetes starts a new pod first and only then terminates an old one. Without it the rollout itself takes away capacity, and the remaining pods get a load spike on top of the switchover.

The separate endpoints for readiness and liveness come from Spring Boot Actuator and have to be enabled:

management.endpoint.health.probes.enabled=true

That makes /actuator/health/readiness and /actuator/health/liveness available. The difference matters: the readiness endpoint includes dependencies such as the database, the liveness endpoint does not. Anyone using the same path for both and checking the database there is building a restart storm: if the database fails, Kubernetes restarts every pod, even though none of them is broken.

The startupProbe covers the start. With failureThreshold: 30 and periodSeconds: 5 the application may take 150 seconds to come up without the liveness probe interfering. For a JVM with a larger Spring context that is a realistic window.

What the preStop hook does not solve

Four limits that show up in practice.

Existing keep-alive connections stay out of scope. The preStop hook protects new connection setups. An already open keep-alive connection from the ingress to the pod stays in place and keeps being used, even after the pod has disappeared from the upstreams. If the application closes it hard during shutdown, a request on it can still fail. Spring Boot’s graceful shutdown handles this case correctly by letting running requests finish.

Long-running requests blow past any window you set. An export that computes for three minutes does not survive a grace period of 60 seconds. Work like that does not belong in an HTTP request that is supposed to survive a rollout, it belongs in a job or a queue.

Persistent connections such as WebSockets and SSE are inevitably cut during termination. No amount of timing helps here, only a client that reconnects.

External load balancers follow their own rhythm. If a cloud load balancer sits in front of the cluster and health-checks the nodes itself, its cadence applies on top. A check interval of ten seconds with two required failures gives twenty seconds during which traffic keeps being sent. The preStop wait then has to exceed that.

One more edge case, because it often comes up while searching: Kubernetes has the conditions serving and terminating in the EndpointSlices. A pod being terminated is terminating but can still be serving. Some proxies use this to send traffic to terminating pods as a last resort when no others are left. That is a lifeline against total outage, not a substitute for clean timing.

Produce the failure before it arrives on its own

The point where everything is decided: a rollout without load proves nothing at all. Almost every rollout is tested in an environment where nobody is using the application, and there the failure naturally does not occur.

The test consists of two terminals. The first runs continuous load, using POST, because GET would be retried:

hey -z 120s -c 20 -m POST 
  -H "Content-Type: application/json" 
  -d '{"sku":"A-1","qty":1}' 
  https://shop.example.com/api/cart

In the second a rollout is triggered while that runs:

kubectl -n shop-prod rollout restart deploy/checkout
kubectl -n shop-prod rollout status deploy/checkout

Afterwards only the status distribution in the hey output counts. Without a preStop hook there are 502s in there, and their number scales with the replica count and the load. With a correctly set hook and graceful shutdown there is nothing but 200.

Cross-check from the ingress log if the numbers are ambiguous:

kubectl -n ingress-nginx logs deploy/ingress-nginx-controller --since=5m 
  | grep ' 502 ' | wc -l

The value before and after the change is the actual result of this article. Everything before it is theory.

A note on the setup: this test belongs in a staging environment with the same replica count and the same ingress configuration as production. A cluster with a single replica shows a different picture, because there is briefly no target at all during the rollout anyway.

The patterns at a glance

When you do not need a preStop hook

Not every workload needs this, and a hook applied everywhere by default extends every shutdown by its wait time.

With a Job or CronJob there is no service and no upstream list. Nobody sends traffic, there is nothing to deregister.

With a queue consumer, a Kafka consumer for example, traffic does not go through a service. The pod fetches its own work. What it needs is a clean exit from the consumer group on SIGTERM, so graceful shutdown, but no wait in front of it.

With an application that has only one replica per node and is reached via hostNetwork, the mechanism does not apply either.

Just check whether the pod IP sits in a list that somebody else maintains. If yes, updating that list takes time, and the pod has to wait it out. If no, graceful shutdown alone is enough.

FAQ

Why does Kubernetes not simply deregister the pod first and terminate it afterwards?
Because both processes are carried out by different components that do not coordinate with each other. The kubelet on the node terminates the pod, the EndpointSlice controller maintains the endpoints. Synchronising them would mean the kubelet waits until every kube-proxy and every ingress controller in the cluster has confirmed. That back channel does not exist, and with several thousand nodes it would not be practical either.

How long does the preStop hook have to wait?
As long as an endpoint change needs in your cluster to reach the last consumer. Five seconds is enough on small clusters, ten is a good starting value, and with an external load balancer that has its own check interval it has to be more than that interval multiplied by the number of required failures. Measure instead of guessing: rollout under load, count 502s, adjust the value.

Does it help to simply raise terminationGracePeriodSeconds?
No, not on its own. The grace period is an upper bound, not a waiting mechanism. Without a preStop hook the SIGTERM goes out immediately, the application shuts down immediately, and the grace period runs into nothing. It still has to be high enough, otherwise the SIGKILL cuts the draining short.

Why does it only hit POST requests?
Because NGINX tries the next upstream on a connection error but, for safety reasons, does not do so for non-idempotent methods. A retried POST could trigger a duplicate order. GET requests are retried silently and therefore never get noticed.

Can I enable non_idempotent instead?
Technically yes, sensibly rarely. It allows NGINX to retry POST requests as well, and you risk duplicate processing if the server had already accepted the first request. The preStop hook fixes the cause, non_idempotent covers it up and creates a new risk.

Does this apply to Traefik, HAProxy or Envoy as well?
Yes, the mechanism is the same because it lives in Kubernetes, not in the proxy. Only the reaction time and the retry behaviour differ. Envoy inside a service mesh usually reacts faster because it gets its endpoints from a dedicated control plane, but even there the distribution is not instantaneous.

Do I need this for calls between two services inside the cluster too?
Yes. There the path runs through kube-proxy instead of the ingress controller, but the race is identical. A calling service then gets a refused connection straight into its HTTP client instead of a 502, which ends as an exception or a retry depending on the client configuration.

Is it enough to set the preStop hook only on the production service?
If that is where the traffic is, yes. It makes more sense to put it into the shared template the deployments are generated from, the base chart for example. Otherwise the service somebody remembered has it and the rest do not.

Conclusion

The cause is a sequence that is not one: deregistration and shutdown run side by side, and the application finishes faster than the cluster manages to pass the information on.

Three settings fix it, and none of them is laborious: a preStop hook that waits, a grace period above wait plus draining, and an application that drains cleanly on SIGTERM instead of stopping instantly. On top of that comes the check that the SIGTERM even reaches PID 1.

Without the fourth one, though, the rest collapses: trigger a rollout under load and count the status codes. Without that step you only know that the new pods come up, and that was never the question.

Start with the service where a lost request hurts most. Run POST load against it, start a rollout, count the 502s. That number is your baseline.

Sources

• Kubernetes, pod lifecycle and termination: kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle
• Kubernetes, container lifecycle hooks: kubernetes.io/docs/concepts/containers/container-lifecycle-hooks
• Kubernetes, KEP-3960 sleep action for the preStop hook: github.com/kubernetes/enhancements
• Kubernetes v1.33, updates to the container lifecycle: kubernetes.io/blog/2025/05/14
• Spring Boot, graceful shutdown: docs.spring.io/spring-boot/reference/web/graceful-shutdown.html
• Spring Boot, Kubernetes probes in the actuator: docs.spring.io/spring-boot/reference/actuator/endpoints.html
• NGINX, proxy_next_upstream: nginx.org/en/docs/http/ngx_http_proxy_module.html
• Ingress NGINX, 502 when scaling pods down: github.com/kubernetes/ingress-nginx/issues/3639

All manifests, configurations and code snippets in this article are my own.