Health Probes & Graceful Lifecycle

Three probes, three jobs

graph TD
START["Pod starts"] --> SU["startupProbe<br/>(generous: still booting?)"]
SU -->|passes| LR["liveness + readiness begin"]
LR --> LV["livenessProbe<br/>process wedged? → RESTART"]
LR --> RD["readinessProbe<br/>can serve now? → add/remove from LB"]

• Liveness — “is the process wedged?” Fails → Kubernetes restarts the container. Keep it cheap and self-contained; never check external dependencies here.
• Readiness — “can I serve traffic right now?” Fails → Kubernetes removes the Pod from the Service but leaves it running. Put dependency checks (DB reachable, caches warm) here.
• Startup — for slow boots: gives the app time to initialize before liveness/readiness apply, so a long startup isn’t mistaken for a hang.

See it: liveness and readiness handlers

/healthz is cheap and always-200 while the process runs; /readyz reflects whether dependencies are ready. This runs here via httptest — readiness starts failing (503) and flips to 200 once initialized:

package main

import (
"fmt"
"net/http"
"net/http/httptest"
"sync/atomic"
)

func main() {
var ready atomic.Bool // flipped true once dependencies are up

mux := http.NewServeMux()
// Liveness: only reports the process is running. No dependency checks!
mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
	w.WriteHeader(http.StatusOK)
	fmt.Fprintln(w, "ok")
})
// Readiness: 503 until ready, 200 after — controls load-balancer traffic.
mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
	if ready.Load() {
		w.WriteHeader(http.StatusOK)
		fmt.Fprintln(w, "ready")
	} else {
		w.WriteHeader(http.StatusServiceUnavailable)
		fmt.Fprintln(w, "not ready")
	}
})

probe := func(path string) int {
	rec := httptest.NewRecorder()
	mux.ServeHTTP(rec, httptest.NewRequest("GET", path, nil))
	return rec.Code
}

fmt.Println("before init:  /healthz", probe("/healthz"), " /readyz", probe("/readyz"))
ready.Store(true) // dependencies connected, caches warmed
fmt.Println("after init:   /healthz", probe("/healthz"), " /readyz", probe("/readyz"))
}

Liveness is 200 throughout (the process is fine); readiness is 503 until initialization completes, so Kubernetes only routes traffic once the app can actually serve it.

Graceful shutdown

On a deploy or scale-down, Kubernetes sends SIGTERM and removes the Pod from the Service — but there’s a propagation delay. The robust sequence avoids dropping in-flight requests (fenced — needs a real server and signals):

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt)
defer stop()

go srv.ListenAndServe()
<-ctx.Done() // SIGTERM received

ready.Store(false)          // 1. fail readiness so the LB stops sending new traffic
time.Sleep(3 * time.Second) // 2. wait for the LB to notice (propagation delay)

shutCtx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
defer cancel()
srv.Shutdown(shutCtx)       // 3. stop accepting, drain in-flight requests, exit

The order matters: fail readiness first, pause, then Shutdown. That drains traffic to other replicas before this Pod stops accepting — eliminating the intermittent 502s that plague naive shutdowns. See hardening services for the server-side Shutdown details.

See also

• Kubernetes basics — where the probe fields live in the Pod spec.
• Hardening services — server timeouts and Shutdown internals.
• Configuration — probe paths and grace periods as config.
• context (concurrency) — the cancellation that drives graceful shutdown.

Next: feeding the service its configuration the 12-factor way — configuration.

Related topics