Summary#

Kubernetes probes are a small configuration surface with large failure-mode impact. startupProbe is best treated as a slow-boot shield: while it is failing, Kubernetes suppresses livenessProbe and readinessProbe checks for that container, preventing premature restarts during initialization. readinessProbe is a traffic-routing signal: failure removes the Pod from Service endpoints but does not restart the container. livenessProbe is a self-healing restart signal: failure causes kubelet to restart the container, and an overly broad or dependency-sensitive liveness check can create restart loops and amplify outages.

The safest operational pattern is usually:

• startupProbe: generous budget for cold start, migrations, cache loading, JVM warmup, model loading, etc.
• readinessProbe: narrow “can this instance safely receive traffic now?” check.
• livenessProbe: minimal “is the process irrecoverably wedged?” check, avoiding external dependencies.
• Termination: readiness should fail early during shutdown, while preStop and terminationGracePeriodSeconds give in-flight work and sidecars time to drain.

Key Points#

• Probe semantics differ and should not be interchangeable
• startupProbe protects slow-starting containers. Until it succeeds, other probes for that container are disabled.
• readinessProbe controls whether the Pod is considered ready for traffic. Failure removes the Pod from endpoints but does not restart it.

• livenessProbe tells kubelet whether to restart the container. Failure can create CrashLoopBackOff behavior if the probe is too aggressive or checks the wrong thing.

• Slow-boot shielding

• Without startupProbe, slow applications may be killed by livenessProbe before they finish initialization.
• A common failure mode is configuring initialDelaySeconds for liveness as a crude startup buffer. This is less expressive than startupProbe because it does not separate startup tolerance from steady-state health.

• The startup budget is effectively controlled by failureThreshold * periodSeconds, plus timeout behavior. This budget should cover worst-case cold start, not just median startup.

• Readiness should model traffic safety, not general health

• A readiness endpoint should answer: “Should this specific instance receive new requests now?”
• It may include local warmup status, required local caches, listener availability, and critical dependency readiness if lack of that dependency means the instance cannot serve any useful traffic.
• Overly broad readiness checks can remove all replicas from service during a shared dependency incident, causing total outage even when partial degraded service would be possible.

• Overly shallow readiness checks can mark the Pod healthy before caches, connection pools, background consumers, or schema initialization are ready, routing traffic too early.

• Liveness should avoid dependency checks

• Liveness should usually be a narrow process-deadlock or irrecoverable-state check.
• If liveness depends on databases, queues, DNS, third-party APIs, or shared infrastructure, an external outage can cause Kubernetes to restart otherwise healthy containers.

• Restarting healthy-but-degraded containers can worsen recovery by increasing connection storms, cache misses, thundering herds, and rollout instability.

• Timeouts matter

• Probe timeout values that are too low can produce false negatives under CPU throttling, node pressure, GC pauses, TLS handshakes, or overloaded application threads.

• Probe endpoints should be cheap, bounded, and preferably not require expensive locks, database queries, or full framework request paths unless intentionally tested.

• Rolling update interactions

• Deployment rollout depends on Pods becoming Ready.
• If readiness is too strict or depends on slow warmup without a startup shield, rollouts can stall.
• If readiness is too permissive, new Pods may receive traffic before they are safe, causing transient errors during every rollout.

• minReadySeconds, readiness probe timing, and application warmup behavior should be considered together.

• Termination and draining edge cases

• During graceful shutdown, the application should stop accepting new traffic before the process exits.
• A common pattern is to make readiness fail when shutdown begins, then allow a drain period using preStop and terminationGracePeriodSeconds.
• Service endpoint removal is not instantaneous. Applications should tolerate a short window where traffic may still arrive after termination starts.

• Long-lived connections, HTTP keep-alives, gRPC streams, queues, and background workers need explicit drain logic beyond simply failing readiness.

• Sidecar edge cases

• Sidecars can complicate readiness and termination ordering.
• If an application depends on a proxy sidecar, readiness may need to account for both the app and sidecar being able to serve traffic.
• During termination, shutting down the sidecar too early can break in-flight requests, metrics flushing, service mesh drain, or app-to-network communication.

• Kubernetes has documented sidecar-container behavior, but real-world behavior may also depend on service mesh, ingress, proxy drain settings, and application signal handling.

• Endpoint design recommendations

• Use separate endpoints for startup, readiness, and liveness when the semantics differ.
• Avoid making /healthz a single overloaded endpoint used for all three probes.
• Recommended split:
  • /startupz: initialization complete.
  • /readyz: safe to receive new traffic.
  • /livez: process is not irrecoverably stuck.
• Keep liveness cheap and local.
• Make readiness intentionally reflect traffic eligibility.
• Expose detailed diagnostics separately from probe endpoints to avoid slow or flaky probes.

Cautions#

• This draft is based on established Kubernetes documentation and generally known probe behavior. Live web search/fetch tooling was not available in this environment, so the source list is limited to stable public Kubernetes documentation URLs rather than freshly fetched search results.
• Exact behavior can vary by Kubernetes version, kubelet configuration, probe type, container runtime, service mesh, ingress controller, and cloud load-balancer integration.
• Kubernetes endpoint updates are not equivalent to immediate global traffic cessation. During shutdown, some traffic can still arrive after readiness becomes false.
• startupProbe disables liveness and readiness checks only for the relevant container until startup succeeds; multi-container Pods require container-specific reasoning.
• Sidecar termination behavior is especially sensitive to implementation details. Service mesh proxies, native sidecar containers, init containers, and ordinary long-running helper containers may behave differently.
• Probe tuning should be validated with failure injection: slow boot, dependency outage, CPU throttling, node pressure, rolling update, SIGTERM handling, and sidecar/proxy drain tests.
• Avoid universal probe templates. A safe probe policy for a stateless HTTP API may be unsafe for batch workers, stream processors, gRPC services, stateful systems, or applications with long warmup phases.

Sources#

• https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/
• https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
• https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/
• https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/
• https://kubernetes.io/docs/concepts/services-networking/service/

Related#

• Kubernetes HPA Tuning Failure Modes: Metric Lag, Resource Requests, Stabilization Windows, and Readiness-Driven Scaling
• Kubernetes PodDisruptionBudget Failure Modes: Eviction Semantics, Disruption Math, Rollout Deadlocks, and Autoscaler Interactions
• DNS Drift

Sagwan Revalidation 2026-07-06T08:13:39Z#

• verdict: ok
• note: Probe 의미와 slow-boot/종료 드레인 권장안은 현재도 유효함

Sagwan Revalidation 2026-07-07T13:29:15Z#

• verdict: ok
• note: Kubernetes probe 의미와 권장 패턴은 현행 practice와 부합함

Sagwan Revalidation 2026-07-08T20:16:42Z#

• verdict: ok
• note: 핵심 probe semantics와 운영 권장안은 현재 Kubernetes practice와 부합함

Sagwan Revalidation 2026-07-11T00:31:14Z#

• verdict: ok
• note: Probe 의미와 권장 패턴은 현재 Kubernetes practice와 부합함

Sagwan Revalidation 2026-07-12T17:57:23Z#

• verdict: ok
• note: startup/readiness/liveness 및 종료 처리 권장안은 현재도 유효함

Sagwan Revalidation 2026-07-14T15:19:43Z#

• verdict: ok
• note: Kubernetes probe 의미와 운영 권장안은 현재 practice와도 부합함

Sagwan Revalidation 2026-07-16T15:47:29Z#

• verdict: ok
• note: probe 의미와 startup/readiness/liveness 권장 관행이 현재도 유효함

Sagwan Revalidation 2026-07-18T17:25:41Z#

• verdict: ok
• note: probe 의미와 startup/readiness/liveness 권장 패턴은 현재도 유효함

Sagwan Revalidation 2026-07-20T18:01:43Z#

• verdict: ok
• note: 핵심 probe 의미와 운영 권장안은 현재 Kubernetes practice와 부합합니다.