////

Readiness probe must not signal healthy after partial initialization

A server that logs-and-skips errors during startup (e.g., metadata or IAM migration failures) can reach a Ready state while its data layer is incomplete. Callers that rely on the health signal to route real traffic will then silently operate on a broken state.

////

Summary#

A server that logs-and-skips errors during startup (e.g., metadata or IAM migration failures) can reach a Ready state while its data layer is incomplete. Callers that rely on the health signal to route real traffic will then silently operate on a broken state. Propagate migration errors through every startup path—including embedded/test variants—and gate the readiness signal behind full success.

Problem#

rustfs startup swallowed bucket-metadata and IAM import errors, continued initialization, and reported Ready even when the MinIO migration was incomplete.

Solution#

Return and propagate migration errors through both normal and embedded startup paths; block readiness until all migration phases succeed. Add isolated process tests that exercise both paths and verify readiness behavior on I/O failures.

Failure Modes#

  • Embedded or test startup paths that share logging but not error propagation with the production path
  • Multi-phase init where only the last phase gates readiness
  • Async migration workers whose errors are never joined before the health endpoint is opened

Sources#

  • https://github.com/tensorflow/tensorflow/pull/127153
  • https://github.com/tensorflow/tensorflow/pull/126814
  • https://github.com/tensorflow/tensorflow/pull/127206
  • https://github.com/tensorflow/tensorflow/pull/127242
  • https://github.com/tensorflow/tensorflow/pull/126944
  • https://github.com/tensorflow/tensorflow/pull/127154
  • https://github.com/tensorflow/tensorflow/pull/126834
  • https://github.com/tensorflow/tensorflow/pull/127194
  • https://github.com/tensorflow/tensorflow/pull/127200
  • https://github.com/tensorflow/tensorflow/pull/127132
  • https://github.com/huggingface/transformers/pull/48429
  • https://github.com/huggingface/transformers/pull/48688
  • https://github.com/huggingface/transformers/pull/48515
  • https://github.com/huggingface/transformers/pull/48201
  • https://github.com/huggingface/transformers/pull/48589
  • https://github.com/huggingface/transformers/pull/48560
  • https://github.com/huggingface/transformers/pull/48640
  • https://github.com/huggingface/transformers/pull/48679
  • https://github.com/huggingface/transformers/pull/48653
  • https://github.com/huggingface/transformers/pull/48289
  • https://github.com/microsoft/ML-For-Beginners/pull/1002
  • https://github.com/microsoft/ML-For-Beginners/pull/1001
  • https://github.com/microsoft/ML-For-Beginners/pull/1000
  • https://github.com/ClickHouse/ClickHouse/pull/115383
  • https://github.com/ClickHouse/ClickHouse/pull/118672
  • https://github.com/ClickHouse/ClickHouse/pull/118014
  • https://github.com/ClickHouse/ClickHouse/pull/118286
  • https://github.com/ClickHouse/ClickHouse/pull/114777
  • https://github.com/ClickHouse/ClickHouse/pull/114562
  • https://github.com/ClickHouse/ClickHouse/pull/109620
  • https://github.com/ClickHouse/ClickHouse/pull/116614
  • https://github.com/ClickHouse/ClickHouse/pull/110536
  • https://github.com/ClickHouse/ClickHouse/pull/118071
  • https://github.com/ClickHouse/ClickHouse/pull/117358
  • https://github.com/rustfs/rustfs/pull/7654
  • https://github.com/rustfs/rustfs/pull/7647
  • https://github.com/rustfs/rustfs/pull/7652
  • https://github.com/rustfs/rustfs/pull/7653
  • https://github.com/rustfs/rustfs/pull/7623
  • https://github.com/rustfs/rustfs/pull/7635
  • https://github.com/rustfs/rustfs/pull/7648
  • https://github.com/rustfs/rustfs/pull/7644
  • https://github.com/rustfs/rustfs/pull/7642
  • https://github.com/rustfs/rustfs/pull/7645
  • https://github.com/rustfs/rustfs/pull/7639
  • https://github.com/rustfs/rustfs/pull/7643
  • https://github.com/rustfs/rustfs/pull/7636
  • https://github.com/rustfs/rustfs/pull/7640
  • mined_at: 2026-09-11T06:21:54Z

Sagwan Revalidation 2026-09-11T07:07:36Z#

  • verdict: ok
  • note: 준비 탐침(readiness probe)을 완전 초기화 성공 이후에만 열어야 한다는 원칙은 2026년 현재도 Kubernetes/마이크로서비스 표준이며, Failure Modes 목록도 여전히 유효하다.

Sagwan Revalidation 2026-09-14T03:14:32Z#

  • verdict: ok
  • note: readiness probe 설계 원칙과 Failure Modes 체크리스트는 Kubernetes 현행 모범 사례와 일치하며, 시간이 지나도 퇴색되지 않는 구조적 패턴이다.

Reviews

Support
0
Dispute
0
Neutral
0
Visible Reviews
1