비동기·분산 시스템에서는 중간 상태와 재시도 가능 오류를 복구 가능한 상태로 모델링하라

ClickHouse와 rustfs PR들에서 서버 시작, heal, bucket migration, scanner handoff, listing, cluster membership 변경 중 transient 상태를 최종 실패처럼 취급해 종료·오판·flaky test가 발생했다. 내구성 있는 작업은 완료 이유, epoch/identity, absence proof, 리더 재발견, post-condition 검증을 명시적으로 기록해 재시도와 handoff 후에도 안

Summary#

Problem#

outdated parts 로딩 중 retryable error가 서버 종료로 이어지거나, bucket sidecar만 쓰이고 metadata 쓰기가 취소되어 반쯤 마이그레이션된 상태가 남았다. bounded listing은 producer가 scan budget 때문에 멈춘 것과 실제 EOF를 구분하지 못했고, scanner checkpoint는 leader handoff 이후 검증 없이 재사용될 위험이 있었으며, Keeper 테스트는 membership 변경 후 old leader에 명령을 보내 flaky하게 실패했다.

Solution#

일시적 오류는 프로세스 치명 오류와 분리해 retry/backoff 대상으로 처리하고, 부분 완료 상태에는 완료 사유와 복구 경로를 저장한다. leader, bucket incarnation, cycle, generation, frontier proof 등 작업 정체성을 검증한 뒤 checkpoint를 재사용하고, heal/admin 작업은 성공 응답 전에 실제 drive/object 상태를 재확인한다. membership 변경 후에는 leader를 재탐색하고, 테스트에서는 비동기 release/detach 완료를 명시적으로 기다린다.

Failure Modes#

retry 가능한 오류를 fatal로 처리해 서버가 불필요하게 종료됨
sidecar 또는 marker만 남은 부분 마이그레이션 상태를 정상 bucket/object로 오인
bounded scan 중단을 EOF로 오해해 IsTruncated=false 같은 잘못된 결과 반환
leader handoff 후 stale leader나 stale checkpoint를 사용해 명령 실패 또는 중복/누락 처리
heal 결과가 완료로 표시되지만 일부 drive가 여전히 unhealthy

Sources#

https://github.com/tensorflow/tensorflow/pull/127549
https://github.com/tensorflow/tensorflow/pull/127458
https://github.com/tensorflow/tensorflow/pull/127449
https://github.com/tensorflow/tensorflow/pull/127550
https://github.com/tensorflow/tensorflow/pull/127623
https://github.com/tensorflow/tensorflow/pull/127622
https://github.com/tensorflow/tensorflow/pull/127621
https://github.com/tensorflow/tensorflow/pull/127617
https://github.com/tensorflow/tensorflow/pull/127459
https://github.com/tensorflow/tensorflow/pull/127561
https://github.com/tensorflow/tensorflow/pull/127494
https://github.com/tensorflow/tensorflow/pull/127533
https://github.com/huggingface/transformers/pull/48907
https://github.com/huggingface/transformers/pull/48926
https://github.com/huggingface/transformers/pull/48923
https://github.com/huggingface/transformers/pull/33737
https://github.com/huggingface/transformers/pull/33736
https://github.com/huggingface/transformers/pull/47992
https://github.com/huggingface/transformers/pull/35778
https://github.com/huggingface/transformers/pull/35321
https://github.com/huggingface/transformers/pull/48593
https://github.com/ClickHouse/ClickHouse/pull/117981
https://github.com/ClickHouse/ClickHouse/pull/118971
https://github.com/ClickHouse/ClickHouse/pull/113909
https://github.com/ClickHouse/ClickHouse/pull/100347
https://github.com/ClickHouse/ClickHouse/pull/116136
https://github.com/ClickHouse/ClickHouse/pull/120768
https://github.com/ClickHouse/ClickHouse/pull/119892
https://github.com/ClickHouse/ClickHouse/pull/119103
https://github.com/ClickHouse/ClickHouse/pull/116208
https://github.com/ClickHouse/ClickHouse/pull/114628
https://github.com/ClickHouse/ClickHouse/pull/120770
https://github.com/ClickHouse/ClickHouse/pull/120769
https://github.com/ClickHouse/ClickHouse/pull/118523
https://github.com/ClickHouse/ClickHouse/pull/116122
https://github.com/rustfs/rustfs/pull/8008
https://github.com/rustfs/rustfs/pull/8004
https://github.com/rustfs/rustfs/pull/8001
https://github.com/rustfs/rustfs/pull/8000
https://github.com/rustfs/rustfs/pull/7994
https://github.com/rustfs/rustfs/pull/7996
https://github.com/rustfs/rustfs/pull/7993
https://github.com/rustfs/rustfs/pull/7991
https://github.com/rustfs/rustfs/pull/7990
https://github.com/rustfs/rustfs/pull/7981
https://github.com/rustfs/rustfs/pull/7986
https://github.com/rustfs/rustfs/pull/7982
https://github.com/rustfs/rustfs/pull/7987
mined_at: 2026-09-18T12:42:45Z

Sagwan Revalidation 2026-09-18T13:21:42Z#

verdict: ok
note: 복구 가능한 상태·정체성 검증·재시도 권장은 여전히 최신 관행이다.