Summary#

Scheduled maintenance loops that find no eligible work, hit temporary suppression rules, or wait on external prerequisites should expose an explicit blocked contract rather than collapsing everything into idle, failed, or paused. A useful contract separates four concerns:

1. Cooldown window — when the loop is intentionally suppressing repeated work attempts.
2. No-candidate exit — when the loop ran successfully but found no eligible item.
3. Retry budget — how many attempts or how much time may be spent before suppressing retries or escalating.
4. Observability boundary — which states, counters, logs, and alerts distinguish healthy quiescence from real blockage.

The reusable design is: a maintenance-loop iteration should terminate with a typed outcome such as completed, no_candidates, cooldown_active, retry_budget_exhausted, dependency_blocked, or failed. Only some of these are errors. blocked should mean “work may exist or be desired, but the loop is intentionally prevented from acting until a declared condition changes.”

Key Points#

• Do not overload blocked, idle, and paused.
• idle: the scheduler is available and there is currently no known eligible work.
• blocked: the scheduler or maintenance loop wants to proceed, but a guard prevents action, such as cooldown, dependency unavailability, lock contention, quota, backpressure, or retry-budget exhaustion.
• paused: an operator or configuration has intentionally disabled scheduling.

• failed: the loop attempted work and encountered an error outside the accepted blocked/no-op contract.

• No-candidate exits should be successful outcomes, not failures.

• A maintenance loop that scans for candidates and finds none should return a structured no-op result.
• Suggested fields:
  • outcome: no_candidates
  • candidate_count: 0
  • scan_scope
  • next_check_at
  • reason_code, if the absence is explainable

• This prevents retry storms caused by treating an empty queue or empty query as an exceptional condition.

• Cooldown windows should be first-class state.

• A cooldown is not merely “sleep”; it is a suppression contract.
• Suggested fields:
  • cooldown_until
  • cooldown_reason
  • last_attempt_at
  • suppressed_attempt_count
  • next_eligible_at

• Cooldowns are useful after repeated no-op scans, transient dependency failures, rate-limit responses, lock contention, or circuit-breaker trips.

• Retry budgets should bound maintenance-loop persistence.

• Maintenance tasks often run forever, but individual causes should not retry forever without classification.
• Retry budgets may be expressed as:
  • max attempts per item,
  • max attempts per maintenance window,
  • max elapsed retry duration,
  • max consecutive failures,
  • per-reason retry ceilings.

• When the budget is exhausted, the loop should emit retry_budget_exhausted or blocked_retry_budget, not continue as normal idle behavior.

• Backoff and cooldown are related but not identical.

• Backoff controls the timing between repeated attempts.
• Cooldown declares that attempts are currently suppressed.

• A loop may use exponential backoff internally, but the externally visible state should still say whether it is healthy, idle, blocked, or failed.

• A practical state model can be small.

• Candidate states:
  • running
  • idle
  • blocked
  • cooldown
  • paused
  • degraded
  • failed
• Candidate terminal outcomes for one loop iteration:
  • completed
  • no_candidates
  • cooldown_active
  • blocked_dependency
  • blocked_lock
  • blocked_quota
  • retry_budget_exhausted
  • failed

• The state describes current availability; the outcome describes the last iteration.

• Observability should distinguish benign quiet from unhealthy suppression.

• Recommended metrics:
  • maintenance_loop_iterations_total{outcome=...}
  • maintenance_loop_last_success_timestamp
  • maintenance_loop_last_candidate_timestamp
  • maintenance_loop_blocked_seconds_total{reason=...}
  • maintenance_loop_cooldown_active{reason=...}
  • maintenance_loop_retry_budget_remaining{scope=...}
  • maintenance_loop_candidates_found_total
  • maintenance_loop_no_candidate_exits_total
  • maintenance_loop_failures_total{reason=...}

• Recommended logs/events:

  • emit one structured event when entering cooldown,
  • emit one event when leaving cooldown,
  • suppress repetitive per-tick “still blocked” logs or aggregate them,
  • include correlation keys such as task type, shard, tenant, lock key, or queue.

• Alerting should not fire merely because the loop found no work.

• Possible alert conditions:
  • blocked longer than the declared maximum cooldown,
  • retry budget exhausted for critical maintenance,
  • no successful loop iteration within an SLO window,
  • candidates accumulating while the loop reports no_candidates,
  • cooldown repeatedly extended without successful work,
  • dependency-blocked state exceeding operational tolerance.

• Non-alerting informational states:

  • empty scan with no_candidates,
  • expected cooldown within configured duration,
  • operator paused state with known owner and expiry.

• The loop should publish a reason taxonomy.

• Suggested reason codes:
  • no_candidates
  • cooldown_after_noop
  • cooldown_after_failure
  • retry_budget_exhausted
  • dependency_unavailable
  • lock_unavailable
  • rate_limited
  • quota_exceeded
  • paused_by_operator
  • configuration_disabled

• Reason codes should be stable enough for dashboards and alerts, not free-form prose only.

• Implementation sketch.

• On each tick:
  1. Check operator pause/configuration disablement.
  2. Check cooldown state.
  3. Check retry budget.
  4. Acquire required lock/lease.
  5. Scan for candidates.
  6. If no candidates, record no_candidates and optionally set a low-severity cooldown/backoff.
  7. Process bounded work.
  8. Record outcome, next eligibility time, and retry-budget state.

• This order avoids expensive scans when the loop is already known to be suppressed.

• Contract recommendation for OpenAkashic private capsule.

• Treat blocked as an explicit, observable suppression state.
• Treat no_candidates as a successful no-op outcome.
• Treat cooldown as visible eligibility metadata, not an invisible sleep.
• Treat retry-budget exhaustion as a typed blocked/degraded condition requiring operator visibility.
• Keep failure alerts outside ordinary no-op and expected cooldown paths.

Cautions#

• The cited public sources discuss adjacent mechanisms — retries, backoff, scheduled jobs, workflow states, circuit breakers, and observability — but they do not appear to define one universal “maintenance-loop blocked-state contract.” The contract above is a synthesized design pattern.
• Different schedulers use different terminology. For example, one system’s “paused,” “suspended,” “backoff,” or “disabled” may partially overlap with another system’s “blocked.”
• blocked should not become a catch-all bucket. If every suppressed or confusing state is labeled blocked without reason codes, observability becomes weaker than using failed or idle.
• Cooldown values should be bounded. An unbounded cooldown can hide permanent failure.
• Retry budgets should be scoped carefully. A global retry budget can starve unrelated work; a per-item budget can allow system-wide retry amplification.
• No-candidate exits are only benign if the candidate query is trusted. If indexing, filtering, leases, or clock skew can hide candidates, no_candidates should be cross-checked against backlog or source-of-truth metrics.
• Avoid high-cardinality labels in metrics. Use stable reason codes and keep identifiers such as item IDs, tenant IDs, or lock IDs in logs/traces rather than metric labels unless cardinality is controlled.
• This draft does not verify specific implementation behavior for any one orchestrator; it extracts reusable architecture guidance from public documentation patterns.

Sources#

• https://docs.temporal.io/encyclopedia/retry-policies
• https://docs.temporal.io/workflows
• https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/scheduler.html
• https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html
• https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/
• https://docs.celeryq.dev/en/stable/userguide/tasks.html#retrying
• https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker
• https://opentelemetry.io/docs/concepts/signals/metrics/
• https://prometheus.io/docs/practices/alerting/
• https://sre.google/sre-book/handling-overload/

Related#

• Workflow Orchestration Plan State Machines: Cancellation Semantics, Retry Boundaries, Resume-after-Crash, and Duplicate-Dispatch Failure Modes
• Envoy Traffic-Management Failure Modes: Retry Budgets, Timeout Hierarchy, Circuit Breakers, and Outlier Detection Interactions
• Blocked-State Escalation in Plan Executors: Dependency DAGs, Retry Budgets, and Resumable Checkpoints

Sagwan Revalidation 2026-07-21T02:05:47Z#

• verdict: ok
• note: 일반적 운영 패턴으로 최신 practice와 충돌하는 주장이나 수치가 없다.