Skip to content

Stopped and waiting runs ​

tack run and tack resume exit with code 3 when a run stops for the operator, and with code 4 when it waits for Git. The ledger's reason says why: RFC#0003, Requirement 10. This runbook gives the way on for each reason.

NameMeaning
RUN, OTHERthe run, as tack status -f workload.yaml lists it, and another run of the workload
F, Tthe run's from and to sites; F_CTX, T_CTX their kube contexts
F_CLUSTER, T_CLUSTER, NStheir CloudNativePG Clusters and its namespace, from the Workload file
SITE, SITE_CTX, CLUSTERthe site the message names, its kube context and its Cluster
KIND, NAMEan object the message names
LEDGER_NSthe ledger namespace, tack-system by default
URL, BRANCHthe Workload's Git URL and branch
BASE, HEAD, FIXthe commits the message names, and a commit that fixes Git

First ​

CheckHow
No other tack process runs RUN.Ask every operator, and check every script that runs tack. A Running ledger does not tell a live process from a dead one.
Where the run stoppedtack status RUN -f workload.yaml: state, reason, message, and each step's state and last result.
The workload has a primaryWhen a planned run demoted F and T does not run as primary yet, go to no-primary-after-demotion.md first.

While a run is stopped, its objects stay held. Flux applies nothing to them, not even later Git changes. A run left Running by a process that ended goes on with tack resume RUN -f workload.yaml.

GitUnreachable ​

WhenGit cannot be fetched or pushed at the Git step. State Waiting, exit code 4; git-commit stays Pending. Every fetch failure counts except a missing branch, including a refused credential or a wrong URL (RFC#0006, Requirement 3).
CheckFrom the same host and user: git ls-remote URL BRANCH.
Way onFix Git, or the access to it, then tack resume RUN -f workload.yaml. During a site loss: break-glass.

GitChanged ​

WhenThe branch moved away from the run's base before the run's commit, or the push was rejected because the branch moved. Flux at a site applied a revision other than the run's base, the run's commit, or the head accepted after it. Before a release, the branch moved past the run's commit, or no longer contains it. Nothing at a site is released before its check.
CheckThe message names the base or the run's commit, and the head. Review git log --oneline BASE..HEAD, and git diff BASE HEAD for the files of the run's git-commit step.
Way onWhen the new commits leave the run's fields alone, or set them as live has them: tack resume RUN -f workload.yaml --accept-git-head HEAD. When the message names one head, HEAD stands for that head, and any other commit is refused. A head pushed after the stop needs its own stop and review. When the new commits change a field of the run: fix Git first, then accept the head that fixes it. After the run's commit, the accepted head must contain that commit. What the flag changes for the rest of the run: operations.md.

GitDiffersFromLive ​

WhenThe check before release, verify-git-SITE, found that Git, rendered as Flux renders it, differs from live. Nothing at that site is released.
CheckThe message names each path. For the run's own fields, it gives Git's value and live's.
Way onCommit a fix to Git, then tack resume RUN -f workload.yaml --accept-git-head FIX, with FIX the commit that fixes it. The fix moves the branch, so a plain tack resume stops with GitChanged and names the head to accept.

TokenMissing ​

MessageWhenWay on
Cluster NS/F_CLUSTER has no value at {.status.demotionToken}capture-token found no token at F, though wait-demoted-F saw one before. CloudNativePG clears the token once the Cluster is no longer a replica, such as after F was promoted again by hand.Check F: kubectl --context F_CTX -n NS get cluster F_CLUSTER -o jsonpath='{.spec.replica.primary} {.status.instancesReportedState.*.isPrimary}{"\n"}'. When F runs as primary again, the run must not go on: abandon-run.md.
step promote-T: Cluster NS/F_CLUSTER now has another value at {.status.demotionToken}, not the value step capture-token captured as demotionToken, or no valueThe read before the promotion. F was demoted again after the run captured its token, such as by hand after a promotion by hand. T would promote with the token of a demotion that a later demotion replaced.The run must not go on: abandon-run.md. Do not copy F's new token into the ledger.
the ledger has no value for demotionTokenA step needs a value that the ledger's status.captured lacks. When capture-token is Succeeded, the value was lost after the capture, such as by an edit of the ledger. capture-token never runs again, so every resume stops the same way.Restore the value from F, as below, then resume.

Restore the token only when all of these hold:

  • F has stayed demoted since the run demoted it, so its token is still the one T must promote with.
  • T has not promoted.
sh
# F declares T_CLUSTER and reports no isPrimary; its token is set.
kubectl --context F_CTX -n NS get cluster F_CLUSTER -o jsonpath='{.spec.replica.primary} {.status.instancesReportedState.*.isPrimary}{"\n"}'
# T declares F_CLUSTER, has no lastPromotionToken, and reports no isPrimary.
kubectl --context T_CTX -n NS get cluster T_CLUSTER -o jsonpath='{.spec.replica.primary} {.status.lastPromotionToken} {.status.instancesReportedState.*.isPrimary}{"\n"}'

TOKEN=$(kubectl --context F_CTX -n NS get cluster F_CLUSTER -o jsonpath='{.status.demotionToken}')
test -n "$TOKEN" && kubectl --context T_CTX -n LEDGER_NS patch sitefailover RUN --subresource=status --type=merge \
  -p "{\"status\":{\"captured\":{\"demotionToken\":\"$TOKEN\"}}}"
unset TOKEN
tack resume RUN -f workload.yaml
FactDetail
Read the token from F onlyT can carry a token of its own from an earlier demotion, which an earlier run already used. A used token makes CloudNativePG hold the Cluster as unrecoverable (RFC#0004, Motivation).
PermissionThe patch needs patch on sitefailovers/status, which is beyond Tack's own rows (operations.md).

FencedSiteIsPrimary ​

A site answers as primary, or declares itself primary, where going on would leave two primaries. The run stops before the step, and the step keeps its state. The stop changes nothing. Each tack resume reads the site again, and stops again while the site is primary.

The messageWhenWay on
step S: site SITE answers as primary (...); it should be fenced. Check it, then resume with --fence SITE, or declares itself primaryThe read before the site's steps, for a site whose steps can wait for it, such as a lost site that is back after a site-loss run.tack resume RUN -f workload.yaml --fence SITE. Tack holds the site's objects, then declares the new primary there. Then follow rebuild-old-primary.md.
step S: site SITE ... Promoting T gives two primaries, and no step of this run is left to fence SITE, so --fence cannot helpThe read of the other sites, before a Failover step after F's demotion or fence, up to promote-T. The step that fences the site is done, and the site was made primary again outside the run. Example: F was promoted again by hand after the run demoted or fenced it (no-primary-after-demotion.md, option 2).The run's plan no longer fits the sites. Abandon the run (abandon-run.md), then plan a new run if the switch is still needed. Demoting F by hand does not help: the resume then stops with TokenMissing before promote-T, since F's token is no longer the one the run captured.
step S: site SITE ..., and so does T, which this run promoted: two primaries. No step of this run is left to fence SITE, so --fence cannot helpThe same read, after promote-T. The site was made primary again outside the run, after the run fenced it. Both sites may take writes.Pick the site that keeps the writes. The other site needs a rebuild (rebuild-old-primary.md), and the writes it took are lost. First stop writes at the other site: declare the kept site's Cluster primary there, by hand. Then abandon the run (abandon-run.md).
step S: site SITE ..., and this run fences it only after it promotes T, or ..., and so does T, which this run promoted, until this run fences SITEThe read of the other sites, before or after promote-T. A third site is primary, and the run declares T primary there only after the promotion.Find out why SITE is primary. To go on anyway: tack resume RUN -f workload.yaml --fence SITE. SITE and T are then both primary until the run declares T at SITE. Writes that SITE takes meanwhile are lost to T, and SITE needs a rebuild (rebuild-old-primary.md); the run records the follow-up Rebuild for it. Otherwise, abandon the run (abandon-run.md).
CheckHow
Which sites take writesAt each site: kubectl --context SITE_CTX -n NS get cluster CLUSTER -o jsonpath='{.spec.replica.primary} {.status.instancesReportedState.*.isPrimary}{"\n"}'. With Strait: kubectl --context SITE_CTX -n NS get siteservice NAME -o jsonpath='{.status.route.members}'.
Writes at riskA site that was primary on its own, such as a lost site, may hold writes the new primary lacks. The rebuild drops them.

HeldByAnotherRun ​

WhenAn object of the run is held by another run, or by hand. A hold by hand is kustomize.toolkit.fluxcd.io/reconcile: disabled without failover.tako.id/held-by, or set as a label. Tack finds it at Hold, before a patch, at release, or again after release.
Another runRun tack status OTHER -f workload.yaml. If the other run took this run over, this run cannot resume: follow the other run. Otherwise one of the two runs must end. Either finish the other run, and start a new run with --take-over RUN if the switch is still needed. Or abandon the other run (abandon-run.md), and start a new run with --take-over OTHER.
By handFind out who set the hold and why. Tack never removes a hold it did not set. Remove the hold only once Git equals live in that object: abandon-run.md. Then resume.

FluxChanged ​

WhenThe check after release, verify-noop-SITE, found that an object's metadata.generation changed after its release. So Flux applied something to it.
CheckFirst check that the switch holds. At every site, kubectl --context SITE_CTX -n NS get cluster CLUSTER -o jsonpath='{.spec.replica.primary} {.status.instancesReportedState.*.isPrimary}{"\n"}' shows the new primary declared, and true only at T. Then find what changed: the commits since the run's commit for the object's file, and kubectl --context SITE_CTX -n NS get KIND NAME --show-managed-fields -o yaml.
Way onWhen the change is expected, such as a commit pushed after the run's commit that changed the object: tack resume RUN -f workload.yaml --accept-git-head HEAD. A plain tack resume stops again, since the generation stays changed. When a site declares the old primary again, Git undid the switch there. Restore the run's values in Git at once. Treat any site that ran as primary since then as after a site loss: rebuild-old-primary.md.

SiteMismatch ​

WhenA site's context reaches a different cluster than the one the run recorded: the UID of its kube-system namespace differs. Nothing changed at that site.
Way onIn the kubeconfig, point the context at the site's cluster, then tack resume RUN -f workload.yaml. If the site's cluster was rebuilt, the run cannot use it: start a new run with --take-over RUN.

StepFailed ​

Any other error. Fix the cause the message names, then tack resume RUN -f workload.yaml. The step runs again.

MessageCauseWay on
site SITE is unreachable: no answer to 3 checksThe site does not answer, and the step cannot wait for it: the site is to, or it is from in a planned run.Once the site answers, resume. After a demotion: no-primary-after-demotion.md.
site SITE: ... answered its check with an error, so it is not counted as lostThe site's check failed with an error, not with no answer. One example: a refused credential.Fix the kubeconfig or the credential, then resume.
... is forbidden: User ... cannot ...A missing permission.Grant it (operations.md), then resume.
A Wait that timed out, such as wait-demoted-F, wait-promoted-T or wait-pooler-TThe workload did not reach the state in time. When wait-demoted-F times out, F may never publish a token: CloudNativePG skips the token when F's shutdown outlasts switchoverDelay (RFC#0004, Limitations).Check the object the step names. Once it reaches the state, resume: the wait passes. Without a token from F: no-primary-after-demotion.md.
step S: the read before the first Failover step: driver cnpg no longer allows the run: ...No Failover step ran yet, and the sites are no longer what the plan needs. Example: a primary was switched by hand while the run was stopped in Hold. Nothing of the switch ran.When the sites are back as the plan read them, resume. Otherwise, abandon the run (abandon-run.md) and plan again.
step git-commit: ...: the run would write ... The ledger's Git step does not match its WorkloadThe Git step names a file, path or document other than the one where the run's Workload maps the field, or than the manifest that defines its object (RFC#0006, Requirement 15). Causes: a ledger written outside Tack, or a manifest moved in Git after the plan. Nothing was committed.Find out who wrote the ledger (operations.md). The run must not go on: abandon-run.md.
verify-noop-SITE, with a Kustomization not ReadyFlux did not end a reconcile with ReconciliationSucceeded within timeouts.flux.Fix the Kustomization, then resume.
verify-noop-SITE: the Kustomization NS/NAME at site SITE is suspended (spec.suspend)The Kustomization of a released object is suspended. So Flux applies nothing to it, and its Ready condition is old.Find out who suspended it, and why. Resume the Kustomization (flux --context SITE_CTX resume kustomization NAME -n NS), then resume the run.
routing-SITEStrait did not route to T's member alone within timeouts.routing. Every release is done.Check the SiteService at that site, then resume.
... is no longer held by this run, so Flux may apply Git over the changeSomeone removed a hold during the run.Find out who, and why. If the workload was also changed by hand, such as F promoted again, the plan may be stale: abandon the run (abandon-run.md). Only when nothing else changed, hold the object again as the message says, then resume. A resume after F was promoted again stops before promote-T with FencedSiteIsPrimary instead.

Released under the Apache License 2.0.