Skip to content

Upgrade ​

Upgrade the CRD before the CLI, and every operator's CLI together.

Upgrade within v1 ​

Upgrade in this order:

  1. Finish or stop every run. For each workload, tack status -f workload.yaml lists every run as Succeeded, or as Stopped or Waiting with no tack process running it. A run that resumes after the upgrade runs its remaining steps by the new CLI's rules.
  2. The CRD at every site, the way it was installed: the helm upgrade --install of Install, or kubectl apply --server-side of the new CRD. Do this before any operator runs the new CLI against the site. A CRD older than the CLI drops the fields it does not know, without an error. At such a site, tack run refuses to create a ledger, and status writes lose those fields.
  3. Every operator's CLI together, and every script that runs tack. When a CLI writes a ledger, it drops the status fields it does not know. From v1.3.0, each write records the CLI's version in status.writtenBy. A CLI of an older release then refuses to resume or abandon that run, and names the version to use (RFC#0003, Requirement 24). A build whose version names no release refuses too. An example is the version dev, from go build -buildvcs=false or from go install of v1.4.0 or older. go install of a later release's tag takes that tag's version. A CLI older than v1.3.0 does not check.

Each release also needs:

FromToCRDAlso before the new CLI runs
v1.6.xthe next release (its version is set when it is tagged)new optional fields: spec.acceptBehind, status.fork (with status.fork.lastCommit), status.writes, status.steps[].earlier, wait.stall of a step, status.followUps[].snapshot and .archive; in the Workload snapshot, cnpg.snapshot, timeouts.snapshot and timeouts.archiving; step types Create and RecordApply the CRD first. A CRD of v1.6.x drops spec.acceptBehind, and tack run --accept-behind refuses to create a ledger there. A CRD that lacks status.fork.lastCommit drops it from each status write, without an error, so the ledger loses the replay time that the run read. Grant get on pods and pods/proxy in each member's namespace (row 18 of docs/operations.md; the chart's objects Role has it where rbac.objects names clusters). The plan uses it to read each site's WAL position, and each replica's replay lag from its metrics. Without it, the plan only warns. To have a site-loss plan say when the new primary's last replayed commit was, add the opt-in replay-time query to each Cluster (docs/workload.md). A site-loss plan refuses a target that is behind another replica that answers, unless --accept-behind is passed (RFC#0004, Requirement 14). A site-loss run records the fork point and the old primary's WAL archive. With cnpg.snapshot, it also takes a snapshot of the old primary, and needs get and create on backups (row 19). A ledger that v1.6.x wrote resumes with the steps it planned. A planned switchover is refused when two sites archive WAL to the same destination; a site-loss plan warns instead (docs/workload.md). For the plan to check this, grant get on the barman-cloud objectstores that the Clusters name (row 20). After each promotion, the run waits up to timeouts.archiving for the new primary to archive WAL. If that time runs out, the run warns and records the follow-up CheckArchiving. Each run records when writes stopped and came back (status.writes). Each step keeps its earlier attempts (status.steps[].earlier). In a planned run, the wait for the promotion stops once replay has stood still for 2 minutes short of the token.
v1.5.xv1.6.0no new field; the column TakenOverBy in kubectl get sfo, lastWith a CRD of v1.5.x, everything works except that column. Tack obtains each site's credentials before its checks and before each step of a run. It runs one credential plugin at a time, with no time limit; Ctrl-C or SIGTERM ends the wait. If a site's credentials cannot be obtained, the command stops, or the run stops before that site's step. Such a site is never counted as lost (RFC#0007, Requirement 12). A planned switchover is refused when a site's Cluster does not name the new primary in spec.externalClusters. In that case a site-loss plan warns, and the run stops at that site's step (RFC#0004, Requirement 12). tack prune deletes a run whose follow-up Rebuild is done. For that, it needs get on the workload's objects (row 7 of docs/operations.md). Without it, prune keeps that run. Prune keeps more than --keep runs when the clocks of two sites cannot order them.
v1.4.xv1.5.0unchangedGrant get, create and update on leases in the ledger namespace (row 17 of docs/operations.md; the chart's ledger Role has it with rbac.create). Without it, a run holds no Lease and says so once. tack abandon and tack run --take-over now refuse a run that a process may still run. With --force, they go on. A build whose version names no release, such as dev, refuses ledgers that a release wrote. Quote a kubeContext that YAML reads as other text, such as on: v1.5.0 refuses it unquoted.
v1.3.xv1.4.0no new field; the column Abandoned in kubectl get sfo, after Reason: Phase and Age move one place rightWith a CRD of v1.3.x, everything works except that column. tack prune needs delete on sitefailovers in the ledger namespace (row 16 of docs/operations.md). The chart grants it only with rbac.prune, so grant it only to the operators who prune. Once v1.4.0 writes a ledger, tack resume and tack abandon of v1.3.x refuse that run and name v1.4.0.
v1.3.0v1.3.1unchangedWhen a site gives no answer, a script that runs tack status -f gets exit code 1, along with the runs that were read. The list is then incomplete (docs/operations.md).
v1.2.xv1.3.0new optional fields: spec.takeOvers, status.abandoned, status.git.changedRevision, status.history, status.historyDropped, status.writtenBy; the column Reason in kubectl get sfoGrant create on events in the ledger namespace, for the ledger's Events. Without it, a run goes on without them (docs/operations.md). In each Workload file, quote every site name that YAML reads as a boolean or a number, such as n, on or 010. v1.3.0 refuses it unquoted (docs/workload.md). A run planned before the upgrade recorded the decoded name, such as false. Its flags, such as --fence, take that name.
v1.2.0v1.2.1unchangednothing
v1.0.x or v1.1.xv1.2.xnew optional fields: spec.takeOver, spec.fingerprints, status.fingerprints, status.supersededBy, status.git.releaseRevisionGrant get on the namespace kube-system at every site. The plan reads its UID to identify the site's cluster, and fails with forbidden without it. Tack refuses an http or https Git URL that holds a user, such as https://user@git.example.com/gitops.git. Move the user into git's configuration (credential.<url>.username). ssh://git@git.example.com/gitops.git and git@git.example.com:gitops.git still work.
A ledger written byUnder v1.6.0
v1.5.xAs for v1.3.x, below. Also, tack prune deletes a run with a follow-up Rebuild once that site's member was created again, 10 s or more after the run's ledger. v1.5.x kept such a run until someone deleted it by hand.
v1.4.xAs for v1.3.x, below. Such a run has no Lease, so tack abandon and tack run --take-over judge it by the ledger. They refuse a Running run whose ledger was written within its Running step's timeout plus 5 minutes, unless --force is passed.
v1.3.xtack status lists and shows it, with its history. tack resume continues it and keeps that history. tack abandon sees its mark. tack prune deletes it when its rules allow.
v1.2.xtack resume continues it.
v1.0.x or v1.1.xtack resume continues it. Each site's cluster is recorded in status.fingerprints on its first read, and checked after.

Upgrade from v0.1.x ​

  1. At every site, upgrade the CRD the way it was installed: the helm upgrade --install of Install, or kubectl apply --server-side of the new CRD. Do this before running tack v1.0.0 against the site. tack v1.0.0 reads ledgers as v1. Against a site still on the v0.1.x CRD, it stops and says to upgrade the CRD.
  2. Install tack v1.0.0 or later.
  3. In each Workload file, set apiVersion: failover.tako.id/v1. Nothing else changes. Tack still reads failover.tako.id/v1alpha1, with a warning.
After the upgradeBehaviour
Ledgers written by v0.1.xtack status reads them and tack resume continues them. The API server serves them as v1 and stores each as v1 the next time it is written.
tack v0.1.xStill reads and writes ledgers, through v1alpha1, until every operator has upgraded.
kubectlWarns on each v1alpha1 request. Until its discovery cache refreshes, kubectl get sfo may still ask for v1alpha1.
Downgrading the CRD to v0.1.xRefused by the API server: v1 is now in the CRD's status.storedVersions.

Released under the Apache License 2.0.