Skip to content

Limitations ​

What Tack does not do yet, and what would make it worth doing, by RFC. Each RFC keeps only the limitations an operator must know. This page is read from the RFCs when the site is built.

RFC#0001: Scope and Glossary ​

LimitationTrigger to revisit
There is no controller. A run lasts only as long as the tack process, and resumes from the ledger.A controller that runs the same ledger.

From RFC#0001.

RFC#0002: The Engine and the Driver Interface ​

None that an operator must know. Features not built yet are tracked as issues.

From RFC#0002.

RFC#0003: The Ledger: Schema, Phases and Resuming ​

LimitationTrigger to revisit
Tack does not check who else may write ledgers. To do so, it would need to read the permissions of other subjects, with create on subjectaccessreviews or a list of every RoleBinding and ClusterRoleBinding. No command needs that access otherwise. The operator runs the check of docs/operations.md at every site.A controller whose service account alone may write ledgers, or an admission policy that refuses every other writer.
Requirement 21 binds a run's steps to its Workload snapshot, but not the snapshot to the operator. A ledger planted with other contexts, Clusters or Git URL composes its own plan, and passes. The confirmation prints each site's context, API server and recorded cluster, and the Git URL (RFC#0009 Requirement 4). status is not checked: captured values, Git revisions, step states, holds and follow-ups are taken as written.A controller whose service account alone may write ledgers.
The ledger lives at one site. If to is lost during a run, its ledger is out of reach. A new run toward a reachable site takes over the holds with --take-over.A controller, or ledger copies at every reachable site.
A newer run whose ledger is at a site that does not answer is not found. tack resume then runs the older run, and prints a progress line that names the site (RFC#0009 Requirement 16). The check before each patch (RFC#0005 Requirement 11) still stops it at an object another run holds.A controller, or ledger copies at every reachable site.
Two sites' API servers whose clocks differ by more than 10 s can still order two runs wrong. A run of another site created less than 10 s before this one, such as by another operator, refuses the resume of this one. tack prune can then keep an older run that succeeded in place of the newest. The way on is a new run with --take-over. The install guide asks for clocks kept by NTP (docs/guide/install.md). Tack does not check them.A clock difference observed between sites, or a refusal of a run that should resume.
A site lost at plan time is recorded by the first process that reads it, which trusts that process's kubeconfig.Fingerprints given in the Workload file.
A value lost from status.captured after its Capture step succeeded is not captured again: the operator restores it (docs/runbooks/stop-reasons.md). Only an edit of the ledger is known to lose such a value. A token read again later may come from another demotion.A value lost without an edit of the ledger.
status.writes ends when the new primary takes writes at its site, not when clients reach it through Strait. In a site loss, its start is the run's start, not the time of the loss. docs/runbooks/measure-failover-time.md measures what clients see.A record of Strait's route change in the ledger.
tack resume does not refuse a run whose Lease another process holds. It takes the Lease over and names that process. Two processes on one run are stopped by the ledger's writes (Requirement 5), not before.Two processes observed to run one run.
The Lease's times come from the clock of the host that renews it. Hosts whose clocks differ by more than 10 s can see a live run's Lease as expired, which lets tack abandon and --take-over go on, as before the Lease. They can also see an ended run's Lease as fresh for longer.A clock difference observed between the operators' hosts.
The Lease is renewed by its own goroutine. So a process whose run is stuck, such as on a credential prompt, keeps the Lease fresh until the process ends.A process observed to hold a run's Lease while it ran no step.
Tack deletes a ledger only when the operator runs tack prune (Requirement 27). Until then, each search for runs reads every ledger of the workload at each site that answers, and tack status -f lists them all. Their cost grows with the number of runs. The measured sizes are in docs/operations.md.A search observed to take seconds while the operator prunes: a retention that runs by itself, in a controller.
kubectl get sfo reads status.supersededBy. A run whose take-over mark did not land shows an empty TakenOverBy. The mark does not land when the run's site gave no answer to the run that took it over (Requirement 17). tack status still lists that run as TakenOver, from the other run's spec.A take-over whose mark is observed missing.
tack prune keeps every abandoned and taken-over run. The operator deletes one by hand (docs/operations.md).A workload whose abandoned or taken-over runs slow its searches.
tack prune compares a member's creation time with the creation time of its run's ledger, each from its own site's API server. Sites whose clocks are more than 10 s apart can make a Rebuild look done before it is.A clock difference of more than 10 s observed between two sites' API servers.

From RFC#0003.

RFC#0004: CloudNativePG Support ​

LimitationTrigger to revisit
One read-write Pooler per site. Read-only Poolers are not moved.A workload with several Poolers that follow the primary.
With three or more sites, the driver does not check that each replica's source chain reaches the new primary.The first three-site deployment.
If the old primary's fast shutdown outlasts switchoverDelay, CloudNativePG shuts down immediately and never publishes a token (pkg/management/postgres/instance.go:695-720). The demotion wait then times out and the run stops. A shutdown that outlasts timeouts.demotion stops the run while CloudNativePG keeps waiting. A resume waits again (docs/operations.md).An observed timeout.
The rebuild of an old primary is manual, and so is the hand-over of its writes (docs/runbooks/recover-old-primary-writes.md).A safe, declarative rebuild in CloudNativePG.
The fork position is read right before the promotion: a replica that replays more WAL before it promotes forks just after it. The runbook reads from that position, so it lists a little more, never less.A record of the exact switch point that the instance status gives.
A replica of a third site that was ahead of to gets a Rebuild, but no snapshot: only from does.A run observed with a third site ahead.
Without the opt-in replay-time query (docs/workload.md), the plan cannot say when a replica's last replayed commit was: the replay lag is 0 once a replica has replayed all it received, which is the usual state after a site loss. A replica whose source stopped long ago then reads as most caught up when no other site is ahead. With the query, the time is no exact RPO line: under concurrent writes, a few commits stamped just before it may be lost. The lag and the replay time can be up to metricsQueriesTTL old. The replay time is the old primary's clock.The time of the last replayed transaction in the instance status, or in CloudNativePG's default queries; a WAL-ordered commit position that PostgreSQL exposes.
Positions are read once in the plan and once after the confirmation. A replica that still restores WAL from an archive can move past another between the read and the promotion.A replica observed to move after the read.
In a SiteLoss run, an old primary that is alive but cut off from the operator keeps taking writes until it is fenced (RFC#0007 Limitations). Those writes are lost to the new primary. The rebuild drops them.A fence that does not need the site's API.

From RFC#0004.

RFC#0005: Flux Hold and Release ​

LimitationTrigger to revisit
A held object encrypted with SOPS cannot be rendered. Tack does not decrypt.A held kind that is encrypted.
Kustomizations that need remote bases cannot be rendered.A user with remote bases.
Tack does not read controller flags, such as --override-manager, or feature gates from the cluster. It assumes the defaults, including strict substitution.A cluster with other flags or gates.
Objects applied by a HelmRelease cannot be held.helm-controller gaining a per-object hold.
An object that carries kustomize.toolkit.fluxcd.io/ssa: IfNotPresent in Git is diffed as if Flux applied it. Flux leaves it alone once it exists, so the check may stop a run that Flux would not disturb.A held object with that annotation.
The check after release compares generation, which covers spec but not labels or annotations.A workload whose behaviour depends on metadata.
The direct field comparison is on raw JSON. A value that the API server rewrites into a canonical form, such as a quantity, reads as a difference.A run that changes such a field.
Even with remote bases disallowed, kustomize may still try to fetch a remote resource from the operator's machine before it fails.A user with remote resources.

From RFC#0005.

RFC#0006: The Git Change ​

LimitationTrigger to revisit
Keys that contain dots cannot be addressed.A repository that needs them.
Flow-style YAML on a path cannot be edited.A repository that uses it.
A filter driver, such as Git LFS, still runs on checkout and add when the repository's .gitattributes names it and the operator's configuration defines it. Git has no single setting that turns filters off, and an attribute source that ignores them would change how files are checked out.A repository with a filter attribute on a file Tack reads or edits.
Tack does not sign commits. Git's configuration decides.A repository that requires signed commits.
A fetch that takes more than 2 minutes counts as unreachable, even while it makes progress. So does a fetch whose server sends nothing for 30 s, because Tack fetches with --quiet (git's uploadpack.keepAlive keeps a fetch alive). Over HTTP or HTTPS, and with the operator's own ssh command, only that 2-minute deadline ends a connection that never completes. So a plan can wait those 2 minutes during a site loss that also takes down Git's host.A fetch observed to end, or to wait, this way.

From RFC#0006.

RFC#0007: Break-Glass ​

LimitationTrigger to revisit
While the run waits for Git, held objects stay held: Flux applies nothing to them, including unrelated Git changes.A long Git outage.
Tack cannot tell a lost site from one that runs but is cut off from the operator's host. Such a site stays primary until Flux there applies the run's commit, which needs Git, or until Tack fences it. For that time there are two primaries. A Routing step at it is deferrable, so Succeeded says nothing about routing there. The runbook docs/runbooks/site-loss.md confirms the loss from a second vantage point, and fences the site out of band before the run.A fence that does not need the site's API, or a second vantage point that Tack reads.
A lost site whose API still answers, such as a load balancer that answers 503 in front of a lost control plane, does not count as lost. Nor does a site whose context needs a credential plugin that fails during the loss. In both cases the plan refuses.A site loss that Tack refused this way. The runbook's way on: docs/runbooks/site-loss.md.
A server name that does not resolve counts as no answer, also when the kubeconfig names the wrong host.A live site read as lost because of a wrong name.
A site that comes back after the run succeeded is fenced by Flux from Git, not by Tack. Tack does not watch for it.A controller.
The read of a site before its steps runs once per site and process, and the read of the other sites runs before each Failover step, not during one. A site that turns primary later, such as during the promotion's wait, is read only before its next step.A controller, or a site observed to turn primary during a run.
A site that gives no answer to the read of the other sites does not stop the run. In a Planned run, a from that is cut off from the operator's host but was promoted again by hand then gives two primaries. A resume with from down promotes to on purpose (docs/runbooks/no-primary-after-demotion.md, option 1).An old primary observed to run as primary again after its demotion, while cut off.
With --fence <site> for a site that the run fences after the promotion, such as a third site that is primary, the site and to are both primary from the promotion until the run declares the new primary there. The site may hold writes that to lacks: the run records the follow-up Rebuild for it, and those writes are lost to to.The first three-site deployment.
Each site that gives no answer costs up to about 30 s of checks, 3 checks, and all such sites are checked at once. This cost comes in each read of a plan, in the search before a run, at a run's start, in tack status without a run name, and, in a tack resume of a Planned run, in the first read of the other sites, while no site may be primary. A read reads a site's objects only once the checks of every site before it ended, up to about 30 s after its own. A site lost in between fails the read, with an error that does not count as a lost site.An outage observed to grow this way, or a read observed to fail this way.
A read outside Requirement 11 is not repeated: Flux's reads in a Hold, Reconcile, VerifyGit, Release or VerifyNoop step, Strait's in a Routing step, which waits until its timeout, the ledger's own reads, and the reads of tack status --context and tack abandon before it asks. Each runs before the switch or after the promotion, or changes nothing. A blip there stops the run with StepFailed, or fails the command. A write that gets no answer is not repeated either, except the ledger's (RFC#0003 Requirement 5). Such a write's step fails, and tack resume runs it again (RFC#0002).A failure from a blip after its site's check, in one of these reads or in a write.
A credential plugin may never end, such as one that waits for an identity provider that went down with a lost site. It then holds the checks and steps of every site whose context has a plugin, until it ends or the operator interrupts tack. Tack waits with no time limit, and prints a line after 2 s. The interrupt ends Tack's wait, not the plugin, which client-go runs with no context. The plugin runs on, also after tack exits, until it ends, and no other plugin starts in that process meanwhile. Ctrl-C reaches the plugin too, as it runs in the terminal's foreground process group. A signal to tack's pid alone does not. The way on is the site-loss runbook's: a context whose credentials need nothing of the lost site (docs/runbooks/site-loss.md).A plugin observed to hang during a site loss.
Credentials that expire inside a step, such as between the polls of a Wait step, or between the fetch before a step and its request, are obtained by client-go inside that request. No line is printed, and an interrupt does not end that wait. A request to a site whose plugin still runs, such as the ledger write that records why an interrupted run stopped, waits for that plugin, whatever its own time limit (observed in a unit test: a request with 500 ms waited 1.9 s). A second signal ends tack (RFC#0009).A run observed to wait inside a request for a credential plugin.
In a SiteLoss run, the base is the revision Flux last applied, even when Git answers. If the branch moved after Flux last fetched it, the run stops with GitChanged at the Git step, after the switch.An observed GitChanged after a site loss.
With three or more sites, sites where Flux last applied different revisions stop a SiteLoss run with GitChanged in Hold, before the switch.The first three-site deployment.

From RFC#0007.

RFC#0008: Confirming Routing from Strait's Status ​

None that an operator must know.

From RFC#0008.

RFC#0009: The CLI and its Dry Run ​

None that an operator must know. What the release workflow does not check is in docs/runbooks/release.md.

From RFC#0009.

Released under the Apache License 2.0.