Skip to content

Rolling Back a Job Site

Rolling back a site undoes the job's import at that one site. Each object the import changed goes back to how it was before the import, and what the import created is deleted, turned off or, for a zone, kept. A rollback only restores: it never restarts a channel. The Hub never rolls a site back on its own; a person asks for it, one site at a time, and it needs no approval.

When a Site Can Be Rolled Back

A site can be rolled back when all of these are true:

  • The site's part of the job has ended, whatever its outcome.
  • The Hub has the site's answer to the job's import step, or the site ended unknown while the import was in flight. The import may have run there, and a site that never received it has nothing to undo, so the rollback ends with no objects. A job without an import step has nothing to roll back.
  • The person asking holds the Jobs grant at that site, through their role. Hub administrators need it too (see Site Access Grants). The Hub checks again as it signs the rollback, which may be later when the rollback waited for a CA rotation or a lockdown to end: when the person has lost the grant or been disabled at the Hub by then, the rollback fails, naming which.
  • No rollback of that site is already under way, from this job or another. Roll Back Site names the other job, for example "'lab' is being rolled back from 'Lab update'. Roll this site back once that rollback ends." When two rollbacks of one site are asked for at the same moment, the Hub tries only one, and the other fails with "the site is being rolled back from '...'; ask again once that rollback ends". The one it tries can still fail, for example when the site is not connected, so look at both before asking again.
  • The site is in no other unfinished job, one that is awaiting approval, ready to start, running or paused, so a rollback and another job never change the site's configuration at the same time. Roll Back Site names the job, for example "'lab' is in 'Lab update', which has not finished. Roll this site back once that job ends." The Hub checks again as it sends the rollback, and the rollback fails, naming the job, when a job submitted in the meantime holds the site.
  • The job's configuration file holds no packages. A rollback restores objects one by one and cannot undo a package import, so for such a job Roll Back Site says "This job's configuration file holds packages, and a rollback cannot undo a package import. Restore this site's configuration by hand."

A rollback that reaches the site while the import it undoes is still running there, or so soon after the site accepted the import that its file may still be arriving, fails with "the import is still running here; ask again once it has finished". Nothing is changed; ask again later.

A rollback that reaches the site while another configuration import runs there, from the console, the Management API or an AI client, fails with "an import is running at this site; ask again once it has finished". Nothing is changed. While a rollback runs, the site turns away other configuration imports the same way it does while any import runs.

A site can be rolled back again once an earlier rollback of it has finished. Rollback stays available as long as the Hub keeps the job's record; see Job Retention.

Rolling Back a Site

Open the job from Sites -> Jobs and select the site on the Sites tab. Click Roll Back Site and confirm.

The Hub sends the rollback to the site on its next pass, with a statement that names the person who asked. While it runs, the site's results show who asked for it and that it is under way; when it ends, they show what happened to each object.

When the Hub cannot reach the site as it sends the rollback, the rollback fails at once with "the site is not connected". When the site turns the rollback away at its request limit, it fails with "the site turned the request away at its request limit; ask again". When a Hub node relaying to the site answers in its place, the Hub asks the site what happened to the rollback, and sends it again only when the site has no record of it or never started it. When the site gives no answer about a sent rollback for 30 minutes, the rollback fails with "the site gave no answer about the rollback for 30 minutes; ask again once it is back", and the Hub stops asking. Ask again once the site is back.

During a CA rotation or a Hub lockdown, a rollback not yet sent waits, and the site's results say it is waiting for the rotation or the lockdown to end. The Hub sends it once the condition ends. A rollback already sent carries on. See Hub Lockdowns.

Who the Rollback Runs As

The rollback runs at the site as the Hub user of the person who asked, not the preparer, with the role the site's Role Map gives that person's Hub role. The site's permissions apply to every object: restoring or deleting one needs edit rights in its zone. The site's revision history shows the changes with Via set to Hub Job, under that person's Hub user.

What a Rollback Restores

Each object the import changed goes back to the revision it was at before the import. The site finds these objects from the revisions it stamped with the import's statement, so an object that the file matched but the import did not write, such as a zone, or anything at all when the import never started, is left as it is. The revision each one goes back to comes from:

  • The site node's own record of the revisions the objects in the file were at just before the import, when the node that ran the import still has it. A node keeps this record in memory for its most recent 200 job imports, so a restart of the node clears it.
  • Otherwise, the revision just before the first one the import wrote.

An object that someone changed after the import is put back as well. An object already at that revision is reported as unchanged. So is a published function, published mapping, certificate or private key that the import saved without changing anything a person can see in it: the import saves every object in its file that matches one at the site, whether or not the file changes it.

A certificate or private key the import did change is not put back, because restoring one would change only its description and keep the certificate or key the import put in. The rollback reports it as failed, and the earlier certificate or key has to be imported again.

Some things stay as they are now, as with the MCP's restore_revision (see MCP Revision History Tools): passwords and other secrets, and a channel's sample messages, cache values and script variable values.

A restored channel is never restarted. A running channel keeps the scripts it has already compiled, together with the published functions they call. A script that no message has reached since the channel started is compiled when a message first needs it, from the channel's running version and the published functions at the site at that time: after a rollback, the restored ones. Such a script can fail when it calls a function the rollback deleted. Restart the channel to run the restored version throughout, for example from the site's console or with a new job.

Objects the Import Created

  • Channels: stopped unless they are already stopped, then deleted. A channel that has received a message is disabled by its enabled flag instead, with its messages kept for someone to delete later, whatever its persistence level and even when its messages have been purged since. So is a channel that something at the site still uses, and one that has not finished stopping after 30 seconds. A channel is deleted before the other objects the import created, so the connections and other objects only it used can be deleted too.
  • Zones: kept.
  • Everything else: deleted, unless something at the site still uses it. An object still in use is kept, and the result says where it is used. Connections are deleted before the certificates and keys they use, so those can be deleted too.

An object the import both created and deleted is left alone.

Objects the Import Deleted

A rollback cannot bring back an object the import deleted. It reports each one as failed, and the object can be restored from its revision history at the site.

The Outcome

A rollback goes on past a failed object, and reports every object by its type and id. Each outcome is one of: restored to a revision, unchanged, deleted, stopped and deleted, already deleted (by an earlier rollback, for example), kept with the reason, stopped and disabled with the reason, disabled with the reason (for example "disabled. This channel has received messages."), or failed with the reason. When any object failed, the rollback as a whole failed.

The Hub writes a JOB_ROLLBACK_REQUESTED entry to its audit log for each request. See What a Site Records for a Job for the site's side.