Running a Job¶
A ready job runs once its preparer or approver starts it. From then on the Hub moves each site through the steps on its own, a few sites at a time, until every site has ended. People control the run from the job's details: they can pause and resume it, continue it past a checkpoint, and stop it.
Starting a Job¶
Open the job from Sites -> Jobs and click Start. Start appears only for a job that is ready to start, and only to its preparer and its approver.
The Hub refuses to start a job during a CA rotation or a Hub lockdown; see Automatic Pauses. A job does not need its preparer or approver to stay signed in once it has started.
How the Hub Moves Sites¶
The Hub's job coordinator makes a pass over every running and paused job every two seconds. An action taken in the console changes the job's record, and the coordinator acts on it on its next pass.
On each pass, for each site, the coordinator does whatever comes next: it starts a site, sends the site its next step, asks the site about a step that has not answered, or keeps a wait or checkpoint. Each step sent to a site goes with its own newly signed statement. The site answers with the step's outcome when the step ends within 10 seconds, and otherwise answers that it is still running. The Hub then asks about it on later passes, each question with its own statement. See Job Site Outcomes for what happens when no answer comes.
Before a site's first step, the Hub checks that the steps can run there. The site must be enabled and connected, must have reported its Hub settings, and must be expected to accept the preparer's statements. The preparer, and the approver when there is one, must hold the Jobs grant for it. When a step starts channels, the site must not have a developer license. A site that fails a check is skipped as not run. See Job Site Outcomes.
Pacing¶
In Progress (sites) sets how many places the job has. A site takes a place when it begins its steps and keeps it while it works through them, including during a Wait. It gives the place up at a Checkpoint or a Wait until a time, and when its part of the job ends.
When a place comes free, it goes first to a site carrying on after a checkpoint or a clock-time wait, and only then to a site that has not begun. New sites begin in the order of the job's site list. The number of places also bounds how far a bad change spreads: once the failure limit is reached, only the sites already working through their steps go on.
A Hub node sends at most two Import the file steps at once, across all its jobs, since each holds the whole configuration file. A site whose import step waits for one of those keeps its place and counts as waiting for a free place.
When a Site Fails¶
A site fails when one of its steps fails, or when the site refuses a step. The site's part of the job ends at that step, and the Hub records the reason, for example "step 2 (Restart channels) failed: ..." The site keeps whatever its earlier steps changed until someone acts; see Rolling Back a Job Site. Other sites carry on.
A step is never retried after the site has answered it. A step the site refuses, including one refused because the person has made more requests than the site's limits allow, fails the site.
The Failure Limit¶
A job halts when the number of failed sites reaches its Failure Limit (sites). A site counts as failed when one of its steps failed or was refused, and when it never answered about a step (shown as unknown). A site that was not run, or was stopped because the preparer or approver lost the Jobs grant there, does not count.
Once the limit is reached, the job's state shows that it is halting, and:
- No new site begins. A site that has not begun ends as "halted, not run", and so does a site that has so far only waited, at a wait, a clock-time wait or a checkpoint at the start of the job, since nothing has been sent to it.
- A site at a wait or a checkpoint, or waiting for a free place, ends as "halted after step N".
- A site working through its steps carries on until it reaches its next wait or checkpoint, or the end of its list, and ends there. While the job is paused, such a site ends where it is.
When every site has ended, the job is halted. A halted job never runs again. To carry the change to the sites it did not run, rerun it; see Copying and Rerunning a Job.
Pausing and Resuming¶
Pause stops the job from starting anything new: no site begins and no site starts its next step. A step already sent to a site finishes, and its result is recorded. A Wait keeps counting during a pause, and a clock-time wait that comes due waits for the resume.
Resume lets the job carry on. Both buttons are for the preparer and the approver. A job that is also paused by the system stays paused until that pause ends.
Automatic Pauses¶
The Hub pauses every running job on its own while:
- A CA rotation is in progress. See Rotating the Internal CA.
- The Hub is locked down: it is over its licensed site count (see Licensed Site Count), or it has gone too many consecutive days without a successful check-in with Qvera.
The job's state then reads "Paused during a CA rotation" or "Paused during a Hub lockdown" (see Hub Lockdowns). When the condition ends, the job resumes on its own, unless a person also paused it, in which case it waits for Resume. A person can pause a job the system has paused. While the condition lasts, no job can be started.
Continuing Past a Checkpoint¶
Continue releases every site waiting at the job's next checkpoint. It appears to the preparer and the approver once every site has reached the checkpoint or ended, and only while the job is running. See Job Waits and Checkpoints.
Stopping a Job¶
Stop is for the preparer, the approver and Hub administrators, and asks for confirmation. It is not offered once a job is stopping or halting, so a halted job keeps the failure limit as its reason.
A job that is ready to start but has not started ends at once, with every site "stopped, not run".
A running or paused job stops the same way the failure limit halts one. No new site begins, sites waiting to begin, and sites that have so far only waited with nothing sent to them, end as "stopped, not run", and other sites at a wait or a checkpoint, or waiting for a free place, end as "stopped after step N". A site working through its steps carries on until its next wait or checkpoint, or the end of its list; while the job is paused, such a site ends where it is. A site with a step in flight still waits for that step's answer. The state reads "Stopping" until every site has ended, and then "Stopped". A stopped job never runs again.
When the Preparer or Approver Changes¶
The Hub reads the preparer's and the approver's current Hub role before every step, so a step always carries their role as it is now.
When either person is disabled or deleted at the Hub while the job is running or paused, the job stops and its state names the person. Unlike Stop, no site gets another step: a step already sent finishes, and every site ends where it is, as "stopped after step N", or "stopped, not run" when nothing has been sent to it. A job already stopping or halting keeps its reason and ends as it was going to, "Stopped" or "Halted", but its sites still get no further step. The Hub audit log still names the person, in one JOB_STOPPED entry.
When either person no longer holds the Jobs grant at a site, that site goes no further. While nothing has been sent to it, even after a wait or checkpoint at the start of the job, it is "not permitted, not run". After that, it ends as "stopped after step N" with the reason. Neither counts as a failure.
On an HA Hub¶
On a Hub cluster, one node at a time runs every job: the node that holds the job coordinator's lock in the database. It reaches sites connected to other nodes through the relay between nodes. When that node shuts down, it releases the lock and another node takes over on its next pass. When a node dies, the node that recovers its work also takes over its lock. See HA Hub Deployment.
Sites in a Running Job¶
A site in an unfinished job cannot be deleted at the Hub. Stop the job, or let it end, first.