How to operate and troubleshoot incus-gh-runner¶

Read logs, collect VM diagnostics, change capacity and configuration safely, upgrade the binary and runner image, and diagnose common failures on a running deployment.

Prerequisites¶

incus-gh-runner deployed as the incus-gh-runner.service systemd unit (see Deploy incus-gh-runner).
journalctl access to the unit on the controller machine.
Shell access to edit /etc/incus-gh-runner/config.yaml, replace its credential sources, and restart the unit on the controller machine.

Read the logs¶

The process writes structured JSON logs (Go slog) to stdout. Under systemd this lands in the journal for the unit:

journalctl -u incus-gh-runner -f

journald stores each JSON line as its MESSAGE field, so -o json-pretty shows journal metadata with the process's line embedded as one escaped string. To inspect the process's own fields, print the raw line with -o cat and pipe it to a JSON tool:

journalctl -u incus-gh-runner -o cat -n 200 | jq .

There is no log-level or verbosity flag. The process always logs at its built-in level; you cannot quiet or increase it via configuration.

The following events carry the fields you need for day-2 monitoring. All other fields on a line (timestamp, logger group, message) are structural slog output.

Event	Fields	Meaning
`runner demand updated`	`assigned_jobs`, `target`, `generation`	Controller recomputed capacity target and current GitHub message-session authority.
`runner operation scheduled`	`operation` (`create`/`fence`/`delete`/`list`), `operation_id`, `runner_id`	An external lifecycle operation was handed to a worker. `fence` removes GitHub registration before idle scale-down.
`runner operation completed`	`operation`, `operation_id`, `runner_id`	The operation succeeded.
`runner operation failed`	`operation`, `operation_id`, `runner_id`, `retry_after`, `error`	The operation failed and entered cooldown; see Troubleshooting below.
`GitHub Actions job started`	`runner_name`, `job_id`	A queued job was assigned to a runner.
`GitHub Actions job completed`	`runner_name`, `job_id`, `result`	The job finished on that runner.
`job proof delivery failed`	`job_id`, `runner_id`, `runner_name`, `error`	Signing or delivering the job's machine proof failed; that job's guest proof helper times out. Emitted only when `job_proof` is enabled.
`GitHub Actions job proof event dropped`	`job_id`, `runner_id`, `runner_name`, `error`	A malformed `JobStarted` event or a full proof queue produced no proof for that job. Emitted only when `job_proof` is enabled.
`GitHub message session disconnected; reconnecting`	`error`, `retry_after`	The GitHub Actions long-poll session dropped; see Troubleshooting below.
`owned Incus runner started`	`runner_id`, `correlation_id`	A VM the controller owns was created, started, and handed its job payload; it is provisioning until the guest reports in.
`owned Incus runner deleted`	`runner_id`	A VM the controller owns was deleted.

Credential contents, including the Incus client private key, are never logged. Review logs for workload and environment identifiers before attaching them to a ticket.

For what these events mean in terms of runner lifecycle and capacity, see How incus-gh-runner works.

Collect VM diagnostics¶

When a runner VM is deleted, the controller captures its serial console log before tearing it down. To keep these captures, point incus.diagnostics_dir at a directory under the unit's LogsDirectory:

incus:
  diagnostics_dir: /var/log/incus-gh-runner/diagnostics

This fragment adds one setting to the existing incus mapping. Retain its complete socket mode or url plus certificate-file mode; do not replace the connection fields with this fragment.

Create the directory with mode 0700, or let the unit's LogsDirectory machinery own it (it produces 0700). The controller refuses any other mode: every capture then fails with a failed to store runner diagnostics warning in the journal while runner deletion proceeds, and no file appears.
Restart the unit for the new value to take effect:
```
systemctl restart incus-gh-runner
```
After a runner VM is deleted, find its capture at <diagnostics_dir>/<runnerID>.console.log. See the guest contract reference for the diagnostics file naming and permissions.

If incus.diagnostics_dir is unset, console captures are discarded and nothing is written to disk.

The controller caps each capture at 1 MiB, marks truncated output, and retains at most 256 capture files by deleting the oldest before a new file is created. The packaged incus-gh-runner.tmpfiles.conf additionally removes files older than 30 days from the recommended directory. Install it as /usr/lib/tmpfiles.d/incus-gh-runner.conf; if you choose a different diagnostics directory, copy the policy and change its path to match. Ensure systemd-tmpfiles-clean.timer is enabled so expiration also runs while the controller is idle.

Console output may contain sensitive workload content

The serial console log is the guest's raw console output, including anything a running Actions job printed to it before exit. Treat diagnostics files as sensitive and restrict directory access to operators who are cleared to see job output. Shorten the shipped 30-day expiration when your workload sensitivity requires it.

For the full cleanup and metadata model behind runner VMs, see How incus-gh-runner works. For every configuration key, see Configuration reference.

Change capacity or configuration safely¶

To change capacity limits, timeouts, or any other setting:

Edit /etc/incus-gh-runner/config.yaml (or the relevant environment override).
Restart the unit:
```
systemctl restart incus-gh-runner
```

systemctl restart sends SIGTERM. The controller does not delete busy VMs on shutdown — jobs already running on a runner continue to completion and are reconciled against the new configuration once the process comes back up. Idle and provisioning runners are unaffected by the restart itself; they are re-evaluated against the new target on the next reconcile.

Rotate Incus HTTPS credentials¶

The controller reads its Incus client certificate, client key, and pinned server certificate only during startup. It does not watch those files.

For a client-certificate rotation:

Generate the replacement key and certificate on the controller machine.
From the existing trusted administration workstation, enroll the replacement certificate as restricted to the runner project before using it:
```
incus config trust add-certificate <remote>: client.crt --restricted --projects github-runners
```
Replace /etc/incus-gh-runner/client.key as root:root mode 0600 and /etc/incus-gh-runner/client.crt as a readable public certificate.
Restart incus-gh-runner.service and confirm it connects through the named project.
Remove the old trust entry only after the replacement is active.

For a server-certificate rotation, obtain the replacement certificate through an authenticated out-of-band channel and verify its SHA-256 fingerprint against a trusted operator record. Coordinate the Incus server change with an atomic replacement of /etc/incus-gh-runner/server.crt, then restart the controller. Do not retrieve the new pin insecurely, accept it on first use without independent verification, or disable TLS checks during the transition.

The HTTPS systemd drop-in continues to expose the root-owned client-key source through LoadCredential= after restart. No daemon-reload is needed when only credential file contents change.

Shutdown budget¶

Each shutdown phase (graceful, then forced) waits up to timeouts.shutdown; the total wait budget across both phases is 2 * timeouts.shutdown. The unit's TimeoutStopSec must exceed this budget, or systemd kills the process mid-shutdown before it finishes waiting out active operations. The packaged unit ships TimeoutStopSec=70s, which covers the default timeouts.shutdown: 30s (budget 60s) with headroom.

If you raise timeouts.shutdown above 35s, raise TimeoutStopSec in the unit to stay above 2 * timeouts.shutdown, run systemctl daemon-reload, then restart the unit.

For the full set of timeout and capacity keys, see Configuration reference.

Upgrade¶

Upgrade the controller¶

Install the new DEB or RPM with the controller machine's package manager. For a raw-binary deployment, replace /usr/bin/incus-gh-runner manually.
Restart the unit:
```
systemctl restart incus-gh-runner
```

The same busy-VM survival and shutdown-budget behavior described above applies during a binary upgrade.

Upgrade the runner image¶

From a trusted Incus administration context, import the new image under a new alias or fingerprint.
Update incus.image in config.yaml to point at it.
Restart the unit.

Existing runner VMs built from the old image are left running until they finish their job and are recycled; new VMs are created from the image now referenced by incus.image. See Build a hardened runner image for building and boot-testing an image before importing it.

Troubleshooting¶

Symptom	Likely cause	Action
Unit exits immediately at start	Invalid config, an unreadable or malformed credential, HTTPS certificate verification or pinning failure, inaccessible Incus project, or image/profile preflight failure	Read the error on stderr / `journalctl -u incus-gh-runner`; the process fails fast and reports the failing setting or startup stage without printing key or certificate contents.
Repeated `GitHub message session disconnected; reconnecting`	A GitHub-side outage, or the App/PAT credential was revoked mid-run	Backoff is capped and automatic; no restart is needed for a transient outage. If it persists, verify the credential is still valid.
Runner stays `provisioning`, then goes `terminal` after a while	Image doesn't implement the guest contract, the wrong image is configured, or the VM has no network reachability to GitHub	Check `incus.image` and `incus.bootstrap_timeout`; boot-test the image against the guest contract; confirm the VM's network path.
`runner operation failed` repeats with growing `retry_after`	An Incus-side failure (API, storage, hypervisor) put that operation into cooldown	Creates share one cooldown; each runner's delete has its own. A fresh successful inventory list (`operation: list`) must land before further mutation is attempted — check Incus itself for the underlying error reported in the `error` field.
A runner VM refuses deletion	The controller's cleanup selector on the instance doesn't match `incus.owner`	Expected behavior — the controller refuses deletion outside its exact selector. Confirm the VM's `user.incus-gh-runner.owner` metadata and your configured `incus.owner`, and delete manually via Incus if it is genuinely orphaned. The selector prevents accidental cleanup; another project writer can forge it.
`job proof delivery failed` or `GitHub Actions job proof event dropped` errors while jobs run	The proof could not be signed, delivered to the runner VM, or enqueued before the job's helper timeout	Read the `error` field for the failing stage. The affected job fails closed through the guest helper timeout; no other jobs are affected. See the `job_proof` keys in Configuration reference.