Self-hosted GitHub Actions runners for five orgs with ARC

All the CI for my GitHub organizations runs on my own Kubernetes cluster: Ollebo, Mantiser, Samma, Hacking Robots & Beer and Viodlar. Runners start when a job is queued and disappear when it's done. Some run on amd64 nodes, some on arm64.

The tool is GitHub's own Actions Runner Controller (ARC), in its newer "runner scale set" mode. Here's the setup and the errors I've hit.

Why self-hosted?

• Minutes. Several orgs building container images on every push adds up quickly on hosted runners.
• Native arm64. Part of my cluster is ARM boards. Building arm64 images natively is much faster than emulating with QEMU.
• Close to the cluster. Runners can reach internal services and registries without opening anything to the internet.

How ARC v2 works

GitHub ◄──long poll── listener pod (one per scale set, in arc-systems)
                           │ "3 jobs queued"
                           ▼
                    controller ──► creates ephemeral runner pods (arc-runners)
                                        │ run one job, then deleted

• The controller is installed once.
• Each runner scale set connects to one GitHub org (or repo) and gets a listener pod that asks GitHub for queued jobs.
• When jobs are queued, the controller creates ephemeral runner pods. Each runs one job and is deleted. No state leaks between jobs, and with no jobs, nothing runs.

Install

Two Helm charts from GitHub's OCI registry. First the controller:

helm upgrade --install arc \
  --namespace arc-systems --create-namespace \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller

Then one scale set per org, each with its own values file:

helm upgrade --install arc-runner-set-samma \
  --namespace arc-runners --create-namespace \
  -f github-runner-samma.yaml \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

A small script runs the controller install and then every scale set, so rebuilding everything is one command.

A values file

githubConfigUrl: https://github.com/<your-org>/
githubConfigSecret: github-runner-<your-org>   # pre-created Secret, see below

containerMode:
  type: "dind"            # Docker-in-Docker sidecar for image builds

controllerServiceAccount:  # required, see the gotchas
  namespace: arc-systems
  name: arc-gha-rs-controller

template:
  spec:
    nodeSelector:
      kubernetes.io/arch: amd64

For arm64, copy the file and change the architecture:

template:
  spec:
    nodeSelector:
      kubernetes.io/arch: arm64

Each scale set is named after its Helm release, and that's the label you use in the workflow:

jobs:
  build:
    runs-on: arc-runner-set-samma      # or arc-runner-set-arm for arm64

I have seven scale sets: one per org, plus arm64 sets for the two orgs that ship ARM images.

Credentials

Each scale set needs a token that can register runners for the org. I use a personal access token with the manage_runners:org scope, one per org, stored in a Secret that I create myself:

kubectl create secret generic github-runner-<your-org> -n arc-runners \
  --from-literal=github_token='<token>'

Don't put the token in the values file. If you do, the chart creates the Secret for you, and the token ends up in your values file, your shell history and maybe git. A GitHub App is the more robust choice for organizations: no expiry, and finer permissions.

Docker-in-Docker

containerMode: dind adds a docker:dind sidecar to every runner pod, so workflows can docker build and docker push as on a hosted runner. It needs a privileged sidecar. That's fine for runners that only build my own code, but don't do it for public repos where anyone can open a PR that runs on your cluster.

The errors you will hit

Almost all problems show up as listener pods crashing in arc-systems. Check the log first:

kubectl logs -n arc-systems <listener-pod>

Error	Cause	Fix
failed to get kubernetes secret	controllerServiceAccount.name is wrong (arc instead of arc-gha-rs-controller)	Fix it in every values file and helm upgrade
409 Conflict: already has an active session	A stale session from a previous install	Uninstall that scale set and install it again
404 Not Found: No runner scale set found	An old scale-set-id annotation points to a registration that GitHub deleted	kubectl annotate autoscalingrunnerset <name> -n arc-runners actions.github.com/scale-set-id-, then reinstall
401 Unauthorized: Bad credentials	The token expired	New token, update the Secret, restart the listener

The 401 is the sneaky one. One token per org means every scale set for that org dies on the same day. Put token expiry in your calendar, or use a GitHub App.

Uninstalling is not always clean

ARC puts finalizers on its resources. If you uninstall in the wrong order, or the controller goes first, namespaces get stuck in Terminating. The fix is to remove the finalizers by hand:

kubectl patch autoscalingrunnerset <name> -n arc-runners \
  --type=merge -p '{"metadata":{"finalizers":[]}}'

Do the same for the listeners in arc-systems and for the gha-rs roles, role bindings, service accounts and secrets in arc-runners. Uninstall the scale sets first, then the controller.

Leftover CRDs from the old ARC

If you used the older "summerwind" ARC, its CRDs conflict with the new controller. Delete the actions.github.com CRDs left from earlier installs, then install the controller again.

What I'd do differently

• Use GitHub Apps instead of tokens. Expiring tokens are the most common cause of "CI is down".
• Put it in GitOps. My runners are installed with a script rather than ArgoCD because the charts are OCI-hosted and I set it up before I had a pattern for that. ArgoCD handles OCI Helm charts fine, and it would show drift.
• Spread the listeners. All seven listener pods ended up on one node. When that node reboots, all CI pauses. A topology spread constraint fixes that.
• Set maxRunners. Without it, a burst of jobs can start more Docker-in-Docker pods than your nodes can handle.

Self-hosted runners turned CI from a monthly bill into something that just runs on hardware I already have. Most of the work is in the first week; after that it's mostly renewing tokens.