Skip to content
gocdnext

Runner profiles

A runner profile is a named bundle of execution policy a job references at apply time. Profiles centralise the knobs that would otherwise repeat across every YAML in the project — fallback image, default + maximum CPU/memory, agent tags the job needs, environment variables (plaintext and secret), and Kubernetes scheduling hints.

Pipelines reference a profile by name:

jobs:
  build:
    agent:
      profile: gradle-heavy
    script:
      - ./gradlew build

The scheduler resolves the profile at dispatch time, so admins can update the profile in place (raise the memory cap, add a toleration, rotate a credential) without re-applying every pipeline that uses it.

Engine

Today profiles are scoped to the Kubernetes engine — Shell and Docker engines accept the YAML field but ignore the profile’s scheduling and resource fields. The engine: kubernetes field on the profile is enforced at apply time.

Default and maximum resources

default_* fills any matching resources: slot the job left empty. max_* caps any value the job tries to set higher.

# admin profile
name: gradle-heavy
engine: kubernetes
default_cpu_request: 500m
default_cpu_limit: "2"
default_mem_request: 2Gi
default_mem_limit: 4Gi
max_cpu: "4"
max_mem: 8Gi

A job referencing this profile without an explicit resources: lands with the defaults; a job that asks for memory: 16Gi fails apply with a clear “exceeds profile max_mem” error.

Fallback to default profile

Since v0.13.1, when a job declares NO profile: AND a profile named default exists in the DB, the scheduler auto-applies the default profile’s resource bounds (and only the bounds — image, tags, env, secrets, caps stay strictly opt-in via explicit profile: default). This closes a footgun where a missing profile reference produced a pod with no resources: block, leaving it OOM-killed by the namespace’s LimitRange or unbounded by the node.

Tags

Tags are how the scheduler routes a job to a compatible agent. Profile tags merge (union) with job-declared tags; an agent must carry every tag for the job to dispatch.

name: docker-builds
tags: [linux, docker]

A job inheriting this profile against an agent declaring tags: [linux, docker, gpu] matches; an agent with only [linux] does not.

Environment and secrets

env is a plain key/value map injected into every container the profile runs. Useful for non-secret runtime config (bucket names, regions, registry mirrors).

secrets are encrypted at rest with the server’s AEAD cipher and unsealed at dispatch. The API only ever returns secret keys, never values. Secret values may reference globals via {{secret:NAME}} templates so a value rotated once globally flows into every profile that references it.

Scheduling hints (v0.14.0+)

Profiles can pin job pods to specific nodes via node_selector, tolerate specific taints via tolerations, and prefer a node class via preferred_node_affinity. Honoured by the Kubernetes engine only.

node_selector: agent baseline + profile, profile wins

The Kubernetes engine builds the pod’s nodeSelector by merging two sources:

  1. Agent baseline — agent.jobNodeSelector in the Helm values.yaml (rendered to the env var GOCDNEXT_K8S_JOB_NODE_SELECTOR on the agent StatefulSet). This is the fleet-wide default applied to every pod the agent creates — task pods (shared + isolated modes), the housekeeper sidecar, and services: sidecar pods.

  2. Profile — node_selector in the runner profile (admin UI at /admin/profiles or Helm runnerProfiles[].node_selector).

On key collision, the profile wins. Profile is more specific than the agent default — a job declaring profile: gradle-heavy with pool: gradle lands on gradle nodes even when the agent baseline says pool: ci.

tolerations: agent baseline + profile, profile appends

Same two sources for tolerations. The lists concatenate with the agent baseline first; profile entries are appended. Kubelet ignores exact duplicates so dedup is not needed on the agent side.

preferred_node_affinity: soft preference, spot-first fallback

A soft node preference (k8s nodeAffinity.preferredDuringScheduling IgnoredDuringExecution). Unlike node_selector — a hard match that leaves a pod Pending when no node qualifies — a preference only biases the scheduler. An unsatisfiable preference simply has no effect, so the pod still schedules elsewhere.

The canonical use is spot-first with on-demand fallback: prefer spot nodes but fall back to on-demand when spot capacity is unavailable, instead of failing the job. Point node_selector (or a shared label) at a set of nodes that includes both classes, then prefer spot:

# Runner profile
preferred_node_affinity:
  - weight: 100                       # 1..100; higher = stronger pull
    match_expressions:
      - key: cloud.google.com/gke-spot
        operator: In
        values: ["true"]

Each term is a weight (1..100) plus one or more match_expressions that must all match (AND) for the weight to apply to a node. Operators are the node-affinity set: In, NotIn, Exists, DoesNotExist, Gt, Lt.

The profile’s affinity is applied to the task/job pod verbatim — there is no agent-baseline affinity to merge with (the agent StatefulSet’s own affinity is separate).

Services inherit the job’s profile scheduling

services:-declared sidecar pods (postgres, redis, etc.) are placed exactly like the task pod: the agent baseline merged with the profile’s node_selector, tolerations, and preferred_node_affinity. A run-shared service lands on the same pool as the jobs that use it instead of wherever the default scheduler happens to put it — which matters when the default choice is a cold pool that has to scale up a node (a real source of service-startup latency and failures).

Because services attach to the run, not a single job, a service pod inherits the scheduling of whichever job brings the services up first (the run’s jobs typically share a profile, or compatible pools, so this is deterministic in practice). One consequence to keep in mind: if a profile hard-pins node_selector to a capacity-starved pool, the service inherits that pin and can go Pending too — the same risk the task pods already carry. The recommended pattern is the shared-label + preferred_node_affinity model above (a selector that matches both spot and on-demand, preferring spot), so both jobs and services stay schedulable with a fallback.

Empty profile scheduling (or a job with no profile) leaves services on the agent baseline only — the prior behaviour, unchanged.

A service can also override the inherited scheduling — pin it to a different (e.g. more stable on-demand) pool than the jobs — via node_selector / tolerations on the service in the pipeline YAML:

services:
  - name: postgres
    image: postgres:16
    node_selector:                       # WINS over the inherited pool
      cloud.google.com/gke-nodepool: ondemand
    tolerations:                         # appended to the inherited ones
      - key: dedicated
        operator: Equal
        value: db
        effect: NoSchedule

Override precedence: node_selector keys win over the inherited ones (so the service leaves the job’s pool for the one you name), tolerations are appended. Values are validated at parse time against the same apiserver rules as a profile — a typo fails gocdnext validate, not a Pending pod hours later. Affinity is inherited-only (no per-service affinity override).

Chart values

agent:
  jobNodeSelector:
    pool: ci
  jobTolerations:
    - key: ci-only
      operator: Equal
      value: "true"
      effect: NoSchedule

Empty defaults skip the env var entirely, so the StatefulSet on an unconfigured chart matches pre-v0.14 behaviour bit-for-bit.

Validation

Both node_selector keys and values validate against the same rules the Kubernetes apiserver enforces at pod admission (IsQualifiedName / IsValidLabelValue), so a misconfig surfaces as a 400 at admin write time, not as a Pending pod hours later when the next job dispatches.

Tolerations validate operator (Equal / Exists), effect (NoSchedule / PreferNoSchedule / NoExecute / empty), Exists-with-value rejection, and toleration_seconds only with effect: NoExecute. Empty operator normalises to Equal server-side so persisted rows always carry the explicit form.

preferred_node_affinity is validated per expression against the same rules the apiserver applies (via labels.NewRequirement): key must be a qualified name; In/NotIn need at least one label-valid value; Exists/DoesNotExist take none; Gt/Lt take exactly one integer. weight must be 1–100, each term needs at least one expression, and the total counts (terms, expressions, values) are bounded so the assignment stays small.

Storage sizing (isolated mode)

A profile can size the Kubernetes isolated-mode job pod’s disks — an opt-in for the heavy jobs (a big container-image build being the classic case) without inflating every pod. All four fields are optional; empty keeps the current behaviour (agent-global workspace default, and DinD on the node’s ephemeral disk).

workspace_size: 100Gi            # override the agent-global workspace PVC size
workspace_storage_class: premium-rwo
dind_storage_size: 300Gi         # dedicated /var/lib/docker PVC for docker:true jobs
dind_storage_class: premium-rwo
  • workspace_size / workspace_storage_class override the agent-global workspace ephemeral-PVC (agent.workspace.size / storageClassName) for every isolated job on the profile.
  • dind_storage_size / dind_storage_class apply only to docker: true jobs: they mount a dedicated ephemeral PVC at /var/lib/docker, so dockerd + buildkit keep the layer store and the image export/push staging on a disk you sized — instead of the node’s ephemeral disk, whose throughput otherwise caps multi-GB image builds. On GCE, PD throughput scales with provisioned size (a 300Gi premium-rwo buys bandwidth even if only a few GB are used); a local-ssd class is faster where available. A dind_storage_size set on a non-docker job allocates nothing.

Sizes validate as positive Kubernetes quantities, classes as DNS-1123 names, at write time (admin API and the Helm seed) — a typo fails with a clear error instead of a pod stuck Pending on an unbindable PVC. See the Kubernetes runtime page for the disk-throughput rationale.

Seed via Helm

Both UI-created and Helm-seeded profiles coexist. The chart’s runnerProfiles list upserts each entry by name on boot — profiles not in the list are left alone (operator-created in the UI), and profiles in the list both create-if-missing and update-in-place.

values.yaml
runnerProfiles:
  - name: default
    description: Sensible runtime bounds for any pipeline.
    engine: kubernetes
    default_cpu_request: 100m
    default_cpu_limit: "1"
    default_mem_request: 256Mi
    default_mem_limit: 1Gi
    max_cpu: "4"
    max_mem: 8Gi
    tags: [linux]
  - name: gradle-heavy
    engine: kubernetes
    default_cpu_request: 1
    default_cpu_limit: "4"
    default_mem_request: 4Gi
    default_mem_limit: 8Gi
    max_cpu: "8"
    max_mem: 16Gi
    tags: [linux]
    node_selector:
      pool: gradle
    tolerations:
      - key: gradle-only
        operator: Equal
        value: "true"
        effect: NoSchedule

Secrets are deliberately not seeded from values.yaml — a values file commonly lives in git, plaintext credentials there are a foot-gun. Manage secrets: post-install via the admin UI (or sealed-secrets that the chart-managed entries can reference via the {{secret:NAME}} template syntax).

Audit

Every profile create / update / delete is recorded in the audit log under audit_events.action values runner_profile.create, runner_profile.update, runner_profile.delete. The metadata captures which fields changed so admins can kubectl -n gocdnext exec into the DB and reconstruct the state at any point.

See also