Although originally using Gitea for hosting, and Drone for my CI engine, this setup proved to be difficult. I continually had problems with Done, and actually had the Gitea database corrupt at one point. With all this I decided to just move back to Gitlab. They run a great service, and it’s what we use at work so I am quite familiar with them and their CI/CD setup. I installed the Gitlab Runner on my internal K3s cluster, and am now using that to do all my builds and deployments behind my firewall. I hit a few bumps along the way, so I wanted to document the final setup for posterity and anyone else who might be having issues as well.
Gitlab Runners
Gitlab runners are deployments/pods that run on your cluster and periodically check in with Gitlab to see if there are any builds to run. They can be attached to your projects in one of two ways. The first way is to register a runner for every project. Although you can do this, frankly it’s a little crazy. Maybe if a single project has a very special build environment that is different from the rest of your setup, but generally this would not be the recommended way to use runners. The second way is to attach the runner to a group. Now, if you are in an enterprise/work environment this could mean that you attach them at the top most level (i.e. your organization), or maybe for each team that has it’s own group. For personal use, although it seems weird, you need to create a group and then attach the Gitlab runner to that specific group. This will allow every project in that group to access the runner.
Once you have your group setup, navigate to the “CI/CD” section and click on “Runners”. Up in the top corner you will see a drop down that says “Register a group runner”. From here you will see the registration token. Save that as we will need it for the Helm chart next.
Installing the Runner using Helm
Now add the Helm chart for the runner and update the repos:
$ helm repo add gitlab https://charts.gitlab.io
$ helm repo update
Next you will need to setup your values.yaml file. Here is the full values.yaml from Gitlab. Below is the values.yaml that I used:
image:
registry: registry.gitlab.com
image: gitlab-org/gitlab-runner
# tag: alpine-v11.6.0
imagePullPolicy: IfNotPresent
revisionHistoryLimit: 2
gitlabUrl: https://gitlab.com
runnerRegistrationToken: "CHANGE ME BEFORE RUNNING"
terminationGracePeriodSeconds: 3600
concurrent: 10
checkInterval: 30
logLevel: warn
logFormat: text
rbac:
create: true
rules:
- apiGroups: [""]
resources: ["pods", "configmaps", "jobs", "secrets", "cronjobs", "services", "deployments", "daemonsets", "statefulsets", "pods/attach", "pods/exec"]
verbs: ["get", "list", "watch", "create", "update","delete"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
clusterWideAccess: true
podSecurityPolicy:
enabled: false
resourceNames:
- gitlab-runner
runners:
config: |
[[runners]]
[runners.kubernetes]
namespace = "{{.Release.Namespace}}"
image = "ubuntu:16.04"
service_account = "gitlab-runner"
maximumTimeout: "3600"
runUntagged: true
A few important things to point out in here. First, make sure you set your runnerRegistrationToken. This is what tells Gitlab what group the runner belongs to. Second, is the RBAC security setup. My runner is fairly restricted to the things that I think it would do on a normal Helm deployment. You may need more or less. I also have clusterWideAccess set to true in order to allow my runner to work across namespaces. Finally, under the runners configuration make sure to set the service_account that the runners will run as! This is kind of a hidden setting and I can tell you from an hour of pounding my head into a wall that deployments will not work without this setting.
Once your values.yaml file is ready to go, install it into your cluster. I like to keep my resources separate so I deployed it to a new namespace named gitlab:
$ k create ns gitlab
$ kns gitlab
$ helm install gitlab-runner gitlab/gitlab-runner -f values.yaml
Once your pod is installed and running you should see it online in the Gitlab interface under the “Runners” section where we got the registration token from before. Now it’s time to use it!
Setting up CI/CD with Gitlab Runner
I’m not going to explain a whole lot here as everyone’s setup is going to be different. I do want to go over two major pieces here though. Building containers without access to a Docker daemon, and deploying into your cluster using Helm.
Building containers
When using shared runners with Gitlab, you generally use a dind container (Docker in Docker) which allows you to build a container as your normally would. This is not the case now with Kubernetes. For building containers with Kubernetes, I highly recommend using Kaniko. Here is the build section out of my .gitlab-ci.yml template that uses Kaniko and the built-in Gitlab container registry (because it’s free). This works out of the box with no additional configuration:
build:
image:
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
stage: build
variables:
REGISTRY: ${CI_REGISTRY_IMAGE}
TAG: ${CI_COMMIT_SHORT_SHA}
KANIKO_VERBOSITY: "info"
script:
- mkdir -p /kaniko/.docker
- export GITLAB_AUTH=$(echo -n "${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')
- echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"${GITLAB_AUTH}\"}}}" > /kaniko/.docker/config.json
- >-
/kaniko/executor
--cache=true
--context "${CI_PROJECT_DIR}"
--dockerfile "${CI_PROJECT_DIR}/Dockerfile"
--destination ${REGISTRY}:${TAG}
--verbosity ${KANIKO_VERBOSITY}
Deploying with Helm
As long as you setup your runners correctly, the snippet of code below should just work™. Make sure to change the name and namespace for your own project.
deploy:
image: alpine/helm:latest
stage: deploy
variables:
TAG: ${CI_COMMIT_SHORT_SHA}
script:
- cd helm
- helm upgrade NAME . -f values.yaml -n NAMESPACE --set image.tag=${TAG}
only:
- master
- main
This is where you will see failures if you didn’t setup the service account for the runners. If that is the case you will probably see an error that says something like:
Error: UPGRADE FAILED: query: failed to query with labels: secrets is forbidden: User "system:serviceaccount:gitlab:default" cannot list resource "secrets" in API group "" in the namespace "gitlab"
This error message is telling you that the serviceAccount it’s using is in the gitlab namespace, and it’s named default. This is not correct, as the RBAC settings for the installation created a serviceAccount user named gitlab-runner. Go back and double check that your serviceAccount was created and make sure your runner configuration is set to use it.
I hope this helps someone to not have to search around and stumble for a few hours like I did!