This is really odd. I’ve given Kubes access but GLR needs access as well.
The below works to give Kubes access to private images, but I still haven’t worked out giving GLR access.
I kind-of assumed that Gitlabs agent would handle this for me already, but I was apparently wrong on that. To add insult to injury, I had to deal with this with Flux CD but I was too busy trying to get it working to document what I did.
I ran into this issue recently while rebuilding my hosting companies infrastructure, and decided to finally deploy Ansible while I have the moment. To achieve this, I have created a defaults private Gitlab repository that contains all of my default configurations, and I have a private Ansible image (based on dxcker/ansible of course) which has some SSH configurations.
In order to use the private Ansible image, my K3S cluster needs a way to authenticate with Gitlab.
The error that I saw when trying to use the private image in my job:
Running with gitlab-runner 16.10.0 (81ab07f6)
on gitlab-runner-host-859d669458-cv8jl bMCC8BY8H, system ID: r_PqWauaOWZGqwe
Preparing the "kubernetes" executor 00:00
Using Kubernetes namespace: gitlab-runner
Using Kubernetes executor with image registry.gitlab.com/host/ansible/image:latest ...
Using attach strategy to execute scripts...
Preparing environment 00:06
Using FF_USE_POD_ACTIVE_DEADLINE_SECONDS, the Pod activeDeadlineSeconds will be set to the job timeout: 1h0m0s...
Waiting for pod gitlab-runner/runner-bmcc8by8h-project-36423659-concurrent-0-0q3chqa4 to be running, status is Pending
Waiting for pod gitlab-runner/runner-bmcc8by8h-project-36423659-concurrent-0-0q3chqa4 to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
WARNING: Failed to pull image with policy "": image pull failed: failed to pull and unpack image "registry.gitlab.com/host/ansible/image:latest": failed to resolve reference "registry.gitlab.com/host/ansibles/image:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
ERROR: Job failed: prepare environment: waiting for pod running: pulling image "registry.gitlab.com/host/ansible/image:latest": image pull failed: failed to pull and unpack image "registry.gitlab.com/host/ansibles/image:latest": failed to resolve reference "registry.gitlab.com/host/ansibles/image:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
To resolve this, you need to create a secret containing your login information. This is actually covered by the kubernetes documentation
It seems to focus rather heavily on the Docker config, so we’ll have to roll with that. I do believe there’s a better option that putting the docker config into a generic secret, but I’m just going to roll with the above because I need it fixed ASAP.
CLI Way:
kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>
Manifest way:
Create a file or something (makes it easier), and enter your docker conf:
{"auths":{"registry.gitlab.com":{"username":"gitlab-runner-host","password":"gldt-xxxxxxxxxxxxxxxxxxxx","email":"xxxxxxxxxxx","auth":""}}}
Save and close, then run the following to get the auth value:
cat dockerconf | jq '.auths."registry.gitlab.com" | .username, .password' | tr '\n' ':' | sed 's/:$//' | sed 's/"//g' | base64
It’s not a very refined script, but it will give you the has code needed. Update your dockerconf file with the hash code in the auth field, then base64 encode the whole file
cat dockerconf | base64 -w 0
Now you can create the secret manifest that the cli otherwise creates:
---
apiVersion: v1
kind: Secret
metadata:
name: gitlab-runner-dockerconf-host
namespace: default
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: eyJhdXRocyI6eyJyZWdpc3RyeS5naXRsYWIuY29tIjp7InVzZXJuYW1lIjoibm9jcmVkcyIsInBhc3N3b3JkIjoieHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eCIsImVtYWlsIjoiZGRkZGRkZGRkZGQiLCJhdXRoIjoieHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4In19fQo=
Once you have applied that, you just have to apply it to the default service account to be globally used. It’s up to you if this is opsec.
To do that, use the below command:
kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "gitlab-runner-dockerconf-host"}]}'
I ended up finding Chris Vermeulens post about this, it came to the same conclusion as I did however he gave me the SA patch:
https://chris-vermeulen.com/using-gitlab-registry-with-kubernetes/