[FEATURE] Opt-in to pull internal k3d images from registry mirror

[erikgb]

Is your feature request related to a problem or a Pull Request

No, but I have an unregistered issue discussed with @iwilltry42 on Slack. 😉

Scope of your request

We are "stranded" in a semi air-gapped environment, where all container images must be pulled through an internal registry proxy (Artifactory). This means that all Internet image urls must be re-written, either explicitly or indirectly using some kind of mirroring configuration.

This was causing some initial challenges when we adopted k3d, but since k3s (using containerd) allows us to configure mirrors for all images pulled from within the cluster, we were almost able to work around the problems. We were still having issues with pulling the k3d images (k3d-proxy and k3d-tools), but the last piece in the puzzle was landed supplying the --registry-mirror Docker setting - allowing us to pull the k3d images through our on-prem proxy - as long as they were hosted on DockerHub.

But after k3d became an independent project, these images are now hosted by ghcr.io. And that blocks our current workaround, since Docker only allow you to configure a mirror for DockerHub itself:

It’s currently not possible to mirror another private registry. Only the central Hub can be mirrored.

Here is the relevant code i k3d:

k3d/pkg/types/images.go

Lines 48 to 64 in f2df55a

	func GetLoadbalancerImage() string {
	if img := os.Getenv(K3dEnvImageLoadbalancer); img != "" {
	l.Log().Infof("Loadbalancer image set from env var $%s: %s", K3dEnvImageLoadbalancer, img)
	return img
	}
	return fmt.Sprintf("%s:%s", DefaultLBImageRepo, GetHelperImageVersion())
	}
	func GetToolsImage() string {
	if img := os.Getenv(K3dEnvImageTools); img != "" {
	l.Log().Infof("Tools image set from env var $%s: %s", K3dEnvImageTools, img)
	return img
	}
	return fmt.Sprintf("%s:%s", DefaultToolsImageRepo, GetHelperImageVersion())
	}

Describe the solution you'd like

I would like to see an opt-in for some registry override/mirror to pull the k3d images, similar to what k3s provides. We are currently pulling cluster workload images from ghcr.io (and a lot of other Internet image registries), so we supply registries config to k3d, that works:

registries:
  config: |
    mirrors:
      docker.io:
        endpoint:
          - dockerhub-docker-remote.hub.mycompany.com
      gcr.io:
        endpoint:
          - gcr-docker-remote.hub.mycompany.com
      ghcr.io:
        endpoint:
          - ghcr-docker-remote.hub.mycompany.com

My preferred solution would be if k3d could use this config to eventually use a mirror configured for k3s to pull it's internal images from ghcr.io. The mirror should only be used if the mirror key matches ghcr.io exactly. There are a few other options that can be configured in k3s registries.yaml (certificates, authentication etc.), but I am not sure if all have to be supported from day one?

An alternative solution could be to allow for configuring image registry (or image repositories) in the k3d specific config. In our case a registry would be the simplest, and sufficient. And that would make it correspond to the k3s settings - more or less. But some users may want to override the image repositories (one per image), and that would also work for us - but will require more configuration.

An idea on how this might fit into the configuration file:

options:
  k3d: # k3d runtime settings
    imageRegistry: ghcr-docker-remote.hub.mycompany.com

Describe alternatives you've considered

We can override the full image URLs, including the version tag, using the environment variables K3D_IMAGE_LOADBALANCER and K3D_IMAGE_TOOLS, and we probably will do for now - as a workaround. But that clutters our configuration of k3d IMO, and we would have to sync the version bumps.

[PierreRustOrange]

Many thanks for this issue, it helped me a lot to understand why my cluster was failing after upgrading k3d !

I'm in the exact same situation, I can only access internal mirrors (served by artifactory) of public repositories. I managed to get it working using the environnement K3D_IMAGE_LOADBALANCER and K3D_IMAGE_TOOLS but these are nowhere to be found in the documentation and the changelog does not provide many insight either ; I doubt I would have figured it out without your explanation.

I personally prefer the second alternative as it makes it clear that there are two different setups and mechanisms for pulling images, one for k3d and one for k3s.

Until a better solution is found, maybe the workaround using the environment variables should be mentioned somewhere in the documentation.

[iwilltry42]

Hi @erikgb , thanks for opening this (exceptionally well described) issue!
I hope I'll find some time to work on this soon.

Given my experience with the Docker SDK, I fear that supporting the whole registry config at some point would be to much of a promise for being feasible.
However, just adding some config options should do the trick and be good enough. That would leave all the auth/certs part to docker itself (docker login).

@PierreRustOrange you're correct. They're only in the changelog (main purpose was my own development): https://github.com/k3d-io/k3d/blob/7b8c0f483ff124f217d14852ca4e937701e9003e/CHANGELOG.md#features--enhancements-2

I personally prefer the second alternative as it makes it clear that there are two different setups and mechanisms for pulling images, one for k3d and one for k3s.

I agree with this 👍

[omniproc]

Could someone explain why the registry - ghcr.io/ - is hardcoded to the image names to be used by k3d?
Why would you enforce a registry? The repository / image / tag combination should be unique. The registry should be replaceable.

That seems to be the reason for the issue to start with. If the registry wouldn't be hardcoded in k3d then the already in place solution --registry-mirror or - in case Docker is used which disables cri in containerd, makeing that flag useless - the default Docker daemon.json could be used (as it was before the switch to ghcr.io).

It's the job of the registry to figure out which image is the right one if multiple with the same repository / image / tags are available in the given registry.