TLS Connection Reset Issue with exo.io Registry

[oguzerdo]

TLS Connection Reset Issue with exo.io Registry

Problem Description

During stackablectl demo install end-to-end-security installations, all connection attempts to sos-ch-gva-2.exo.io consistently fail with connection reset errors. This prevents the demo from being installed successfully.

Error Details

ERROR: Go wrapper function go_install_helm_release encountered an error: failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://sos-ch-gva-2.exo.io/great-sammet.s3.container-registry.com/docker/registry/v2/blobs/sha256/[...]/data?[...]": read tcp 192.168.1.6:[port]->194.182.160.205:443: read: connection reset by peer

Direct curl tests confirm the issue:

curl -v "https://sos-ch-gva-2.exo.io"
* TLS handshake, Client hello (1)
* Recv failure: Connection reset by peer

Questions

1. Is there an alternative endpoint/registry that can be used instead of sos-ch-gva-2.exo.io?
2. Is there a configuration option for stackablectl to use a different registry mirror?
3. Are there known connectivity issues with this registry from certain geographic regions?

System details:

• OS: macOS Apple Silicon
• Kubernetes: Kind cluster

[sbernauer]

That is indeed very strange!
I have no idea where this URL is coming from, I ran stackablectl demo install end-to-end-security --log-level trace | tee log.txt (demo installed normally) and did not find any matches for exo.io. curl -v "https://sos-ch-gva-2.exo.io" works just fine for me (from Germany), so it seem to be some geographical thing...

The only helm-chart the end-to-end-security demo installs is the postgresql chart from bitnami (see https://raw.githubusercontent.com/stackabletech/demos/release-24.11/stacks/_templates/postgresql-hive-iceberg.yaml).
You you please confirm this by trying to run helm install foo postgresql --repo https://charts.bitnami.com/bitnami/ --version 16.1.2 --debug?

[nightkr]

Hm, the URL implies that it's either a redirect managed by Container Registry (our OCI hosting provider) or a mirror configured on your end. To rule out the latter, it would be helpful to check your daemon.json (when using a native Docker Engine it should be located in /etc/docker/daemon.json, when using Docker Desktop it should be in ~/.docker/daemon.json).

I also have to mirror @sbernauer's comment, curl -v "https://sos-ch-gva-2.exo.io" gives me a 403 error, not an instant disconnect.

[nightkr]

Okay, I tried reproducing the flow by hand, and it is indeed a resource used by our hosting provider.

This is what happens during a pull (I'm using trino-operator 0.0.0-dev here, but the same thing should happen for any of our operators...):

1. Try to download the manifest list - curl -v 'https://oci.stackable.tech/v2/sdp/trino-operator/manifests/0.0.0-dev' (docs)
   The endpoint replies with a 401 since you have no authentication token, header www-authenticate: Bearer realm="https://oci.stackable.tech/service/token",service="harbor-registry",scope="repository:sdp/trino-operator:pull" indicates that you need to request a token from https://oci.stackable.tech/service/token
2. Request a token - curl -v 'https://oci.stackable.tech/service/token?service=harbor-registry&scope=repository:sdp/trino-operator:pull' (docs)
   The endpoint replies with a token that we'll need to provide from now on, let's save that - BEARER="$(curl -v 'https://oci.stackable.tech/service/token?service=harbor-registry&scope=repository:sdp/trino-operator:pull' | jq --raw-output .token)"
3. Try to download the manifest list again, with the token this time - curl -v 'https://oci.stackable.tech/v2/sdp/trino-operator/manifests/0.0.0-dev' --oauth2-bearer $BEARER
   This now returns a manifest list, I select 4c277d3324a18f28c7b7c90cae5d3798d9e34dd26b97f3887d15fc1ffea82212 since I'm on an amd64 system.
4. Try to download the specific manifest - curl -v 'https://oci.stackable.tech/v2/sdp/trino-operator/manifests/sha256:4c277d3324a18f28c7b7c90cae5d3798d9e34dd26b97f3887d15fc1ffea82212' --oauth2-bearer $BEARER
   This returns a list of layers, which we need to grab separately.
5. Try to download one of the layers - curl -v 'https://oci.stackable.tech/v2/sdp/trino-operator/blobs/sha256:1f58f788813e359fac3b5b98864d02a578dbb122adf4e02b373b34161fb74c11' --oauth2-bearer $BEARER
   Instead of returning the blob directly, this returns a redirect to https://sos-ch-gva-2.exo.io/great-sammet.s3.container-registry.com/docker/registry/v2/blobs/sha256/1f/1f58f788813e359fac3b5b98864d02a578dbb122adf4e02b373b34161fb74c11/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=EXO61840b833cddf0ed913e4004%2F20250325%2Fch-gva-2%2Fs3%2Faws4_request&X-Amz-Date=20250325T225650Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=0d87f94462b66ba15ab4eb421bf8907200db52f8d1686487b28e2a814ffd01b9, which is a presigned S3-ish URL.
6. Follow the redirect - curl -v 'https://sos-ch-gva-2.exo.io/great-sammet.s3.container-registry.com/docker/registry/v2/blobs/sha256/1f/1f58f788813e359fac3b5b98864d02a578dbb122adf4e02b373b34161fb74c11/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=EXO61840b833cddf0ed913e4004%2F20250325%2Fch-gva-2%2Fs3%2Faws4_request&X-Amz-Date=20250325T225650Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=0d87f94462b66ba15ab4eb421bf8907200db52f8d1686487b28e2a814ffd01b9'

This whole chain works fine on my end, but step 6 seems to be where things are going wrong. exo.io doesn't have a useful website, but going via .io's WHOIS service, it seems to delegate to exoscale.com, which does. In particular, this looks an awful lot like their Object Storage product. Are you able to access their other zones? https://sos-de-fra-1.exo.io/ (Frankfurt, DE) seems to give me the same 403 error as ch-gva-2, for example.

[Jimvin]

Are you going and DNS-based filtering on your network? The hostname sos-de-fra-1.exo.io does appear on some popular DNS RBLs so it could be you are blocking this. Would you try running host sos-de-fra-1.exo.io on your end to see what IP address the hostname resolves to?

For info I see a connection refused in my home network when trying to connect to this hostname because the default behaviour of my ad blocking DNS resolvers is to return an address of 0.0.0.0 for blocked entities. Searching for this hostname in the Pi-hole console shows an exact match on two of the RBL lists I have configured.

It could be another network device such as a content aware firewall that's causing your issue, that would be more in line with the connection resets you are seeing.

[oguzerdo]

Thanks for your responses. I think it's due to the restrictions in my country (Turkey). Now I tried and I can access with VPN.

[lfrancke]

We just migrated the registry to being hosted on Cloudflare R2 last night.
Maybe it's worth checking if that works better for you.

Other than that: We don't have much experience with the restrictions in Turkey. If you can provide any recommendations let us know!