Loop in reconcile from operator version 0.13.0

[disaster37]

Describe the bug
I look from ArgoCD operator logs that is in on infiny loop. It never stop to reconcile my ArgoCD ressource.
On same ArgoCD ressource, no problem with operator in version 0.12.0. The problem appear on 0.13.0

After some tests, it seems there are matter on server ingress.
If I disable the ingress on server, the loop stop.

spec:
  server:
    ingress:
      enabled: false

To Reproduce
Install operator one version 0.13.0 with OLM
Create new ArgoCD cluster with

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
  name: argocd
spec:
  server:
    host: argocd.domain.local
    ingress:
      enabled: true

Expected behavior
At one moment, the operator stop to reconcile the ArgoCD ressource.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[sarfarazgit]

Kindly share some of the screen shots along with a short video for more understanding.

[disaster37]

Hum, finnaly it's seem not linked with enabled / disabled ingress. After one minute it start to loop for ever.

Little film about stdout of operator:

The logs that loop:

2025-02-11T10:11:40Z    INFO    Reconciling ArgoCD      {"controller": "argocd", "controllerGroup": "argoproj.io", "controllerKind": "ArgoCD", "ArgoCD": {"name":"argocd","namespace":"argocd"}, "namespace": "argocd", "name": "argocd", "reconcileID": "7fdb4e20-12e0-4fe5-8a83-70f1b3e089c9", "namespace": "argocd", "name": "argocd"}
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling SSO
2025-02-11T10:11:40Z    INFO    controller_argocd       Delete Keycloak deployment for ArgoCD argocd in namespace argocd
2025-02-11T10:11:40Z    INFO    controller_argocd       Found 'annotation+label' as tracking method
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling status
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling roles
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling roles for source namespaces
2025-02-11T10:11:40Z    INFO    controller_argocd       performing cleanup for source namespaces
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling rolebindings
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling service accounts
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling certificate authority
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling CA secret
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling CA config map
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling secrets
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling config maps
2025-02-11T10:11:40Z    INFO    controller_argocd       Found 'annotation+label' as tracking method
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling services
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling deployments
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling statefulsets
2025-02-11T10:11:40Z    INFO    controller_argocd       existing argocd export not found, skipping import
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling autoscalers
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling ingresses
2025-02-11T10:11:40Z    INFO    controller_argocd       not enabled
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling prometheus
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling ApplicationSet controller
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling applicationset serviceaccounts
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling applicationset roles
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling applicationset role bindings
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling applicationset deployments
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling applicationset service
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling applicationset service
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling applicationset clusterroles
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling applicationset clusterrolebindings
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling applicationset roles & rolebindings in source namespaces
2025-02-11T10:11:40Z    INFO    controller_argocd       performing cleanup for applicationset source namespaces
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling repo-server TLS secret
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling redis-server TLS secret
2025-02-11T10:11:40Z    INFO    Reconciling ArgoCD      {"controller": "argocd", "controllerGroup": "argoproj.io", "controllerKind": "ArgoCD", "ArgoCD": {"name":"argocd","namespace":"argocd"}, "namespace": "argocd", "name": "argocd", "reconcileID": "5f71b877-6766-45cd-96b4-80814aa3cbbf", "namespace": "argocd", "name": "argocd"}
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling SSO
2025-02-11T10:11:40Z    INFO    controller_argocd       Delete Keycloak deployment for ArgoCD argocd in namespace argocd
2025-02-11T10:11:40Z    INFO    controller_argocd       Found 'annotation+label' as tracking method
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling status
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling roles
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling roles for source namespaces
2025-02-11T10:11:40Z    INFO    controller_argocd       performing cleanup for source namespaces
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling rolebindings
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling service accounts
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling certificate authority
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling CA secret
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling CA config map
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling secrets
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling config maps
2025-02-11T10:11:40Z    INFO    controller_argocd       Found 'annotation+label' as tracking method
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling services
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling deployments
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling statefulsets
2025-02-11T10:11:40Z    INFO    controller_argocd       existing argocd export not found, skipping import
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling autoscalers
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling ingresses
2025-02-11T10:11:40Z    INFO    controller_argocd       not enabled
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling prometheus
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling ApplicationSet controller
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling applicationset serviceaccounts
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling applicationset roles
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling applicationset role bindings
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling applicationset deployments
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling applicationset service
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling applicationset service
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling applicationset clusterroles
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling applicationset clusterrolebindings
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling applicationset roles & rolebindings in source namespaces
2025-02-11T10:11:40Z    INFO    controller_argocd       performing cleanup for applicationset source namespaces
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling repo-server TLS secret
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling redis-server TLS secret
2025-02-11T10:11:40Z    INFO    Reconciling ArgoCD      {"controller": "argocd", "controllerGroup": "argoproj.io", "controllerKind": "ArgoCD", "ArgoCD": {"name":"argocd","namespace":"argocd"}, "namespace": "argocd", "name": "argocd", "reconcileID": "d524af61-2b05-47ac-8e2e-c0545f357b40", "namespace": "argocd", "name": "argocd"}
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling SSO
2025-02-11T10:11:40Z    INFO    controller_argocd       Delete Keycloak deployment for ArgoCD argocd in namespace argocd
2025-02-11T10:11:40Z    INFO    controller_argocd       Found 'annotation+label' as tracking method
2025-02-11T10:11:40Z    INFO    controller_argocd       reconciling status

My argocds ressources:

apiVersion: argoproj.io/v1alpha1
kind: ArgoCD
metadata:
  name: argocd
  namespace: argocd
spec:
  applicationSet:
    resources:
      limits:
        cpu: '2'
        memory: 1Gi
      requests:
        cpu: 250m
        memory: 512Mi
  banner:
    content: Gitops pipeline
  controller:
    resources:
      limits:
        cpu: '2'
        memory: 2Gi
      requests:
        cpu: 250m
        memory: 1Gi
  extraConfig:
    resource.compareoptions: |
      ignoreResourceStatusField: crd
    resource.customizations.ignoreDifferences.admissionregistration.k8s.io_MutatingWebhookConfiguration: |
      jqPathExpressions:
        - '.webhooks[]?.clientConfig.caBundle'
    resource.customizations.ignoreDifferences.admissionregistration.k8s.io_ValidatingWebhookConfiguration: |
      jqPathExpressions:
        - '.webhooks[]?.clientConfig.caBundle'
    resource.customizations.knownTypeFields.argoproj.io_Rollout: |
      - field: spec.template.spec
        type: core/v1/PodSpec
    timeout.reconciliation: 10m
    url: https://argocd.local.domain
  ha:
    enabled: true
    redisProxyImage: haproxy
  rbac:
    defaultPolicy: role:none
    policy: |
      p, role:none, *, *, */*, deny
      g, HM_ADM_ETL_Outils, role:admin
    scopes: '[groups]'
  redis:
    image: redis
    resources:
      limits:
        cpu: 500m
        memory: 256Mi
      requests:
        cpu: 250m
        memory: 128Mi
  repo:
    env:
      - name: http_proxy
        valueFrom:
          secretKeyRef:
            key: http_proxy
            name: proxy
      - name: https_proxy
        valueFrom:
          secretKeyRef:
            key: https_proxy
            name: proxy
      - name: no_proxy
        value: >-
          .domain.local,localhost,127.0.0.1,.cluster.local,rgocd-application-controller,argocd-dex-server,redis-ha,argocd-repo-server,argocd-server
    initContainers:
      - args:
          - >
            set -e


            # Download avp

            curl -L
            https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64
            -o argocd-vault-plugin &&

            chmod +x argocd-vault-plugin &&

            mv argocd-vault-plugin /custom-tools/


            # Download helm

            curl -L https://get.helm.sh/helm-v${HELM_VERSION}-linux-amd64.tar.gz
            | tar -xvzf -

            chmod +x linux-amd64/helm

            mv linux-amd64/helm /custom-tools/
        command:
          - sh
          - '-c'
        env:
          - name: AVP_VERSION
            value: 1.18.1
          - name: HELM_VERSION
            value: 3.17.0
        envFrom:
          - secretRef:
              name: proxy
        image: registry.access.redhat.com/ubi8
        name: download-tools
        volumeMounts:
          - mountPath: /custom-tools
            name: custom-tools
      - args:
          - >
            helm registry login -u ${HELM_REGISTRY_HM_USERNAME} -p
            ${HELM_REGISTRY_HM_PASSWORD} ${HELM_REGISTRY_HM_URL}
        command:
          - sh
          - '-c'
        env:
          - name: HELM_CONFIG_HOME
            value: /helm-working-dir
        envFrom:
          - secretRef:
              name: proxy
          - secretRef:
              name: helm
        image: registry.access.redhat.com/ubi8
        name: helm-registry-auth
        securityContext:
          runAsNonRoot: true
          runAsUser: 999
        volumeMounts:
          - mountPath: /usr/local/bin/helm
            name: custom-tools
            subPath: helm
          - mountPath: /helm-working-dir
            name: helm-working-dir
          - mountPath: /etc/ssl/certs/hm-ca.crt
            name: hm-ca
            subPath: hm-ca.crt
    resources:
      limits:
        cpu: '1'
        memory: 1Gi
      requests:
        cpu: 500m
        memory: 256Mi
    sidecarContainers:
      - command:
          - /var/run/argocd/argocd-cmp-server
        env:
          - name: VAULT_ADDR
            value: https://vault.domain.local
          - name: AVP_TYPE
            value: vault
          - name: AVP_AUTH_TYPE
            value: approle
          - name: AVP_ROLE_ID
            valueFrom:
              secretKeyRef:
                key: AVP_ROLE_ID
                name: vault
          - name: AVP_SECRET_ID
            valueFrom:
              secretKeyRef:
                key: AVP_SECRET_ID
                name: vault
        image: registry.access.redhat.com/ubi8
        name: avp
        securityContext:
          runAsNonRoot: true
          runAsUser: 999
        volumeMounts:
          - mountPath: /var/run/argocd
            name: var-files
          - mountPath: /home/argocd/cmp-server/plugins
            name: plugins
          - mountPath: /tmp
            name: cmp-avp-tmp
          - mountPath: /etc/ssl/certs/hm-ca.crt
            name: hm-ca
            subPath: hm-ca.crt
          - mountPath: /home/argocd/cmp-server/config/plugin.yaml
            name: cmp-plugin
            subPath: avp.yaml
          - mountPath: /usr/local/bin/argocd-vault-plugin
            name: custom-tools
            subPath: argocd-vault-plugin
      - command:
          - /var/run/argocd/argocd-cmp-server
        env:
          - name: VAULT_ADDR
            value: https//vault.domain.local
          - name: AVP_TYPE
            value: vault
          - name: AVP_AUTH_TYPE
            value: approle
          - name: AVP_ROLE_ID
            valueFrom:
              secretKeyRef:
                key: AVP_ROLE_ID
                name: vault
          - name: AVP_SECRET_ID
            valueFrom:
              secretKeyRef:
                key: AVP_SECRET_ID
                name: vault
          - name: http_proxy
            valueFrom:
              secretKeyRef:
                key: http_proxy
                name: proxy
          - name: https_proxy
            valueFrom:
              secretKeyRef:
                key: https_proxy
                name: proxy
          - name: no_proxy
            value: >-
              .domain.local,localhost,127.0.0.1,.cluster.local,rgocd-application-controller,argocd-dex-server,redis-ha,argocd-repo-server,argocd-server
          - name: HELM_CACHE_HOME
            value: /tmp
          - name: HELM_CONFIG_HOME
            value: /helm-working-dir
        image: registry.access.redhat.com/ubi8
        name: avp-helm
        securityContext:
          runAsNonRoot: true
          runAsUser: 999
        volumeMounts:
          - mountPath: /var/run/argocd
            name: var-files
          - mountPath: /home/argocd/cmp-server/plugins
            name: plugins
          - mountPath: /tmp
            name: cmp-avp-helm-tmp
          - mountPath: /etc/ssl/certs/hm-ca.crt
            name: hm-ca
            subPath: hm-ca.crt
          - mountPath: /home/argocd/cmp-server/config/plugin.yaml
            name: cmp-plugin
            subPath: avp-helm.yaml
          - mountPath: /usr/local/bin/argocd-vault-plugin
            name: custom-tools
            subPath: argocd-vault-plugin
          - mountPath: /usr/local/bin/helm
            name: custom-tools
            subPath: helm
          - mountPath: /helm-working-dir
            name: helm-working-dir
    volumeMounts:
      - mountPath: /etc/ssl/certs/hm-ca.crt
        name: hm-ca
        subPath: hm-ca.crt
    volumes:
      - name: hm-ca
        secret:
          secretName: pki
      - emptyDir: {}
        name: custom-tools
      - configMap:
          name: cmp-plugin
        name: cmp-plugin
      - emptyDir: {}
        name: cmp-avp-helm-tmp
      - emptyDir: {}
        name: cmp-avp-tmp
      - emptyDir: {}
        name: helm-working-dir
  repositoryCredentials: |
    - url: https://github.com/disaster37
      passwordSecret:
        name: github
        key: password
      usernameSecret:
        name: github
        key: username
      proxy: 'http://squid.squid.svc.cluster.local:8080'
  resourceTrackingMethod: annotation+label
  server:
    autoscale:
      enabled: true
    host: argocd.domain.local
    ingress:
      annotations:
        nginx.ingress.kubernetes.io/backend-protocol: HTTPS
        nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'
        nginx.ingress.kubernetes.io/proxy-buffer-size: 16k
        nginx.ingress.kubernetes.io/ssl-passthrough: 'true'
      enabled: true
      tls:
        - hosts:
            - argocd.domain.local
    resources:
      limits:
        cpu: '1'
        memory: 1Gi
      requests:
        cpu: 500m
        memory: 256Mi
  sso:
    dex:
      config: |
        connectors:
          - type: ldap
            name: Active Directory
            id: ad
            config:
              host: dcn.domain.local:636
              insecureNoSSL: false
              insecureSkipVerify: true
              bindDN: $dex:ldapUsername
              bindPW: $dex:ldapPassword
              usernamePrompt: Username
              userSearch:
                baseDN: "DC=DOMAIN,DC=LOCAL"
                filter: ""
                username: sAMAccountName
                idAttr: distinguishedName
                emailAttr: sAMAccountName
                nameAttr: displayName
              groupSearch:
                baseDN: "DC=DOMAIN,DC=LOCAL"
                filter: ""
                userAttr: distinguishedName
                groupAttr: member
                nameAttr: name
      resources:
        limits:
          cpu: 500m
          memory: 256Mi
        requests:
          cpu: 250m
          memory: 128Mi
    provider: dex
  statusBadgeEnabled: true
  version: v2.14.2

[svghadi]

Can you share output of .status of your ArgoCD? Also, try to restart the operator manager deployment, see if that fixes it.

[sarfarazgit]

I am observing the reconcilation logs but slowly when I set the enabled: true :

---

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
creationTimestamp: '2025-02-11T11:50:30Z'

spec:
server:
autoscale:
enabled: false
ingress:
enabled: true

---

2025-02-11T12:32:07Z INFO controller_argocd reconciling CA secret
2025-02-11T12:32:07Z INFO controller_argocd reconciling CA config map
2025-02-11T12:32:07Z INFO controller_argocd reconciling secrets
2025-02-11T12:32:07Z INFO controller_argocd reconciling config maps
2025-02-11T12:32:07Z INFO controller_argocd Using default resource tracking method 'label'
2025-02-11T12:32:07Z INFO controller_argocd reconciling services
2025-02-11T12:32:07Z INFO controller_argocd skipping AutoTLS on service argocd-sample-server since the TLS secret is already present
2025-02-11T12:32:07Z INFO controller_argocd reconciling deployments
2025-02-11T12:32:08Z INFO controller_argocd reconciling statefulsets
2025-02-11T12:32:08Z INFO controller_argocd existing argocd export not found, skipping import
2025-02-11T12:32:08Z INFO controller_argocd reconciling autoscalers
2025-02-11T12:32:08Z INFO controller_argocd reconciling ingresses
2025-02-11T12:32:08Z INFO controller_argocd not enabled
2025-02-11T12:32:08Z INFO controller_argocd reconciling routes
2025-02-11T12:32:08Z INFO controller_argocd Using reencrypt termination policy for the Server Route
2025-02-11T12:32:08Z INFO controller_argocd reconciling prometheus
2025-02-11T12:32:08Z INFO controller_argocd reconciling repo-server TLS secret
2025-02-11T12:32:08Z INFO controller_argocd reconciling redis-server TLS secret

[sarfarazgit]

Hi,

Can you try with this annotation?

---

metadata:
annotations:
argocd.argoproj.io/skip-reconcile: "true"

---

https://argo-cd.readthedocs.io/en/latest/user-guide/skip_reconcile/#:~:text=Argo%20CD%20allows%20users%20to,is%20stopped%20for%20the%20Application.

[disaster37]

My status:

status:
  applicationController: Running
  applicationSetController: Running
  host: 10.221.101.140, 10.221.101.141, 10.221.101.142
  phase: Available
  redis: Running
  repo: Running
  server: Running
  sso: Running

Same matter after restart argocd-operator pod.

[xrow]

Hi,

I have this loop because of rancher changing my ingress. rancher/rancher#42058 (comment)

The worst part ist the cert-manger can`t add a acme tls cert on a changing object.

I am going to create now my own ingress for argo

[xrow]

@sarfarazgit I can not see how argocd.argoproj.io/skip-reconcile: "true" can help since the operator does the loop. The proposal seems invalid I can`t find this annoation in the operators code base.

[svghadi]

Hi @xrow , currently argocd-operator overrides/removes labels and annotations added on it's owned resources by third party applications. I feels this behaviour needs to be changed. The operator should be only concerned about labels and annotations it adds on the resources and shouldn't remove any extra labels and annotations added by external applications. We have discussed this internally in past and have decide to make a design change to allow this. Hopefully that should resolve your issue.

[xrow]

@svghadi Thanks for your comment. This sounds like the right thing todo.

[disaster37]

Hi, it still the same on 0.14.1