Generic Database connection design

[sbernauer]

We want a generic way of for database connections like mysql, postgres, mongodb

This is done when

### Tasks
- [ ] https://github.com/stackabletech/documentation/pull/321
- [ ] https://github.com/stackabletech/druid-operator/issues/352
- [ ] https://github.com/stackabletech/hive-operator/issues/148

Please note that this list only includes the products that currently define the credentials within the Product CRD (which is very bad). There are other operators, which should be adopted to the new connection design as well for consistency.

Current Status

Airflow

• spec.clusterConfig.credentialsSecret points to a secret
• The secret contains user / pw credentials as well as Database connection information -> should be split up (ADR: ADR 29: Standardize database connections documentation#321)
• The secret contains a connections.secretKey that should be created / read by the operator

Druid

• spec.clusterConfig.metaStorageDatabase should use a DatabaseConnection (ADR: ADR 29: Standardize database connections documentation#321)

HBase

Hadoop HDFS

Hive

• spec.clusterConfig.database should use a DatabaseConnection (ADR: ADR 29: Standardize database connections documentation#321)

Kafka

NiFi

OPA

Spark

Superset

• spec.clusterConfig.credentialsSecret points to a secret
• The secret contains user / pw credentials as well as Database connection information -> should be split up
• The secret contains a connections.secretKey that should be created / read by the operator

Trino

ZooKeeper

These are my findings. I hope this is complete. Does not look as bad or as much as expected.

[fhennig]

@razvan I'm not sure this can be closed, there are still open tasks

[razvan]

It was closed automatically by merging the ADR PR. I missed the comment.

[Maleware]

As long as this is not implemented you can workaround with Podovrerrites e.g. like this:

To avoid searching, this is the line added to args: sed -i "s|XX_PASS_XX|$(cat /tmp/postgres-password/password)|g" /stackable/config/hive-site.xml

---
apiVersion: hive.stackable.tech/v1alpha1
kind: HiveCluster
metadata:
  name: hive
  namespace: my-namespace
spec:
  image:
    productVersion: 3.1.3
    stackableVersion: "23.7.0"
  clusterConfig:
    database:
      connString: jdbc:postgresql://my-postgress-connector/hive
      user: hive
###### This is what will be taken from a secret ########
      password: XX_PASS_XX
      dbType: postgres
    s3:
      reference: ionos
  metastore:
    config:
      resources:
        memory:
          limit: '1Gi'
####### Podoverrides to mount the volume and sed the pw from the secret
    podOverrides:
      spec:
        containers:
          - name: hive
            args:
              - echo copying /stackable/mount/config to /stackable/config && cp -RL /stackable/mount/config/* /stackable/config && echo copying /stackable/mount/log-config/hive-log4j2.properties to /stackable/config/hive-log4j2.properties && cp -RL /stackable/mount/log-config/hive-log4j2.properties /stackable/config/hive-log4j2.properties && keytool -importkeystore -srckeystore /etc/pki/java/cacerts -srcstoretype jks -srcstorepass changeit -destkeystore /stackable/truststore.p12 -deststoretype pkcs12 -deststorepass changeit -noprompt && echo replacing xxx_access_key_xxx and xxx_secret_key_xxx with secret values. && sed -i "s|xxx_access_key_xxx|$(cat /stackable/secrets/accessKey)|g" /stackable/config/hive-site.xml && sed -i "s|xxx_secret_key_xxx|$(cat /stackable/secrets/secretKey)|g" /stackable/config/hive-site.xml && sed -i "s|XX_PASS_XX|$(cat /tmp/postgres-password/password)|g" /stackable/config/hive-site.xml && bin/start-metastore --config /stackable/config --db-type postgres --hive-bin-dir bin
            volumeMounts:
              - mountPath: /tmp/postgres-password/
                name: postgres-password
        volumes:
          - name: postgres-password
            secret:
              secretName: hive-postgres-secret
      default:
        replicas: 1

[ruslanguns]

• echo copying /stackable/mount/config to /stackable/config && cp -RL /stackable/mount/config/* /stackable/config && echo copying /stackable/mount/log-config/hive-log4j2.properties to /stackable/config/hive-log4j2.properties && cp -RL /stackable/mount/log-config/hive-log4j2.properties /stackable/config/hive-log4j2.properties && keytool -importkeystore -srckeystore /etc/pki/java/cacerts -srcstoretype jks -srcstorepass changeit -destkeystore /stackable/truststore.p12 -deststoretype pkcs12 -deststorepass changeit -noprompt && echo replacing xxx_access_key_xxx and xxx_secret_key_xxx with secret values. && sed -i "s|xxx_access_key_xxx|$(cat /stackable/secrets/accessKey)|g" /stackable/config/hive-site.xml && sed -i "s|xxx_secret_key_xxx|$(cat /stackable/secrets/secretKey)|g" /stackable/config/hive-site.xml && sed -i "s|XX_PASS_XX|$(cat /tmp/postgres-password/password)|g" /stackable/config/hive-site.xml && bin/start-metastore --config /stackable/config --db-type postgres --hive-bin-dir bin

I would like to comment that this workaround has worked perfectly for me. Although I would like to understand a little bit how it works underneath., as I can appreciate that there is more to it than just replacing the database password with the placeholder, @Maleware could you comment on why?

[razvan]

Hive expects the password in hive-site.xml. We have a policy of not reading product related secrets in the operators. So we put a placeholder in hive-site.xml and use sed to replace it in the container.

When we have implemented this ADR, the mechanism will hopefully be a bit more elegant and transparent. Until then, this is all there is to it.