Druid Operator GCS Deep Storage Support

[ghost]

Hi,

We are looking into implementing Druid using stackable operator.
I saw that you support HDFS and S3 as deep storage.
I was wondering if GCS is also supported and if it's possible to push it as an inline parameter.

Thanks,

[sbernauer]

Hi @pl-avig,

as you correctly recognized we currently don't support GCS natively (yet).

However we do support configOverrides, where you can overwrite the stuff we write out with our operators.
So you can modify the list of extensions we load as well as the configuration settings needed for GCS.

The tricky part is that the GCS extension references a file via GOOGLE_APPLICATION_CREDENTIALS=/path/to/service_account_keyfile.
Just now we merged a PR to support mounting arbitrary stuff (such as files from Secrets), here.
Is used this feature to build the following example, which successfully used GCS as deep storage.

Please note that this feature is only available in our nightly version - but it will be shipped with the next 23.4 release.
First of we need to install the operators stackablectl operator install commons=23.1.0 secret=23.1.0 zookeeper=23.1.0 druid=0.0.0-dev Please note the druid=0.0.0-dev to get the nightly version.

Afterwards the following YAML spins up a Druid using GCS as deep storage.

I hope that helps. Please don't hesitate to ask any questions!

Regards,
Sebastian

---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperCluster
metadata:
  name: simple-zk
spec:
  image:
    productVersion: 3.8.0
    stackableVersion: 23.4.0-rc2
  servers:
    roleGroups:
      default:
        replicas: 1
---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperZnode
metadata:
  name: simple-druid-znode
spec:
  clusterRef:
    name: simple-zk
---
apiVersion: v1
kind: Secret
metadata:
  name: google-service-account
stringData:
  # extracted from ~/.config/gcloud/legacy_credentials/[email protected]/adc.json
  google-service-account.json: |
    {
      "client_id": "XXX.apps.googleusercontent.com",
      "client_secret": "XXX",
      "refresh_token": "XXX",
      "type": "authorized_user"
    }
---
apiVersion: druid.stackable.tech/v1alpha1
kind: DruidCluster
metadata:
  name: simple-druid
spec:
  image:
    productVersion: 24.0.0
    stackableVersion: 23.4.0-rc2
  clusterConfig:
    deepStorage:
      s3:
        bucket:
          inline:
            bucketName: druid
            connection:
              inline:
                host: does-not-exist
                port: 1234
    metadataStorageDatabase:
      dbType: derby
      connString: jdbc:derby://localhost:1527/var/druid/metadata.db;create=true
      host: localhost
      port: 1527
    zookeeperConfigMapName: simple-druid-znode
    extraVolumes:
      - name: google-service-account
        secret:
          secretName: google-service-account
  brokers:
    roleGroups:
      default:
        replicas: 1
        configOverrides: &configOverrides
          runtime.properties:
            druid.extensions.loadList: '["druid-kafka-indexing-service", "druid-datasketches", "prometheus-emitter", "druid-basic-security", "druid-opa-authorizer", "simple-client-sslcontext", "postgresql-metadata-storage", "druid-google-extensions"]'
            druid.storage.type: google
            druid.google.bucket: sbernauer-stackable-druid-gcs
            druid.google.prefix: druid
        envOverrides: &envOverrides
          GOOGLE_APPLICATION_CREDENTIALS: /stackable/userdata/google-service-account/google-service-account.json
  coordinators:
    roleGroups:
      default:
        replicas: 1
        configOverrides: *configOverrides
        envOverrides: *envOverrides
  historicals:
    roleGroups:
      default:
        replicas: 1
        configOverrides: *configOverrides
        envOverrides: *envOverrides
  middleManagers:
    roleGroups:
      default:
        replicas: 1
        configOverrides: *configOverrides
        envOverrides: *envOverrides
  routers:
    roleGroups:
      default:
        replicas: 1
        configOverrides: *configOverrides
        envOverrides: *envOverrides

[sbernauer]

Hi @pl-avig,

did you have a chance of trying out the described solution?

All the best,
Sebastian