Demo Idea: Exemplary GitOps Setup with Stackable

[stackfab]

Stackable data platforms are defined declaratively, enabling automated, standardized, and efficient deployments.

This demo showcases an exemplary GitOps setup for the Stackable data platform using e.g. ArgoCD and distinct Dev/Prod stages. This approach offers consistency, version control, speed, enhanced collaboration, robust security, and observability, making it easier for teams to manage infrastructure as code and adopt Stackable.

For new, non-advanced users, this setup simplifies the learning curve, reduces manual errors, and provides a clear pathway to modern DevOps/GitOps practices.

[stefanigel]

There is already some ArgoCD experience from some projects that can serve as an input.

[dervoeti]

When doing this, I think it would be good to at least showcase how secret management can be done. Like: Don't store plaintext secrets in Git, encrypt them or fetch them from a store like Vault or AWS Secrets Manager. Let ArgoCD or some Kubernetes controller do the decryption/fetching.

[soenkeliebau]

Example of how this might possible look here: https://github.com/stackabletech/demos/tree/spike/argocd-demo

[stefanigel]

Talking to some CSVs and integration partners recently: They often start with some Terraform scripts. Can we consider this (in this issue or as a phase 2)?

[stackfab]

Refined by Malte, Stefan, Fabian:

• no Dev/Prod stages (only 1 cluster)
• without terraform
• including secrets via git (sealed secrets)
• part A:
  - argo-fy/extend existing demo "airflow-scheduled-job"
  - show Argo
  - scale setting via gitops
  - documentation
• part B:
  - integrate existing demo "trino-taxi-data"
  - add a new airflow DAG to submit Trino jobs (create schema, create table,...)
  - deploy new artifacts via dataops from source to target
  - documentation

[stackfab]

Ran across this one. Might me worth checking out:

https://github.com/zapier/kubechecks