Skip to content

chore: modularize Dockerfiles #1118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
May 19, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/update-base-java.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ we should also make new versions of Java available for use.

```shell
# See the latest version at https://pypi.org/project/image-tools-stackabletech/
pip install image-tools-stackabletech==0.0.13
pip install image-tools-stackabletech==0.0.16

# Test a product image can build, eg: ZooKeeper
bake --product zookeeper=x.y.z # where x.y.z is a valid product version using the newly added Java version
Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/update-base-stackable.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ Add/Change/Remove anything that isn't applicable anymore

```shell
# See the latest version at https://pypi.org/project/image-tools-stackabletech/
pip install image-tools-stackabletech==0.0.13
pip install image-tools-stackabletech==0.0.16

bake --product vector=x.y.z # where x.y.z is a valid version
```
Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/update-base-vector.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ Add/Change/Remove anything that isn't applicable anymore

```shell
# See the latest version at https://pypi.org/project/image-tools-stackabletech/
pip install image-tools-stackabletech==0.0.13
pip install image-tools-stackabletech==0.0.16

bake --product vector=x.y.z # where x.y.z is the new version added in this PR

Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/update-product-airflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ Add/Change/Remove anything that isn't applicable anymore

```shell
# See the latest version at https://pypi.org/project/image-tools-stackabletech/
pip install image-tools-stackabletech==0.0.13
pip install image-tools-stackabletech==0.0.16

bake --product airflow=x.y.z # where x.y.z is the new version added in this PR

Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/update-product-druid.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ Add/Change/Remove anything that isn't applicable anymore

```shell
# See the latest version at https://pypi.org/project/image-tools-stackabletech/
pip install image-tools-stackabletech==0.0.13
pip install image-tools-stackabletech==0.0.16

bake --product druid=x.y.z # where x.y.z is the new version added in this PR

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ Add/Change/Remove anything that isn't applicable anymore

```shell
# See the latest version at https://pypi.org/project/image-tools-stackabletech/
pip install image-tools-stackabletech==0.0.13
pip install image-tools-stackabletech==0.0.16

bake --product hbase=x.y.z # where x.y.z is the new version added in this PR
bake --product omid=x.y.z # where x.y.z is the new version added in this PR
Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/update-product-hdfs.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ Add/Change/Remove anything that isn't applicable anymore

```shell
# See the latest version at https://pypi.org/project/image-tools-stackabletech/
pip install image-tools-stackabletech==0.0.13
pip install image-tools-stackabletech==0.0.16

bake --product hadoop=x.y.z # where x.y.z is the new version added in this PR

Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/update-product-hive.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ Add/Change/Remove anything that isn't applicable anymore

```shell
# See the latest version at https://pypi.org/project/image-tools-stackabletech/
pip install image-tools-stackabletech==0.0.13
pip install image-tools-stackabletech==0.0.16

bake --product hive=x.y.z # where x.y.z is the new version added in this PR

Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/update-product-kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ Add/Change/Remove anything that isn't applicable anymore

```shell
# See the latest version at https://pypi.org/project/image-tools-stackabletech/
pip install image-tools-stackabletech==0.0.13
pip install image-tools-stackabletech==0.0.16

bake --product kafka=x.y.z # where x.y.z is the new version added in this PR
bake --product kafka-testing-tools=1.0.0 # This version doesn't change
Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/update-product-nifi.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ Add/Change/Remove anything that isn't applicable anymore

```shell
# See the latest version at https://pypi.org/project/image-tools-stackabletech/
pip install image-tools-stackabletech==0.0.13
pip install image-tools-stackabletech==0.0.16

bake --product nifi=x.y.z # where x.y.z is the new version added in this PR

Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/update-product-opa.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ Add/Change/Remove anything that isn't applicable anymore

```shell
# See the latest version at https://pypi.org/project/image-tools-stackabletech/
pip install image-tools-stackabletech==0.0.13
pip install image-tools-stackabletech==0.0.16

bake --product opa=x.y.z # where x.y.z is the new version added in this PR

Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/update-product-spark.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ Add/Change/Remove anything that isn't applicable anymore

```shell
# See the latest version at https://pypi.org/project/image-tools-stackabletech/
pip install image-tools-stackabletech==0.0.13
pip install image-tools-stackabletech==0.0.16

bake --product spark-k8s=x.y.z # where x.y.z is the new version added in this PR

Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/update-product-superset.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Add/Change/Remove anything that isn't applicable anymore

```shell
# See the latest version at https://pypi.org/project/image-tools-stackabletech/
pip install image-tools-stackabletech==0.0.13
pip install image-tools-stackabletech==0.0.16

bake --product superset=x.y.z # where x.y.z is the new version added in this PR

Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/update-product-trino.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ Add/Change/Remove anything that isn't applicable anymore

```shell
# See the latest version at https://pypi.org/project/image-tools-stackabletech/
pip install image-tools-stackabletech==0.0.13
pip install image-tools-stackabletech==0.0.16

bake --product trino=x.y.z # where x.y.z is the new version added in this PR
bake --product trino-cli=x.y.z # where x.y.z is the new version added in this PR
Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/update-product-zookeeper.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ Add/Change/Remove anything that isn't applicable anymore

```shell
# See the latest version at https://pypi.org/project/image-tools-stackabletech/
pip install image-tools-stackabletech==0.0.13
pip install image-tools-stackabletech==0.0.16

bake --product zookeeper=x.y.z # where x.y.z is the new version added in this PR

Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/mirror.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
echo "IMAGE_REPOSITORY=$(.scripts/get_repo_name.sh)" | tee -a "$GITHUB_ENV"

- name: Publish Container Image on oci.stackable.tech
uses: stackabletech/actions/publish-image@55d2f9fcbcd7884ac929ea65fd6f069e7b7a49d2 # 0.8.1
uses: stackabletech/actions/publish-image@9aae2d1c14239021bfa33c041010f6fb7adec815 # 0.8.2
with:
image-registry-uri: oci.stackable.tech
image-registry-username: robot$sdp+github-action-build
Expand All @@ -84,7 +84,7 @@ jobs:
echo "IMAGE_REPOSITORY=$(.scripts/get_repo_name.sh)" | tee -a "$GITHUB_ENV"

- name: Publish and Sign Image Index Manifest to oci.stackable.tech
uses: stackabletech/actions/publish-index-manifest@55d2f9fcbcd7884ac929ea65fd6f069e7b7a49d2 # 0.8.1
uses: stackabletech/actions/publish-index-manifest@9aae2d1c14239021bfa33c041010f6fb7adec815 # 0.8.2
with:
image-registry-uri: oci.stackable.tech
image-registry-username: robot$sdp+github-action-build
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pr_pre-commit.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:
with:
persist-credentials: false
fetch-depth: 0
- uses: stackabletech/actions/run-pre-commit@55d2f9fcbcd7884ac929ea65fd6f069e7b7a49d2 # 0.8.1
- uses: stackabletech/actions/run-pre-commit@9aae2d1c14239021bfa33c041010f6fb7adec815 # 0.8.2
with:
python-version: ${{ env.PYTHON_VERSION }}
hadolint: ${{ env.HADOLINT_VERSION }}
2 changes: 1 addition & 1 deletion .github/workflows/preflight.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ jobs:
- uses: actions/setup-python@f677139bbe7f9c59b41e40162b753c062f5d49a3 # v5.2.0
with:
python-version: '3.x'
- run: pip install image-tools-stackabletech==0.0.13
- run: pip install image-tools-stackabletech==0.0.16
- name: Install preflight
run: |
wget https://github.com/redhat-openshift-ecosystem/openshift-preflight/releases/download/1.10.0/preflight-linux-amd64
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/reusable_build_image.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
with:
persist-credentials: false
- id: shard
uses: stackabletech/actions/shard@55d2f9fcbcd7884ac929ea65fd6f069e7b7a49d2 # 0.8.1
uses: stackabletech/actions/shard@9aae2d1c14239021bfa33c041010f6fb7adec815 # 0.8.2
with:
product-name: ${{ inputs.product-name }}
outputs:
Expand All @@ -53,18 +53,18 @@ jobs:
persist-credentials: false

- name: Free Disk Space
uses: stackabletech/actions/free-disk-space@55d2f9fcbcd7884ac929ea65fd6f069e7b7a49d2 # 0.8.1
uses: stackabletech/actions/free-disk-space@9aae2d1c14239021bfa33c041010f6fb7adec815 # 0.8.2

- name: Build Product Image
id: build
uses: stackabletech/actions/build-product-image@55d2f9fcbcd7884ac929ea65fd6f069e7b7a49d2 # 0.8.1
uses: stackabletech/actions/build-product-image@9aae2d1c14239021bfa33c041010f6fb7adec815 # 0.8.2
with:
product-name: ${{ inputs.product-name }}
product-version: ${{ matrix.versions }}
sdp-version: ${{ inputs.sdp-version }}

- name: Publish Container Image on oci.stackable.tech
uses: stackabletech/actions/publish-image@55d2f9fcbcd7884ac929ea65fd6f069e7b7a49d2 # 0.8.1
uses: stackabletech/actions/publish-image@9aae2d1c14239021bfa33c041010f6fb7adec815 # 0.8.2
with:
image-registry-uri: oci.stackable.tech
image-registry-username: robot$${{ inputs.registry-namespace }}+github-action-build
Expand All @@ -90,7 +90,7 @@ jobs:
persist-credentials: false

- name: Publish and Sign Image Index Manifest to oci.stackable.tech
uses: stackabletech/actions/publish-index-manifest@55d2f9fcbcd7884ac929ea65fd6f069e7b7a49d2 # 0.8.1
uses: stackabletech/actions/publish-index-manifest@9aae2d1c14239021bfa33c041010f6fb7adec815 # 0.8.2
with:
image-registry-uri: oci.stackable.tech
image-registry-username: robot$${{ inputs.registry-namespace }}+github-action-build
Expand Down
6 changes: 3 additions & 3 deletions airflow/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ ARG GIT_SYNC
# which should contain a image location (e.g. registry.k8s.io/git-sync/git-sync:v3.6.8)
FROM oci.stackable.tech/sdp/git-sync/git-sync:${GIT_SYNC} AS gitsync-image

FROM stackable/image/statsd_exporter AS statsd_exporter-builder
FROM stackable/image/shared/statsd-exporter AS statsd_exporter-builder

FROM python:3.12-bookworm AS opa-auth-manager-builder

Expand All @@ -25,7 +25,7 @@ EOF
FROM stackable/image/vector AS airflow-build-image

ARG PRODUCT
ARG STATSD_EXPORTER
ARG SHARED_STATSD_EXPORTER
ARG PYTHON
ARG TARGETARCH
ARG STACKABLE_USER_UID
Expand Down Expand Up @@ -95,7 +95,7 @@ end)' /tmp/sbom.json > /stackable/app/airflow-${PRODUCT}.cdx.json
EOF

COPY --from=statsd_exporter-builder /statsd_exporter/statsd_exporter /stackable/statsd_exporter
COPY --from=statsd_exporter-builder /statsd_exporter/statsd_exporter-${STATSD_EXPORTER}.cdx.json /stackable/statsd_exporter-${STATSD_EXPORTER}.cdx.json
COPY --from=statsd_exporter-builder /statsd_exporter/statsd_exporter-${SHARED_STATSD_EXPORTER}.cdx.json /stackable/statsd_exporter-${SHARED_STATSD_EXPORTER}.cdx.json
COPY --from=gitsync-image --chown=${STACKABLE_USER_UID}:0 /git-sync /stackable/git-sync

RUN <<EOF
Expand Down
6 changes: 3 additions & 3 deletions airflow/versions.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"git_sync": "v4.4.0",
"s3fs": "2024.9.0",
"cyclonedx_bom": "6.0.0",
"statsd_exporter": "0.28.0",
"shared/statsd-exporter": "0.28.0",
"tini": "0.19.0",
"vector": "0.46.1",
"uv": "0.7.3",
Expand All @@ -16,7 +16,7 @@
"git_sync": "v4.4.0",
"s3fs": "2024.9.0",
"cyclonedx_bom": "6.0.0",
"statsd_exporter": "0.28.0",
"shared/statsd-exporter": "0.28.0",
"tini": "0.19.0",
"vector": "0.46.1",
"uv": "0.7.3",
Expand All @@ -27,7 +27,7 @@
"git_sync": "v4.4.0",
"s3fs": "2024.9.0",
"cyclonedx_bom": "6.0.0",
"statsd_exporter": "0.28.0",
"shared/statsd-exporter": "0.28.0",
"tini": "0.19.0",
"vector": "0.46.1",
"uv": "0.7.3",
Expand Down
18 changes: 12 additions & 6 deletions conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@
druid = importlib.import_module("druid.versions")
hadoop = importlib.import_module("hadoop.versions")
hbase = importlib.import_module("hbase.versions")
hbase_phoenix = importlib.import_module("hbase.phoenix.versions")
hbase_opa_authorizer = importlib.import_module("hbase.hbase-opa-authorizer.versions")
hbase_operator_tools = importlib.import_module("hbase.hbase-operator-tools.versions")
hello_world = importlib.import_module("hello-world.versions")
hive = importlib.import_module("hive.versions")
java_base = importlib.import_module("java-base.versions")
Expand All @@ -30,20 +33,23 @@
superset = importlib.import_module("superset.versions")
trino_cli = importlib.import_module("trino-cli.versions")
trino = importlib.import_module("trino.versions")
trino_storage_connector = importlib.import_module("trino-storage-connector.versions")
trino_storage_connector = importlib.import_module("trino.storage-connector.versions")
kafka_testing_tools = importlib.import_module("kafka-testing-tools.versions")
kcat = importlib.import_module("kcat.versions")
kcat = importlib.import_module("kafka.kcat.versions")
testing_tools = importlib.import_module("testing-tools.versions")
zookeeper = importlib.import_module("zookeeper.versions")
tools = importlib.import_module("tools.versions")
statsd_exporter = importlib.import_module("statsd_exporter.versions")
statsd_exporter = importlib.import_module("shared.statsd-exporter.versions")
spark_connect_client = importlib.import_module("spark-connect-client.versions")

products = [
{"name": "airflow", "versions": airflow.versions},
{"name": "druid", "versions": druid.versions},
{"name": "hadoop", "versions": hadoop.versions},
{"name": "hbase", "versions": hbase.versions},
{"name": "hbase/phoenix", "versions": hbase_phoenix.versions},
{"name": "hbase/hbase-opa-authorizer", "versions": hbase_opa_authorizer.versions},
{"name": "hbase/hbase-operator-tools", "versions": hbase_operator_tools.versions},
{"name": "hello-world", "versions": hello_world.versions},
{"name": "hive", "versions": hive.versions},
{"name": "java-base", "versions": java_base.versions},
Expand All @@ -60,13 +66,13 @@
{"name": "superset", "versions": superset.versions},
{"name": "trino-cli", "versions": trino_cli.versions},
{"name": "trino", "versions": trino.versions},
{"name": "trino-storage-connector", "versions": trino_storage_connector.versions},
{"name": "trino/storage-connector", "versions": trino_storage_connector.versions},
{"name": "kafka-testing-tools", "versions": kafka_testing_tools.versions},
{"name": "kcat", "versions": kcat.versions},
{"name": "kafka/kcat", "versions": kcat.versions},
{"name": "testing-tools", "versions": testing_tools.versions},
{"name": "zookeeper", "versions": zookeeper.versions},
{"name": "tools", "versions": tools.versions},
{"name": "statsd_exporter", "versions": statsd_exporter.versions},
{"name": "shared/statsd-exporter", "versions": statsd_exporter.versions},
{"name": "spark-connect-client", "versions": spark_connect_client.versions},
]

Expand Down
Loading