Skip to content

Commit 11f40bb

Browse files
dervoetidervoeti
committed
Merge branch 'main' of https://github.com/stackabletech/docker-images into feat/modularize-dockerfiles
2 parents 3d9502d + b8ff8d3 commit 11f40bb

File tree

7 files changed

+22
-66
lines changed

7 files changed

+22
-66
lines changed

‎CHANGELOG.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ All notable changes to this project will be documented in this file.
2525
`check-permissions-ownership.sh` provided in stackable-base image ([#1027]).
2626
- nifi: Add [nifi-iceberg-bundle] for NiFi `2.4.0` ([#1060], [#1106]).
2727
- nifi: Add `2.4.0` ([#1114]).
28+
- nifi: Add git-sync ([#1107]).
2829
- opa: check for correct permissions and ownerships in /stackable folder via
2930
`check-permissions-ownership.sh` provided in stackable-base image ([#1038]).
3031
- opa: Add `1.4.2` ([#1103]).
@@ -54,6 +55,7 @@ All notable changes to this project will be documented in this file.
5455
- yq: Bump products to use `4.45.2` ([#1090]).
5556
- cyclonedx-bom: Bump airflow and superset to use `6.0.0` ([#1090]).
5657
- vector: Bump to `0.46.1` ([#1098]).
58+
- spark: update dependencies for 3.5.5 ([#1094])
5759

5860
### Fixed
5961

@@ -79,6 +81,7 @@ All notable changes to this project will be documented in this file.
7981
Also remove the old release workflow.
8082
- zookeeper: Remove 3.9.2 ([#1093]).
8183
- Remove ubi8-rust-builder image ([#1091]).
84+
- spark: remove 3.5.2 ([#1094])
8285
- hadoop: Remove `3.3.4` and `3.4.0` ([#1099]).
8386
- opa: Remove `0.67.1` ([#1103]).
8487
- opa: Remove legacy bundle-builder from container build ([#1103]).
@@ -109,13 +112,15 @@ All notable changes to this project will be documented in this file.
109112
[#1090]: https://github.com/stackabletech/docker-images/pull/1090
110113
[#1091]: https://github.com/stackabletech/docker-images/pull/1091
111114
[#1093]: https://github.com/stackabletech/docker-images/pull/1093
115+
[#1094]: https://github.com/stackabletech/docker-images/pull/1094
112116
[#1097]: https://github.com/stackabletech/docker-images/pull/1097
113117
[#1098]: https://github.com/stackabletech/docker-images/pull/1098
114118
[#1099]: https://github.com/stackabletech/docker-images/pull/1099
115119
[#1101]: https://github.com/stackabletech/docker-images/pull/1101
116120
[#1102]: https://github.com/stackabletech/docker-images/pull/1102
117121
[#1103]: https://github.com/stackabletech/docker-images/pull/1103
118122
[#1106]: https://github.com/stackabletech/docker-images/pull/1106
123+
[#1107]: https://github.com/stackabletech/docker-images/pull/1107
119124
[#1108]: https://github.com/stackabletech/docker-images/pull/1108
120125
[#1110]: https://github.com/stackabletech/docker-images/pull/1110
121126
[#1114]: https://github.com/stackabletech/docker-images/pull/1114

‎nifi/Dockerfile

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
# syntax=docker/dockerfile:1.10.0@sha256:865e5dd094beca432e8c0a1d5e1c465db5f998dca4e439981029b3b81fb39ed5
2-
# check=error=true
2+
# check=error=true;skip=InvalidDefaultArgInFrom
3+
4+
ARG GIT_SYNC
5+
6+
FROM oci.stackable.tech/sdp/git-sync/git-sync:${GIT_SYNC} AS git-sync-image
37

48
FROM stackable/image/java-devel AS nifi-builder
59

@@ -32,6 +36,7 @@ WORKDIR /stackable
3236

3337
COPY --chown=${STACKABLE_USER_UID}:0 nifi/stackable/patches/patchable.toml /stackable/src/nifi/stackable/patches/patchable.toml
3438
COPY --chown=${STACKABLE_USER_UID}:0 nifi/stackable/patches/${PRODUCT} /stackable/src/nifi/stackable/patches/${PRODUCT}
39+
COPY --chown=${STACKABLE_USER_UID}:0 --from=git-sync-image /git-sync /stackable/git-sync
3540

3641
RUN <<EOF
3742
# This used to be located in /bin/stackable-bcrypt.jar. We create a softlink for /bin/stackable-bcrypt.jar in the main container for backwards compatibility.
@@ -82,6 +87,7 @@ COPY --chown=${STACKABLE_USER_UID}:0 --from=nifi-builder /stackable/nifi-${PRODU
8287
COPY --chown=${STACKABLE_USER_UID}:0 --from=nifi-builder /stackable/stackable-bcrypt.jar /stackable/stackable-bcrypt.jar
8388
COPY --chown=${STACKABLE_USER_UID}:0 --from=stackable/image/nifi/iceberg-bundle /stackable/*.nar /stackable/nifi-${PRODUCT}/lib/
8489
COPY --chown=${STACKABLE_USER_UID}:0 --from=stackable/image/nifi/iceberg-bundle /stackable/*.sbom.json /stackable/nifi-${PRODUCT}/lib/
90+
COPY --chown=${STACKABLE_USER_UID}:0 --from=nifi-builder /stackable/git-sync /stackable/git-sync
8591

8692
COPY --chown=${STACKABLE_USER_UID}:0 nifi/stackable/bin /stackable/bin
8793
COPY --chown=${STACKABLE_USER_UID}:0 nifi/licenses /licenses
@@ -90,9 +96,11 @@ COPY --chown=${STACKABLE_USER_UID}:0 nifi/python /stackable/python
9096
RUN <<EOF
9197
microdnf update
9298

99+
# git: Required for git-sync
93100
# python-pip: Required to install Python packages
94101
# Note: Python is also required for the Python processors (ExecuteScript and other for NiFi < 2.0 and custom Python processors for NiFi > 2.0)
95102
microdnf install \
103+
git \
96104
python-pip
97105

98106
microdnf clean all

‎nifi/versions.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,16 +3,19 @@
33
"product": "1.27.0",
44
"java-base": "11",
55
"java-devel": "11", # There is an error when trying to use the jdk 21 (since nifi 1.26.0)
6+
"git_sync": "v4.4.0",
67
},
78
{
89
"product": "1.28.1",
910
"java-base": "11",
1011
"java-devel": "11",
12+
"git_sync": "v4.4.0",
1113
},
1214
{
1315
"product": "2.4.0",
1416
"java-base": "21",
1517
"java-devel": "21",
1618
"nifi/iceberg-bundle": "0.0.4",
19+
"git_sync": "v4.4.0",
1720
},
1821
]

‎spark-k8s/Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -155,7 +155,7 @@ WORKDIR /stackable/spark-${PRODUCT}/dist/jars
155155
# Copy modules required for s3a://
156156
COPY --from=hadoop-builder --chown=${STACKABLE_USER_UID}:0 \
157157
/stackable/hadoop/share/hadoop/tools/lib/hadoop-aws-${HADOOP}.jar \
158-
/stackable/hadoop/share/hadoop/tools/lib/aws-java-sdk-bundle-${AWS_JAVA_SDK_BUNDLE}.jar \
158+
/stackable/hadoop/share/hadoop/tools/lib/bundle-${AWS_JAVA_SDK_BUNDLE}.jar \
159159
./
160160

161161
# Copy modules required for abfs://

‎spark-k8s/stackable/patches/3.5.2/0001-Update-CycloneDX-plugin.patch

Lines changed: 0 additions & 40 deletions
This file was deleted.

‎spark-k8s/stackable/patches/3.5.2/patchable.toml

Lines changed: 0 additions & 2 deletions
This file was deleted.

‎spark-k8s/versions.py

Lines changed: 4 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,13 @@
11
versions = [
2-
{
3-
"product": "3.5.2",
4-
"java-base": "17",
5-
"java-devel": "17",
6-
"python": "3.11",
7-
"hadoop": "3.3.6", # Hadoop version defined in ../hbase/versions.py to reduce build time and disk requirements
8-
"hbase": "2.6.1", # current Stackable LTS version
9-
"aws_java_sdk_bundle": "1.12.367", # https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.3.6
10-
"azure_storage": "7.0.1", # https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/3.3.6
11-
"azure_keyvault_core": "1.0.0", # https://mvnrepository.com/artifact/com.microsoft.azure/azure-storage/7.0.1
12-
"jackson_dataformat_xml": "2.15.2", # https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.13/3.5.1
13-
"stax2_api": "4.2.1", # https://mvnrepository.com/artifact/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.15.2
14-
"woodstox_core": "6.5.1", # https://mvnrepository.com/artifact/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.15.2
15-
"vector": "0.46.1",
16-
"jmx_exporter": "1.2.0",
17-
"tini": "0.19.0",
18-
"hbase_connector": "1.0.1",
19-
},
202
{
213
"product": "3.5.5",
224
"java-base": "17",
235
"java-devel": "17",
246
"python": "3.11",
25-
"hadoop": "3.3.6", # Hadoop version defined in ../hbase/versions.py to reduce build time and disk requirements
26-
"hbase": "2.6.1", # current Stackable LTS version
27-
"aws_java_sdk_bundle": "1.12.367", # https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.3.6
28-
"azure_storage": "7.0.1", # https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/3.3.6
7+
"hadoop": "3.4.1", # Current Stackable LTS version. Source of the AWS and Azure artifacts to Spark's classpath.
8+
"hbase": "2.6.2", # Current Stackable LTS version. Used to build the HBase connector.
9+
"aws_java_sdk_bundle": "2.24.6", # https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.4.1
10+
"azure_storage": "7.0.1", # https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/3.4.1
2911
"azure_keyvault_core": "1.0.0", # https://mvnrepository.com/artifact/com.microsoft.azure/azure-storage/7.0.1
3012
"jackson_dataformat_xml": "2.15.2", # https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.13/3.5.2
3113
"stax2_api": "4.2.1", # https://mvnrepository.com/artifact/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.15.2

0 commit comments

Comments
 (0)