Skip to content

Commit 7833850

Browse files
pan3793dongjoon-hyun
pan3793
authored and
dongjoon-hyun
committed
[SPARK-54177][BUILD] Upgrade gRPC to 1.76 and protobuf to 6.33
### What changes were proposed in this pull request? Bump gRPC from 1.67 to 1.76, with additional Python package upgrades for consistency: - `googleapis-common-protos==1.71.0` - `protobuf==6.33.0` And `buf v33.0` Fix the shading leaks of the `spark-connect` jar before ``` $ jar tf spark-connect_2.13-4.1.0-preview3.jar | grep '.class$' | grep -v 'org/apache/spark' | grep -v 'org/sparkproject' | grep -v 'META-INF' javax/annotation/Generated.class ... javax/ejb/EJB.class ... javax/persistence/PersistenceContext.class ... javax/xml/ws/WebServiceRef.class ... com/google/shopping/type/Price$Builder.class ... com/google/apps/card/v1/Widget$DataCase.class ... ``` after ``` $ jar tf spark-connect_2.13-4.2.0-SNAPSHOT.jar | grep '.class$' | grep -v 'org/apache/spark' | grep -v 'org/sparkproject' | grep -v 'META-INF' <no-output> ``` ### Why are the changes needed? For Python: - [grpcio v1.75.1](https://github.com/grpc/grpc/releases/tag/v1.75.1) addes official Python 3.14 support - googleapis-common-proto v1.71.0 addes official Python 3.14 support, see googleapis/google-cloud-python#14699 For Java: - v1.74 removes dependency on Tomcat's annotation API, see grpc/grpc-java#9179 Check full release notes at: https://github.com/grpc/grpc/releases ### Does this PR introduce _any_ user-facing change? Maybe, reduce the potential conflict risks between Spark and user classes. ### How was this patch tested? Pass GHA, plus manual checks (see above sections). ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52874 from pan3793/SPARK-54177. Authored-by: Cheng Pan <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit e55333c) Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent d0680d6 commit 7833850

File tree

43 files changed

+83
-116
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+83
-116
lines changed

‎.github/workflows/build_and_test.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -361,7 +361,7 @@ jobs:
361361
- name: Install Python packages (Python 3.11)
362362
if: (contains(matrix.modules, 'sql') && !contains(matrix.modules, 'sql-')) || contains(matrix.modules, 'connect') || contains(matrix.modules, 'yarn')
363363
run: |
364-
python3.11 -m pip install 'numpy>=1.22' pyarrow pandas pyyaml scipy unittest-xml-reporting 'lxml==4.9.4' 'grpcio==1.67.0' 'grpcio-status==1.67.0' 'protobuf==5.29.5'
364+
python3.11 -m pip install 'numpy>=1.22' pyarrow pandas pyyaml scipy unittest-xml-reporting 'lxml==4.9.4' 'grpcio==1.76.0' 'grpcio-status==1.76.0' 'protobuf==6.33.0'
365365
python3.11 -m pip list
366366
# Run the tests.
367367
- name: Run tests
@@ -765,7 +765,7 @@ jobs:
765765
python-version: '3.11'
766766
- name: Install dependencies for Python CodeGen check
767767
run: |
768-
python3.11 -m pip install 'black==23.12.1' 'protobuf==5.29.5' 'mypy==1.8.0' 'mypy-protobuf==3.3.0'
768+
python3.11 -m pip install 'black==23.12.1' 'protobuf==6.33.0' 'mypy==1.8.0' 'mypy-protobuf==3.3.0'
769769
python3.11 -m pip list
770770
- name: Python CodeGen check for branch-3.5
771771
if: inputs.branch == 'branch-3.5'

‎.github/workflows/build_python_connect.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ jobs:
7272
python packaging/client/setup.py sdist
7373
cd dist
7474
pip install pyspark*client-*.tar.gz
75-
pip install 'grpcio==1.67.0' 'grpcio-status==1.67.0' 'protobuf==5.29.5' 'googleapis-common-protos==1.65.0' 'graphviz==0.20.3' 'six==1.16.0' 'pandas==2.3.3' scipy 'plotly<6.0.0' 'mlflow>=2.8.1' coverage matplotlib openpyxl 'memory-profiler>=0.61.0' 'scikit-learn>=1.3.2' 'graphviz==0.20.3' 'torch<2.6.0' torchvision torcheval deepspeed unittest-xml-reporting
75+
pip install 'grpcio==1.76.0' 'grpcio-status==1.76.0' 'protobuf==6.33.0' 'googleapis-common-protos==1.71.0' 'graphviz==0.20.3' 'six==1.16.0' 'pandas==2.3.3' scipy 'plotly<6.0.0' 'mlflow>=2.8.1' coverage matplotlib openpyxl 'memory-profiler>=0.61.0' 'scikit-learn>=1.3.2' 'graphviz==0.20.3' 'torch<2.6.0' torchvision torcheval deepspeed unittest-xml-reporting
7676
- name: List Python packages
7777
run: python -m pip list
7878
- name: Run tests (local)

‎.github/workflows/maven_test.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -175,7 +175,7 @@ jobs:
175175
- name: Install Python packages (Python 3.11)
176176
if: contains(matrix.modules, 'resource-managers#yarn') || (contains(matrix.modules, 'sql#core')) || contains(matrix.modules, 'connect')
177177
run: |
178-
python3.11 -m pip install 'numpy>=1.22' pyarrow pandas pyyaml scipy unittest-xml-reporting 'grpcio==1.67.0' 'grpcio-status==1.67.0' 'protobuf==5.29.5'
178+
python3.11 -m pip install 'numpy>=1.22' pyarrow pandas pyyaml scipy unittest-xml-reporting 'grpcio==1.76.0' 'grpcio-status==1.76.0' 'protobuf==6.33.0'
179179
python3.11 -m pip list
180180
# Run the tests using script command.
181181
# BSD's script command doesn't support -c option, and the usage is different from Linux's one.

‎.github/workflows/pages.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ jobs:
6363
pip install 'sphinx==4.5.0' mkdocs 'pydata_sphinx_theme>=0.13' sphinx-copybutton nbsphinx numpydoc jinja2 markupsafe 'pyzmq<24.0.0' \
6464
ipython ipython_genutils sphinx_plotly_directive 'numpy>=1.22' pyarrow 'pandas==2.3.3' 'plotly>=4.8' 'docutils<0.18.0' \
6565
'flake8==3.9.0' 'mypy==1.8.0' 'pytest==7.1.3' 'pytest-mypy-plugins==1.9.3' 'black==23.12.1' \
66-
'pandas-stubs==1.2.0.53' 'grpcio==1.67.0' 'grpcio-status==1.67.0' 'protobuf==5.29.5' 'grpc-stubs==1.24.11' 'googleapis-common-protos-stubs==2.2.0' \
66+
'pandas-stubs==1.2.0.53' 'grpcio==1.76.0' 'grpcio-status==1.76.0' 'protobuf==6.33.0' 'grpc-stubs==1.24.11' 'googleapis-common-protos-stubs==2.2.0' \
6767
'sphinxcontrib-applehelp==1.0.4' 'sphinxcontrib-devhelp==1.0.2' 'sphinxcontrib-htmlhelp==2.0.1' 'sphinxcontrib-qthelp==1.0.3' 'sphinxcontrib-serializinghtml==1.1.5'
6868
- name: Install Ruby for documentation generation
6969
uses: ruby/setup-ruby@v1

‎.github/workflows/python_hosted_runner_test.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -148,7 +148,7 @@ jobs:
148148
python${{matrix.python}} -m pip install --ignore-installed 'blinker>=1.6.2'
149149
python${{matrix.python}} -m pip install --ignore-installed 'six==1.16.0'
150150
python${{matrix.python}} -m pip install numpy 'pyarrow>=21.0.0' 'six==1.16.0' 'pandas==2.3.3' scipy 'plotly<6.0.0' 'mlflow>=2.8.1' coverage matplotlib openpyxl 'memory-profiler>=0.61.0' 'scikit-learn>=1.3.2' unittest-xml-reporting && \
151-
python${{matrix.python}} -m pip install 'grpcio==1.67.0' 'grpcio-status==1.67.0' 'protobuf==5.29.5' 'googleapis-common-protos==1.65.0' 'graphviz==0.20.3' && \
151+
python${{matrix.python}} -m pip install 'grpcio==1.76.0' 'grpcio-status==1.76.0' 'protobuf==6.33.0' 'googleapis-common-protos==1.71.0' 'graphviz==0.20.3' && \
152152
python${{matrix.python}} -m pip cache purge
153153
- name: List Python packages
154154
run: python${{matrix.python}} -m pip list

‎dev/create-release/spark-rm/Dockerfile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -94,7 +94,7 @@ ENV R_LIBS_SITE="/usr/local/lib/R/site-library:${R_LIBS_SITE}:/usr/lib/R/library
9494

9595
ARG BASIC_PIP_PKGS="numpy pyarrow>=18.0.0 six==1.16.0 pandas==2.3.3 scipy plotly<6.0.0 mlflow>=2.8.1 coverage matplotlib openpyxl memory-profiler>=0.61.0 scikit-learn>=1.3.2 twine==3.4.1"
9696
# Python deps for Spark Connect
97-
ARG CONNECT_PIP_PKGS="grpcio==1.67.0 grpcio-status==1.67.0 protobuf==5.29.5 googleapis-common-protos==1.65.0 graphviz==0.20.3"
97+
ARG CONNECT_PIP_PKGS="grpcio==1.76.0 grpcio-status==1.76.0 protobuf==6.33.0 googleapis-common-protos==1.71.0 graphviz==0.20.3"
9898

9999
# Install Python 3.10 packages
100100
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
@@ -111,7 +111,7 @@ RUN python3.10 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting $CONNECT_PI
111111
RUN python3.10 -m pip install 'sphinx==4.5.0' mkdocs 'pydata_sphinx_theme>=0.13' sphinx-copybutton nbsphinx numpydoc jinja2 markupsafe 'pyzmq<24.0.0' \
112112
ipython ipython_genutils sphinx_plotly_directive 'numpy>=1.22' pyarrow pandas 'plotly>=4.8' 'docutils<0.18.0' \
113113
'flake8==3.9.0' 'mypy==1.8.0' 'pytest==7.1.3' 'pytest-mypy-plugins==1.9.3' 'black==23.12.1' \
114-
'pandas-stubs==1.2.0.53' 'grpcio==1.67.0' 'grpc-stubs==1.24.11' 'googleapis-common-protos-stubs==2.2.0' \
114+
'pandas-stubs==1.2.0.53' 'grpcio==1.76.0' 'grpc-stubs==1.24.11' 'googleapis-common-protos-stubs==2.2.0' \
115115
'sphinxcontrib-applehelp==1.0.4' 'sphinxcontrib-devhelp==1.0.2' 'sphinxcontrib-htmlhelp==2.0.1' 'sphinxcontrib-qthelp==1.0.3' 'sphinxcontrib-serializinghtml==1.1.5'
116116
RUN python3.10 -m pip list
117117

‎dev/infra/Dockerfile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,7 @@ RUN pypy3 -m pip install numpy 'six==1.16.0' 'pandas==2.3.3' scipy coverage matp
9797

9898
ARG BASIC_PIP_PKGS="numpy pyarrow>=18.0.0 six==1.16.0 pandas==2.3.3 scipy plotly>=4.8 mlflow>=2.8.1 coverage matplotlib openpyxl memory-profiler>=0.61.0 scikit-learn>=1.3.2"
9999
# Python deps for Spark Connect
100-
ARG CONNECT_PIP_PKGS="grpcio==1.67.0 grpcio-status==1.67.0 protobuf==5.29.5 googleapis-common-protos==1.65.0 graphviz==0.20.3"
100+
ARG CONNECT_PIP_PKGS="grpcio==1.76.0 grpcio-status==1.76.0 protobuf==6.33.0 googleapis-common-protos==1.71.0 graphviz==0.20.3"
101101

102102
# Install Python 3.10 packages
103103
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
@@ -149,7 +149,7 @@ RUN apt-get update && apt-get install -y \
149149
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.13
150150
# TODO(SPARK-49862) Add BASIC_PIP_PKGS and CONNECT_PIP_PKGS to Python 3.13 image when it supports Python 3.13
151151
RUN python3.13 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs this
152-
RUN python3.13 -m pip install numpy>=2.1 pyarrow>=18.0.0 six==1.16.0 pandas==2.3.3 scipy coverage matplotlib openpyxl grpcio==1.67.0 grpcio-status==1.67.0 lxml jinja2 && \
152+
RUN python3.13 -m pip install numpy>=2.1 pyarrow>=18.0.0 six==1.16.0 pandas==2.3.3 scipy coverage matplotlib openpyxl grpcio==1.76.0 grpcio-status==1.76.0 lxml jinja2 && \
153153
python3.13 -m pip cache purge
154154

155155
# Remove unused installation packages to free up disk space

‎dev/requirements.txt‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -61,10 +61,10 @@ black==23.12.1
6161
py
6262

6363
# Spark Connect (required)
64-
grpcio>=1.67.0
65-
grpcio-status>=1.67.0
66-
googleapis-common-protos>=1.65.0
67-
protobuf==5.29.5
64+
grpcio>=1.76.0
65+
grpcio-status>=1.76.0
66+
googleapis-common-protos>=1.71.0
67+
protobuf==6.33.0
6868

6969
# Spark Connect python proto generation plugin (optional)
7070
mypy-protobuf==3.3.0

‎dev/spark-test-image/docs/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,6 @@ RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11
9191
RUN python3.11 -m pip install 'sphinx==4.5.0' mkdocs 'pydata_sphinx_theme>=0.13' sphinx-copybutton nbsphinx numpydoc jinja2 markupsafe 'pyzmq<24.0.0' \
9292
ipython ipython_genutils sphinx_plotly_directive 'numpy>=1.22' pyarrow 'pandas==2.3.3' 'plotly>=4.8' 'docutils<0.18.0' \
9393
'flake8==3.9.0' 'mypy==1.8.0' 'pytest==7.1.3' 'pytest-mypy-plugins==1.9.3' 'black==23.12.1' \
94-
'pandas-stubs==1.2.0.53' 'grpcio==1.67.0' 'grpcio-status==1.67.0' 'protobuf==5.29.5' 'grpc-stubs==1.24.11' 'googleapis-common-protos-stubs==2.2.0' \
94+
'pandas-stubs==1.2.0.53' 'grpcio==1.76.0' 'grpcio-status==1.76.0' 'protobuf==6.33.0' 'grpc-stubs==1.24.11' 'googleapis-common-protos-stubs==2.2.0' \
9595
'sphinxcontrib-applehelp==1.0.4' 'sphinxcontrib-devhelp==1.0.2' 'sphinxcontrib-htmlhelp==2.0.1' 'sphinxcontrib-qthelp==1.0.3' 'sphinxcontrib-serializinghtml==1.1.5' \
9696
&& python3.11 -m pip cache purge

‎dev/spark-test-image/lint/Dockerfile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -82,8 +82,8 @@ RUN python3.11 -m pip install \
8282
'flake8==3.9.0' \
8383
'googleapis-common-protos-stubs==2.2.0' \
8484
'grpc-stubs==1.24.11' \
85-
'grpcio-status==1.67.0' \
86-
'grpcio==1.67.0' \
85+
'grpcio-status==1.76.0' \
86+
'grpcio==1.76.0' \
8787
'ipython' \
8888
'ipython_genutils' \
8989
'jinja2' \

0 commit comments

Comments
 (0)