Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,19 +1,15 @@
---
title: Deploy Apache Flink on Google Cloud C4A (Arm-based Axion VMs)

draft: true
cascade:
draft: true

minutes_to_complete: 30

who_is_this_for: This learning path is intended for software developers deploying and optimizing Apache Flink workloads on Linux/Arm64 environments, specifically using Google Cloud C4A virtual machines powered by Axion processors.
who_is_this_for: This is an introductory topic for developers deploying and optimizing Apache Flink workloads on Linux/Arm64 environments, specifically using Google Cloud C4A virtual machines powered by Axion processors.

learning_objectives:
- Provision an Arm-based SUSE SLES virtual machine on Google Cloud (C4A with Axion processors)
- Install Apache Flink on a SUSE Arm64 (C4A) instance
- Validate Flink functionality by starting the Flink cluster and running a simple baseline job (e.g., WordCount) on the Arm64 VM
- Benchmark Flink performance using internal JMH-based micro-benchmarks on Arm64 (Aarch64) architecture
- Install and configure Apache Flink on an Arm64 instance
- Validate Flink functionality by starting the cluster and running a baseline job
- Benchmark Flink performance using JMH-based microbenchmarks

prerequisites:
- A [Google Cloud Platform (GCP)](https://cloud.google.com/free) account with billing enabled
Expand All @@ -23,7 +19,7 @@ author: Pareena Verma

##### Tags
skilllevels: Introductory
subjects: Databases
subjects: Performance and Architecture
cloud_service_providers: Google Cloud

armips:
Expand All @@ -48,7 +44,7 @@ further_reading:

- resource:
title: Flink documentation
link: https://nightlies.apache.org/flink/flink-docs-lts/
link: https://nightlies.apache.org/flink/flink-docs-lts/
type: documentation

- resource:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
---
title: Getting started with Apache Flink on Google Axion C4A (Arm Neoverse-V2)
title: Get started with Apache Flink on Google Axion C4A

weight: 2

layout: "learningpathall"
---

## Google Axion C4A Arm instances in Google Cloud
## Explore Google Axion C4A Arm instances

Google Axion C4A is a family of Arm-based virtual machines built on Google’s custom Axion CPU, which is based on Arm Neoverse-V2 cores. Designed for high-performance and energy-efficient computing, these virtual machines offer strong performance for modern cloud workloads such as CI/CD pipelines, microservices, media processing, and general-purpose applications.

The C4A series provides a cost-effective alternative to x86 virtual machines while leveraging the scalability and performance benefits of the Arm architecture in Google Cloud.
The C4A series provides a cost-effective alternative to x86 virtual machines while leveraging the scalability and performance benefits of Arm architecture in Google Cloud.

To learn more about Google Axion, refer to the [Introducing Google Axion Processors, our new Arm-based CPUs](https://cloud.google.com/blog/products/compute/introducing-googles-new-arm-based-cpu) blog.
To learn more about Google Axion, see the Google blog [Introducing Google Axion Processors, our new Arm-based CPUs](https://cloud.google.com/blog/products/compute/introducing-googles-new-arm-based-cpu).

## Apache Flink
## Explore Apache Flink

[Apache Flink](https://flink.apache.org/) is an open-source, distributed stream and batch data processing framework** developed under the [Apache Software Foundation](https://www.apache.org/).
[Apache Flink](https://flink.apache.org/) is an open-source, distributed stream and batch data processing framework developed under the [Apache Software Foundation](https://www.apache.org/).

Flink is designed for high-performance, low-latency, and stateful computations on both unbounded (streaming) and bounded (batch) data. It provides a robust runtime and APIs in Java, Scala, and Python for building scalable, fault-tolerant data processing pipelines.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: Apache Flink Baseline Testing on Google Axion C4A Arm Virtual Machine
title: Test Flink baseline functionality
weight: 5

### FIXED, DO NOT MODIFY
Expand All @@ -9,10 +9,10 @@ layout: learningpathall
## Apache Flink Baseline Testing on GCP SUSE VM
In this section you will perform baseline testing for Apache Flink after installation on a GCP SUSE VM. Baseline testing validates that your installation is correct, the JVM is functioning properly, and Flink’s JobManager/TaskManager can execute jobs successfully.

### Install Maven (Required to Build and Run Flink Jobs)
## Install Maven (Required to Build and Run Flink Jobs)
Before running Flink jobs, ensure that Maven is installed on your VM. Many Flink examples and real-world jobs require Apache Maven to compile Java applications.

Download Maven and extract it:
## Install Maven

```console
cd /opt
Expand All @@ -21,21 +21,22 @@ sudo tar -xvzf apache-maven-3.8.6-bin.tar.gz
sudo mv apache-maven-3.8.6 /opt/maven
```

### Configure Environment Variables
## Configure environment variables
Configure the environment so Maven commands can be run system-wide:

```console
echo "export M2_HOME=/opt/maven" >> ~/.bashrc
echo "export PATH=\$M2_HOME/bin:\$PATH" >> ~/.bashrc
source ~/.bashrc
```

Verify the Maven installation:

```console
mvn -version
```

The output should look like:
The output is similar to:

```output
pache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Expand All @@ -45,62 +46,60 @@ Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "5.14.21-150500.55.124-default", arch: "aarch64", family: "unix"
```

At this point, both Java and Maven are installed and ready to use.
## Start the Flink cluster

### Start the Flink Cluster
Before launching Flink, open port 8081 in the Google Cloud Firewall Rules so that the Web UI is reachable externally.

Start the standalone Flink cluster using the provided startup script:

```console
cd $FLINK_HOME
./bin/start-cluster.sh
```

You should see output similar to:
The output is similar to:

```output
Starting cluster.
[INFO] 1 instance(s) of standalonesession are already running on lpprojectsusearm64.
Starting standalonesession daemon on host lpprojectsusearm64.
Starting taskexecutor daemon on host lpprojectsusearm64.
```

Verify that the Flink Processes (JobManager and TaskManager) are running:
Verify that the Flink processes (JobManager and TaskManager) are running:

```console
jps
```

You should see output similar to:
The output is similar to:

```output
21723 StandaloneSessionClusterEntrypoint
2621 Jps
2559 TaskManagerRunner
```
StandaloneSessionClusterEntrypoint is the JobManager process
TaskManagerRunner is the worker responsible for executing tasks and maintaining state.

### Access the Flink Web UI
`StandaloneSessionClusterEntrypoint` is the JobManager process, and `TaskManagerRunner` is the worker responsible for executing tasks and maintaining state.

In a browser, navigate to:
## Access the Flink Web UI

In a browser, navigate to `http://<VM_IP>:8081`.

```console
http://<VM_IP>:8081
```
You should see the Flink Dashboard:
![Flink Dashboard alt-text#center](images/flink-dashboard.png "Figure 1: Flink Dashboard")

![Screenshot of the Apache Flink Dashboard web interface showing the Overview page with cluster status, available task slots, running jobs count, and system metrics displayed in a clean web UI alt-text#center](images/flink-dashboard.png "Flink Dashboard")

A successfully loaded dashboard confirms the cluster network and UI functionality. This serves as the baseline for network and UI validation.

### Run a Simple Example Job
A basic sanity test is to run the built-in WordCount example:
## Run a simple example job
A basic check is to run the built-in WordCount example:

```console
cd $FLINK_HOME
./bin/flink run examples/streaming/WordCount.jar
```
You can monitor the job in the Web UI or check console logs. A successful WordCount run confirms that your Flink cluster lifecycle works end-to-end.

![Flink Dashboard alt-text#center](images/wordcount.png "Figure 2: Word Count Job")
You can monitor the job in the Web UI or check console logs.

![Screenshot of the Flink Dashboard showing a completed WordCount job with execution details, task metrics, and job timeline visible in the web interface alt-text#center](images/wordcount.png "WordCount job in Flink Dashboard")

Flink baseline testing has been completed. You can now proceed to Flink benchmarking.
Flink baseline testing is complete. You can now proceed to Flink benchmarking.
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
---
title: Apache Flink Benchmarking
title: Benchmark Flink performance
weight: 6

### FIXED, DO NOT MODIFY
layout: learningpathall
---

## Benchmark Apache Flink performance

## Apache Flink Benchmarking
This section walks you through running Apache Flink microbenchmarks on a Google Cloud Axion C4A (Arm64) SUSE VM. You will clone the official Flink-benchmarks repository, build the benchmark suite, explore available tests, and run the Remote Channel Throughput Benchmark, one of the key indicators of Flink’s communication and data-transfer performance.
This section walks you through running Apache Flink microbenchmarks on a Google Cloud Axion C4A (Arm64) SUSE VM. You will clone the official Flink-benchmarks repository, build the benchmark suite, and run the Remote Channel Throughput Benchmark.

### Clone the Repository
## Clone the repository
Clone the official Flink microbenchmarks repository:

```console
Expand All @@ -20,39 +20,41 @@ cd flink-benchmarks
```
This repository contains microbenchmarks built using JMH (Java Microbenchmark Harness), widely used for JVM-level performance testing.

### Build the Benchmarks with Maven
## Build the benchmarks with Maven
Compile the benchmarks and create the executable JAR:

```console
mvn clean package -DskipTests
```
What this does:
* clean removes previous build artifacts
* package compiles the code and produces benchmark JARs
* `clean` removes previous build artifacts
* `package` compiles the code and produces benchmark JARs
* -DskipTests speeds up the build since unit tests aren’t needed for benchmarking

After building, the compiled **benchmarks.jar** files appear under:
After building, the compiled benchmarks.jar files appear under:

```output
flink-benchmarks/target/
```

### Explore the JAR Contents
## Explore the JAR contents
Verify that benchmarks.jar was generated:

```console
cd target
ls
```
You should see:

The output is similar to:

```output
benchmark-0.1.jar classes generated-test-sources maven-status protoc-plugins
benchmarks.jar generated-sources maven-archiver protoc-dependencies test-classes
```
benchmarks.jar — Contains all Flink microbenchmarks packaged with JMH.

### List Available Benchmarks
`benchmarks.jar` contains all Flink microbenchmarks packaged with JMH.

## List available benchmarks
View all benchmarks included in the JAR:

```console
Expand All @@ -61,16 +63,18 @@ java -jar benchmarks.jar -l
- `-l` → Lists all benchmarks packaged in the JAR.
- This helps you identify which benchmarks you want to execute on your VM.

### Run Selected Benchmarks
## Run the Remote Channel Throughput benchmark
While the Flink benchmarking project includes multiple suites for state backends, windowing, checkpointing, and scheduler performance, in this Learning path you will run the Remote Channel Throughput benchmark to evaluate network and I/O performance.

**Remote Channel Throughput**: This benchmark measures the data transfer rate between remote channels in Flink, helping to evaluate network and I/O performance.
Remote Channel Throughput: This benchmark measures the data transfer rate between remote channels in Flink, helping to evaluate network and I/O performance.

Run the benchmark:
```console
java -jar benchmarks.jar org.apache.flink.benchmark.RemoteChannelThroughputBenchmark.remoteRebalance
```
You should see output similar to:

The output is similar to:

```output

Result "org.apache.flink.benchmark.RemoteChannelThroughputBenchmark.remoteRebalance":
Expand All @@ -84,26 +88,28 @@ RemoteChannelThroughputBenchmark.remoteRebalance ALIGNED thrpt 30 17445.341
RemoteChannelThroughputBenchmark.remoteRebalance DEBLOAT thrpt 30 10536.511 ± 60.121 ops/ms
```

### Flink Benchmark Metrics Explained
## Understand benchmark metrics

The output from the Remote Channel Throughput benchmark includes several statistical measures that help you interpret Flink's performance on Arm64. Understanding these metrics enables you to assess both the average performance and the consistency of results across multiple iterations.

- **Run Count**: Total benchmark iterations executed, higher count improves reliability.
- **Average Throughput**: Mean operations per second across all iterations.
- **Standard Deviation**: Variation from average throughput, smaller means more consistent.
- **Confidence Interval (99.9%)**: Range where the true average throughput lies with 99.9% certainty.
- **Min Throughput**: The lowest throughput was observed, and it shows worst-case performance.
- **Max Throughput**: Highest throughput observed, shows best-case performance.
- Run Count: total benchmark iterations executed, higher count improves reliability.
- Average Throughput: mean operations per second across all iterations.
- Standard Deviation: variation from average throughput, smaller means more consistent.
- Confidence Interval (99.9%): range where the true average throughput lies with 99.9% certainty.
- Min Throughput: the lowest throughput was observed, and it shows worst-case performance.
- Max Throughput: highest throughput observed, shows best-case performance.

### Benchmark summary on Arm64
## Benchmark summary for Arm64
Results from the run on the `c4a-standard-4` (4 vCPU, 16 GB memory) Arm64 VM in GCP (SUSE) are summarized below:

| Benchmark | Mode | Count | Score (ops/ms) | Error (±) | Min | Max | Stdev | CI (99.9%) | Units |
|---------------------------------------------------|---------|-------|----------------|-----------|-----------|-----------|---------|------------------------|--------|
| RemoteChannelThroughputBenchmark.remoteRebalance | ALIGNED | 30 | 17445.341 | 153.256 | 10289.593 | 10687.736 | 89.987 | [10476.390, 10596.633] | ops/ms |
| RemoteChannelThroughputBenchmark.remoteRebalance | DEBLOAT | 30 | 10536.511 | 60.121 | 10289.593 | 10687.736 | 89.987 | [10476.390, 10596.633] | ops/ms |

### Apache Flink performance benchmarking observations on Arm64
## What you've accomplished and what's next

- Both the ALIGNED mode and DEBLOAT modes demonstrate a strong throughput on the Arm64 VM.
- The benchmark confirms that the Arm64 architecture efficiently handles Flink's remote channel throughput workloads.
You've successfully deployed Apache Flink on a Google Axion C4A Arm-based virtual machine, validated its functionality, and measured its performance using JMH-based microbenchmarks.
The benchmark results confirm that Google Axion C4A Arm-based instances deliver strong throughput in both ALIGNED and DEBLOAT modes, demonstrating that Arm64 architecture efficiently handles Flink's remote channel throughput workloads.

Overall, Arm64 VMs have shown that they are highly suitable for real-time Flink workloads, especially streaming analytics, ETL pipelines, and JVM-based microbenchmarks.
Arm64 VMs are highly suitable for real-time Flink workloads, including streaming analytics, ETL pipelines, and JVM-based microbenchmarks. The consistent performance across different modes demonstrates that Arm-based infrastructure can effectively handle Flink's demanding real-time processing requirements.
Loading