Skip to content

feat: Airflow 3.0.1 (experimental) #630

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 28 commits into from
Jun 3, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
f57633a
wip: working tests (except auth: opa, oidc)
adwk67 May 5, 2025
80041fc
wip: changing python test files
adwk67 May 20, 2025
626d496
single health.py for all airflow versions
razvan May 20, 2025
1c06eb2
single metrics.py for all airflow versions
razvan May 20, 2025
de63b71
update tests with new commons scripts
razvan May 20, 2025
5e195cc
tests: use "airflow-latest" instead of "airflow"
razvan May 20, 2025
999dca0
cleanup and code comments
adwk67 May 20, 2025
502bd19
merge main and resolve conflicts
adwk67 May 21, 2025
83c0394
restore deleted env
adwk67 May 21, 2025
1da6681
use correct webserver service
adwk67 May 21, 2025
b97aaca
restore operator to release list
adwk67 May 21, 2025
31e8c23
test: fix oidc
razvan May 21, 2025
5b772cc
wip: get logging tests to work post-merge
adwk67 May 22, 2025
1e105ee
make env-vars version-specific
adwk67 May 23, 2025
714afb6
fixed resolution of webserver url for execution api
adwk67 May 23, 2025
98cc63a
relaxed default resources
adwk67 May 23, 2025
3585de5
update test defs for oidc/opa
adwk67 May 27, 2025
4f57856
changelog
adwk67 May 27, 2025
536270f
Merge branch 'main' into feat/airflow-3.0.0
adwk67 May 27, 2025
4a2bee3
Update tests/templates/kuttl/logging/51-assert.yaml.j2
adwk67 Jun 2, 2025
2b2f8e9
code review changes
adwk67 Jun 2, 2025
8a0b0b8
Merge branch 'main' into feat/airflow-3.0.0
adwk67 Jun 2, 2025
8d65054
code review changes
adwk67 Jun 2, 2025
f391768
move containerdebug cmd to a function
adwk67 Jun 2, 2025
9dfe57e
replaced random key with hard-coded one with comment
adwk67 Jun 2, 2025
675a70b
cleanup and better comment
adwk67 Jun 2, 2025
bb25ae1
for 3.x: only scheduler updates FAB permissions and restrict workers
adwk67 Jun 3, 2025
8160f37
make api-server env-vars role dependent
adwk67 Jun 3, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
- Use `--file-log-rotation-period` (or `FILE_LOG_ROTATION_PERIOD`) to configure the frequency of rotation.
- Use `--console-log-format` (or `CONSOLE_LOG_FORMAT`) to set the format to `plain` (default) or `json`.
- Add support for airflow `2.10.5` ([#625]).
- Add experimental support for airflow `3.0.1` ([#630]).

### Changed

Expand Down Expand Up @@ -41,6 +42,7 @@
[#623]: https://github.com/stackabletech/airflow-operator/pull/623
[#624]: https://github.com/stackabletech/airflow-operator/pull/624
[#625]: https://github.com/stackabletech/airflow-operator/pull/625
[#630]: https://github.com/stackabletech/airflow-operator/pull/630

## [25.3.0] - 2025-03-21

Expand Down
16 changes: 8 additions & 8 deletions docs/modules/airflow/pages/usage-guide/storage-resources.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ include::home:concepts:stackable_resource_requests.adoc[]

A minimal HA setup consisting of 2 schedulers, 2 workers and 2 webservers has the following https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/[resource requirements]:

* `5700m` CPU request
* `8700m` CPU request
* `17400m` CPU limit
* `10752Mi` memory request and limit
* `15872Mi` memory request and limit

Corresponding to the values above, the operator uses the following resource defaults:

Expand All @@ -18,24 +18,24 @@ spec:
config:
resources:
cpu:
min: 500m
min: "1"
max: "2"
memory:
limit: 512Mi
limit: 1Gi
celeryExecutors:
config:
resources:
cpu:
min: 500m
min: "1"
max: "2"
memory:
limit: 2Gi
limit: 3Gi
webservers:
config:
resources:
cpu:
min: 500m
min: "1"
max: "2"
memory:
limit: 2Gi
limit: 3Gi
----
1 change: 1 addition & 0 deletions docs/modules/airflow/partials/supported-versions.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
// This is a separate file, since it is used by both the direct Airflow-Operator documentation, and the overarching
// Stackable Platform documentation.

- 3.0.1 (experimental)
- 2.10.5
- 2.10.4 (deprecated)
- 2.9.3 (LTS)
10 changes: 8 additions & 2 deletions rust/operator-binary/src/airflow_controller.rs
Original file line number Diff line number Diff line change
Expand Up @@ -957,7 +957,8 @@ fn build_server_rolegroup_statefulset(
.context(GracefulShutdownSnafu)?;

let mut airflow_container_args = Vec::new();
airflow_container_args.extend(airflow_role.get_commands(authentication_config));
airflow_container_args
.extend(airflow_role.get_commands(authentication_config, resolved_product_image));

airflow_container
.image_from_product_image(resolved_product_image)
Expand All @@ -980,6 +981,7 @@ fn build_server_rolegroup_statefulset(
authentication_config,
authorization_config,
git_sync_resources,
resolved_product_image,
)
.context(BuildStatefulsetEnvVarsSnafu)?,
);
Expand Down Expand Up @@ -1170,7 +1172,10 @@ fn build_server_rolegroup_statefulset(
match_labels: Some(statefulset_match_labels.into()),
..LabelSelector::default()
},
service_name: Some(rolegroup_ref.object_name()),
service_name: Some(format!(
"{name}-metrics",
name = rolegroup_ref.object_name()
)),
template: pod_template,
volume_claim_templates: pvcs,
..StatefulSetSpec::default()
Expand Down Expand Up @@ -1263,6 +1268,7 @@ fn build_executor_template_config_map(
env_overrides,
merged_executor_config,
git_sync_resources,
resolved_product_image,
))
.add_volume_mounts(airflow.volume_mounts())
.context(AddVolumeMountSnafu)?
Expand Down
126 changes: 81 additions & 45 deletions rust/operator-binary/src/crd/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ use stackable_operator::{
cache::UserInformationCache,
cluster_operation::ClusterOperation,
opa::OpaConfig,
product_image_selection::ProductImage,
product_image_selection::{ProductImage, ResolvedProductImage},
resources::{
CpuLimitsFragment, MemoryLimitsFragment, NoRuntimeLimits, NoRuntimeLimitsFragment,
Resources, ResourcesFragment,
Expand Down Expand Up @@ -324,11 +324,6 @@ impl v1alpha1::AirflowCluster {
self.spec.cluster_config.volume_mounts.clone()
}

/// The name of the role-level load-balanced Kubernetes `Service`
pub fn node_role_service_name(&self) -> Option<String> {
self.metadata.name.clone()
}

/// Retrieve and merge resource configs for role and role groups
pub fn merged_config(
&self,
Expand Down Expand Up @@ -551,6 +546,7 @@ impl AirflowRole {
pub fn get_commands(
&self,
auth_config: &AirflowClientAuthenticationDetailsResolved,
resolved_product_image: &ResolvedProductImage,
) -> Vec<String> {
let mut command = vec![
format!(
Expand All @@ -561,43 +557,79 @@ impl AirflowRole {
remove_vector_shutdown_file_command(STACKABLE_LOG_DIR),
];

match &self {
AirflowRole::Webserver => {
// Getting auth commands for AuthClass
command.extend(Self::authentication_start_commands(auth_config));
command.extend(vec![
if resolved_product_image.product_version.starts_with("3.") {
// Start-up commands have changed in 3.x.
// See https://airflow.apache.org/docs/apache-airflow/3.0.1/installation/upgrading_to_airflow3.html#step-6-changes-to-your-startup-scripts and
// https://airflow.apache.org/docs/apache-airflow/3.0.1/installation/setting-up-the-database.html#setting-up-the-database.
// `airflow db migrate` is not run for each role so there may be
// re-starts of webserver and/or workers (which require the DB).
// DB-migrations should be eventually be optional:
// See https://github.com/stackabletech/airflow-operator/issues/589.
match &self {
AirflowRole::Webserver => {
command.extend(Self::authentication_start_commands(auth_config));
command.extend(vec![
"prepare_signal_handlers".to_string(),
container_debug_command(),
"airflow api-server &".to_string(),
]);
}
AirflowRole::Scheduler => command.extend(vec![
"airflow db migrate".to_string(),
"airflow users create \
--username \"$ADMIN_USERNAME\" \
--firstname \"$ADMIN_FIRSTNAME\" \
--lastname \"$ADMIN_LASTNAME\" \
--email \"$ADMIN_EMAIL\" \
--password \"$ADMIN_PASSWORD\" \
--role \"Admin\""
.to_string(),
"prepare_signal_handlers".to_string(),
container_debug_command(),
"airflow dag-processor &".to_string(),
"airflow scheduler &".to_string(),
]),
AirflowRole::Worker => command.extend(vec![
"prepare_signal_handlers".to_string(),
format!("containerdebug --output={STACKABLE_LOG_DIR}/containerdebug-state.json --loop &"),
"airflow webserver &".to_string(),
]);
container_debug_command(),
"airflow celery worker &".to_string(),
]),
}
} else {
match &self {
AirflowRole::Webserver => {
// Getting auth commands for AuthClass
command.extend(Self::authentication_start_commands(auth_config));
command.extend(vec![
"prepare_signal_handlers".to_string(),
container_debug_command(),
"airflow webserver &".to_string(),
]);
}
AirflowRole::Scheduler => command.extend(vec![
// Database initialization is limited to the scheduler, see https://github.com/stackabletech/airflow-operator/issues/259
"airflow db init".to_string(),
"airflow db upgrade".to_string(),
"airflow users create \
--username \"$ADMIN_USERNAME\" \
--firstname \"$ADMIN_FIRSTNAME\" \
--lastname \"$ADMIN_LASTNAME\" \
--email \"$ADMIN_EMAIL\" \
--password \"$ADMIN_PASSWORD\" \
--role \"Admin\""
.to_string(),
"prepare_signal_handlers".to_string(),
container_debug_command(),
"airflow scheduler &".to_string(),
]),
AirflowRole::Worker => command.extend(vec![
"prepare_signal_handlers".to_string(),
container_debug_command(),
"airflow celery worker &".to_string(),
]),
}

AirflowRole::Scheduler => command.extend(vec![
// Database initialization is limited to the scheduler, see https://github.com/stackabletech/airflow-operator/issues/259
"airflow db init".to_string(),
"airflow db upgrade".to_string(),
"airflow users create \
--username \"$ADMIN_USERNAME\" \
--firstname \"$ADMIN_FIRSTNAME\" \
--lastname \"$ADMIN_LASTNAME\" \
--email \"$ADMIN_EMAIL\" \
--password \"$ADMIN_PASSWORD\" \
--role \"Admin\""
.to_string(),
"prepare_signal_handlers".to_string(),
format!(
"containerdebug --output={STACKABLE_LOG_DIR}/containerdebug-state.json --loop &"
),
"airflow scheduler &".to_string(),
]),
AirflowRole::Worker => command.extend(vec![
"prepare_signal_handlers".to_string(),
format!(
"containerdebug --output={STACKABLE_LOG_DIR}/containerdebug-state.json --loop &"
),
"airflow celery worker &".to_string(),
]),
}

// graceful shutdown part
command.extend(vec![
"wait_for_termination $!".to_string(),
Expand Down Expand Up @@ -658,6 +690,10 @@ impl AirflowRole {
}
}

fn container_debug_command() -> String {
format!("containerdebug --output={STACKABLE_LOG_DIR}/containerdebug-state.json --loop &")
}

#[derive(Clone, Debug, Deserialize, Display, JsonSchema, PartialEq, Serialize)]
pub enum AirflowExecutor {
/// The celery executor.
Expand Down Expand Up @@ -855,17 +891,17 @@ fn default_resources(role: &AirflowRole) -> ResourcesFragment<AirflowStorageConf
let (cpu, memory) = match role {
AirflowRole::Worker => (
CpuLimitsFragment {
min: Some(Quantity("500m".into())),
min: Some(Quantity("1".into())),
max: Some(Quantity("2".into())),
},
MemoryLimitsFragment {
limit: Some(Quantity("2Gi".into())),
limit: Some(Quantity("3Gi".into())),
runtime_limits: NoRuntimeLimitsFragment {},
},
),
AirflowRole::Webserver => (
CpuLimitsFragment {
min: Some(Quantity("500m".into())),
min: Some(Quantity("1".into())),
max: Some(Quantity("2".into())),
},
MemoryLimitsFragment {
Expand All @@ -875,11 +911,11 @@ fn default_resources(role: &AirflowRole) -> ResourcesFragment<AirflowStorageConf
),
AirflowRole::Scheduler => (
CpuLimitsFragment {
min: Some(Quantity("500m".to_owned())),
min: Some(Quantity("1".to_owned())),
max: Some(Quantity("2".to_owned())),
},
MemoryLimitsFragment {
limit: Some(Quantity("512Mi".to_owned())),
limit: Some(Quantity("1Gi".to_owned())),
runtime_limits: NoRuntimeLimitsFragment {},
},
),
Expand Down
Loading
Loading