Skip to content

Conversation

bkontur
Copy link
Contributor

@bkontur bkontur commented Sep 10, 2025

TODO: just to see if anything is not working

@bkontur
Copy link
Contributor Author

bkontur commented Sep 11, 2025

@kianenigma @sigurpol I enabled check-migrations, there are just AssetHubs that do not pass,
for example AssetHubKusama: https://github.com/polkadot-fellows/runtimes/actions/runs/17627858405/job/50116968149?pr=904#step:8:10415

[2025-09-11T08:30:17Z ERROR runtime::frame-support] ❌ "Staking" try_state checks failed: Other("BondedEras range incorrect")
[2025-09-11T08:30:17Z ERROR runtime] panicked at /home/runner/work/runtimes/runtimes/system-parachains/asset-hubs/asset-hub-kusama/src/lib.rs:2551:65:
    called `Result::unwrap()` on an `Err` value: Other("Detected errors while executing `try_state` checks. See logs for more info.")

thread 'main' panicked at cli/main.rs:326:10:
called `Result::unwrap()` on an `Err` value: Input("failed to execute TryRuntime_on_runtime_upgrade: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed\nWASM backtrace:\nerror while executing at wasm backtrace:\n    0: 0xc742d0 - asset_hub_kusama_runtime.wasm!__rustc[4794b31dd7191200]::rust_begin_unwind\n    1: 0x58dd - asset_hub_kusama_runtime.wasm!core::panicking::panic_fmt::hd534225921b41838\n    2: 0x631b - asset_hub_kusama_runtime.wasm!core::result::unwrap_failed::he2f060e878986789\n    3: 0xacb427 - asset_hub_kusama_runtime.wasm!TryRuntime_on_runtime_upgrade")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

#workflow_dispatch:
branches: ["main", "release-*"]
pull_request:
workflow_dispatch:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ggwpez Should/could we enable this now for AHM branch? Or maybe just to disable for AssetHubs or something?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should keep the check enabled, given that we want to merge it into master soon.

@sigurpol
Copy link

sigurpol commented Sep 11, 2025

@kianenigma @sigurpol I enabled check-migrations, there are just AssetHubs that do not pass, for example AssetHubKusama: https://github.com/polkadot-fellows/runtimes/actions/runs/17627858405/job/50116968149?pr=904#step:8:10415

[2025-09-11T08:30:17Z ERROR runtime::frame-support] ❌ "Staking" try_state checks failed: Other("BondedEras range incorrect")
[2025-09-11T08:30:17Z ERROR runtime] panicked at /home/runner/work/runtimes/runtimes/system-parachains/asset-hubs/asset-hub-kusama/src/lib.rs:2551:65:
    called `Result::unwrap()` on an `Err` value: Other("Detected errors while executing `try_state` checks. See logs for more info.")

thread 'main' panicked at cli/main.rs:326:10:
called `Result::unwrap()` on an `Err` value: Input("failed to execute TryRuntime_on_runtime_upgrade: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed\nWASM backtrace:\nerror while executing at wasm backtrace:\n    0: 0xc742d0 - asset_hub_kusama_runtime.wasm!__rustc[4794b31dd7191200]::rust_begin_unwind\n    1: 0x58dd - asset_hub_kusama_runtime.wasm!core::panicking::panic_fmt::hd534225921b41838\n    2: 0x631b - asset_hub_kusama_runtime.wasm!core::result::unwrap_failed::he2f060e878986789\n    3: 0xacb427 - asset_hub_kusama_runtime.wasm!TryRuntime_on_runtime_upgrade")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

This is a known issue unfortunately , ideally I would like to do it in the scope of SDK PR9016 - it's not a regression on this branch, it's broken also on master until we remove all deps from old staking pallet from Westend. A horrible temporary hack would be to turn it off via CLI flag or just nuke the staking pallet from the list of pallets in the try-runtime on-runtime-upgrade Sorry - that's a different story (RC side), haven't looked carefully at the logs and I assumed you ended up in the same situation

@bkontur
Copy link
Contributor Author

bkontur commented Sep 11, 2025

it's broken also on master until we remove all deps from old staking pallet from Westend

@sigurpol I don't quite understand why removing pallets from the Westend (do you mean the relay chain) should fix try-runtime for AssetHub?

@sigurpol
Copy link

it's broken also on master until we remove all deps from old staking pallet from Westend

@sigurpol I don't quite understand why removing pallets from the Westend (do you mean the relay chain) should fix try-runtime for AssetHub?

a-ha, I didn't look into the logs in details and I assumed the failure was the same as https://github.com/paritytech/polkadot-sdk/actions/runs/17401852305/job/49396795142 on westend (RC!)
... but it's not your case, here we fail miserably (on AH indeed) because of

 [2025-09-11T08:30:17Z ERROR runtime::frame-support] ❌ "Staking" try_state checks failed: Other("BondedEras range incorrect")

Let me check

@bkontur
Copy link
Contributor Author

bkontur commented Sep 11, 2025

it's broken also on master until we remove all deps from old staking pallet from Westend

@sigurpol I don't quite understand why removing pallets from the Westend (do you mean the relay chain) should fix try-runtime for AssetHub?

a-ha, I didn't look into the logs in details and I assumed the failure was the same as https://github.com/paritytech/polkadot-sdk/actions/runs/17401852305/job/49396795142 on westend (RC!) ... but it's not your case, here we fail miserably (on AH indeed) because of

 [2025-09-11T08:30:17Z ERROR runtime::frame-support] ❌ "Staking" try_state checks failed: Other("BondedEras range incorrect")

Let me check

and AssetHubPolkadot has exactly the same error: https://github.com/polkadot-fellows/runtimes/actions/runs/17627858405/job/50127739849?pr=904

2025-09-11T12:57:17.0165679Z [2025-09-11T12:57:17Z ERROR runtime::frame-support] ❌ "Staking" try_state checks failed: Other("BondedEras range incorrect")

@sigurpol
Copy link

sigurpol commented Sep 11, 2025

it's broken also on master until we remove all deps from old staking pallet from Westend

@sigurpol I don't quite understand why removing pallets from the Westend (do you mean the relay chain) should fix try-runtime for AssetHub?

a-ha, I didn't look into the logs in details and I assumed the failure was the same as https://github.com/paritytech/polkadot-sdk/actions/runs/17401852305/job/49396795142 on westend (RC!) ... but it's not your case, here we fail miserably (on AH indeed) because of

 [2025-09-11T08:30:17Z ERROR runtime::frame-support] ❌ "Staking" try_state checks failed: Other("BondedEras range incorrect")

Let me check

and AssetHubPolkadot has exactly the same error: https://github.com/polkadot-fellows/runtimes/actions/runs/17627858405/job/50127739849?pr=904

2025-09-11T12:57:17.0165679Z [2025-09-11T12:57:17Z ERROR runtime::frame-support] ❌ "Staking" try_state checks failed: Other("BondedEras range incorrect")

The issue is that ah-migrator and rc-migrator have not run so nothing has been migrated from RC to AH which means that we get this

[2025-09-11T14:03:48Z INFO  runtime::staking] [AH] Pre-check staking state: active_era=None, current_era=None, bonded_eras_count=0
[2025-09-11T14:03:48Z INFO  runtime::staking] [AH] BondedEras is EMPTY

which doesn't play well with the check in staking pallet

		// bonded eras must always be the range [active - bonding_duration .. active_era]
		let bonded = BondedEras::<T>::get();
		ensure!(
			bonded.into_iter().map(|(era, _sess)| era).collect::<Vec<_>>() ==
				(active.saturating_sub(T::BondingDuration::get())..=active).collect::<Vec<_>>(),
			"BondedEras range incorrect"

A trivial fix in the staking pallet would be to discard this check in do_try_state() if era is 0 and BondedEras is empty meaning we should assert that BondedEras is empty for era=0.
(cc @kianenigma )

@sigurpol
Copy link

A trivial fix in the staking pallet would be to discard this check in do_try_state() if era is 0 and BondedEras is empty meaning we should assert that BondedEras is empty for era=0. (cc @kianenigma )

A potential fix in the SDK: paritytech/polkadot-sdk#9721

@kianenigma
Copy link
Contributor

Instead of waiting for the fix, we can also do the check-migrations but set env.CHECKS to none for now.

@kianenigma kianenigma closed this Sep 12, 2025
@kianenigma kianenigma reopened this Sep 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants