pool operations cause kernel panic, unable to mount or scrub pool #18098
Description
System information
| Type | Version/Name |
|---|---|
| Distribution Name | Ubuntu |
| Distribution Version | 24.04 |
| Kernel Version | 6.14 / 6.8 |
| Architecture | amd64 |
| OpenZFS Version | zfs-2.2.9-0arter97~ubuntu24.04.1 |
| zfs-kmod-2.3.1-1ubuntu2 |
Describe the problem you're observing
System pool, HIWRITE, had some vdevs moved around. It was fine and then last night it would not delete some old snapshots. Received Kernel Panics on the server (nidhoggur). Rebooting, and the system was not usable. Apparently snapshots had been blocked for about a day.
Attempting to mount the problem disks in a separate laptop system results in a kernel panic.
I upgraded my system from 2.2.2 to 2.2.9 via the zfs-lts ppa.
How do I fix the pool and repair it to extract the gapped data?
Describe how to reproduce the problem
attempt to mount or scrub the pool.
Include any warning/errors/backtraces from the system logs
2025-12-31T00:12:20.069771-05:00 nidhoggur kernel: ? vdev_indirect_mapping_entry_for_offset_impl+0x5f/0xd0 [zfs]
2025-12-31T00:12:20.069772-05:00 nidhoggur kernel: spl_assert.constprop.0+0x2a/0x40 [zfs]
2025-12-31T00:12:20.069773-05:00 nidhoggur kernel: vdev_indirect_mark_obsolete+0xe3/0xf0 [zfs]
2025-12-31T00:12:20.069773-05:00 nidhoggur kernel: spa_vdev_indirect_mark_obsolete+0x25/0x40 [zfs]
2025-12-31T00:12:20.069774-05:00 nidhoggur kernel: dsl_scan_obsolete_block_cb+0x5c/0xc0 [zfs]
2025-12-31T00:12:20.069774-05:00 nidhoggur kernel: bpobj_iterate_blkptrs+0x1bb/0x480 [zfs]
2025-12-31T00:12:20.069775-05:00 nidhoggur kernel: ? list_is_empty+0xd/0x20 [zfs]
2025-12-31T00:12:20.069775-05:00 nidhoggur kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T00:12:20.069776-05:00 nidhoggur kernel: bpobj_iterate_impl+0x246/0x3f0 [zfs]
2025-12-31T00:12:20.069777-05:00 nidhoggur kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T00:12:20.069777-05:00 nidhoggur kernel: bpobj_iterate+0x17/0x30 [zfs]
2025-12-31T00:12:20.069778-05:00 nidhoggur kernel: dsl_process_async_destroys+0x2e9/0x5c0 [zfs]
2025-12-31T00:12:20.069779-05:00 nidhoggur kernel: dsl_scan_sync+0x17a/0xa10 [zfs]
2025-12-31T00:12:20.069785-05:00 nidhoggur kernel: ? ddt_sync+0xb6/0xf0 [zfs]
2025-12-31T00:12:20.069786-05:00 nidhoggur kernel: spa_sync_iterate_to_convergence+0x129/0x220 [zfs]
2025-12-31T00:12:20.069786-05:00 nidhoggur kernel: spa_sync+0x321/0x640 [zfs]
2025-12-31T00:12:20.069787-05:00 nidhoggur kernel: txg_sync_thread+0x1f5/0x270 [zfs]
2025-12-31T00:12:20.069787-05:00 nidhoggur kernel: ? __pfx_txg_sync_thread+0x10/0x10 [zfs]
2025-12-31T00:15:35.166369-05:00 nidhoggur kernel: ? vdev_indirect_mapping_entry_for_offset_impl+0x5f/0xd0 [zfs]
2025-12-31T00:15:35.166370-05:00 nidhoggur kernel: spl_assert.constprop.0+0x2a/0x40 [zfs]
2025-12-31T00:15:35.166370-05:00 nidhoggur kernel: vdev_indirect_mark_obsolete+0xe3/0xf0 [zfs]
2025-12-31T00:15:35.167259-05:00 nidhoggur kernel: spa_vdev_indirect_mark_obsolete+0x25/0x40 [zfs]
2025-12-31T00:15:35.167265-05:00 nidhoggur kernel: dsl_scan_obsolete_block_cb+0x5c/0xc0 [zfs]
2025-12-31T00:15:35.167267-05:00 nidhoggur kernel: bpobj_iterate_blkptrs+0x1bb/0x480 [zfs]
2025-12-31T00:15:35.168528-05:00 nidhoggur kernel: ? list_is_empty+0xd/0x20 [zfs]
2025-12-31T00:15:35.168536-05:00 nidhoggur kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T00:15:35.168539-05:00 nidhoggur kernel: bpobj_iterate_impl+0x246/0x3f0 [zfs]
2025-12-31T00:15:35.168540-05:00 nidhoggur kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T00:15:35.169458-05:00 nidhoggur kernel: bpobj_iterate+0x17/0x30 [zfs]
2025-12-31T00:15:35.169466-05:00 nidhoggur kernel: dsl_process_async_destroys+0x2e9/0x5c0 [zfs]
2025-12-31T00:15:35.169468-05:00 nidhoggur kernel: dsl_scan_sync+0x17a/0xa10 [zfs]
2025-12-31T00:15:35.170385-05:00 nidhoggur kernel: ? ddt_sync+0xb6/0xf0 [zfs]
2025-12-31T00:15:35.170397-05:00 nidhoggur kernel: spa_sync_iterate_to_convergence+0x129/0x220 [zfs]
2025-12-31T00:15:35.170399-05:00 nidhoggur kernel: spa_sync+0x321/0x640 [zfs]
2025-12-31T00:15:35.171219-05:00 nidhoggur kernel: txg_sync_thread+0x1f5/0x270 [zfs]
2025-12-31T00:15:35.171227-05:00 nidhoggur kernel: ? __pfx_txg_sync_thread+0x10/0x10 [zfs]
2025-12-31T00:17:38.046305-05:00 nidhoggur kernel: ? vdev_indirect_mapping_entry_for_offset_impl+0x5f/0xd0 [zfs]
2025-12-31T00:17:38.046306-05:00 nidhoggur kernel: spl_assert.constprop.0+0x2a/0x40 [zfs]
2025-12-31T00:17:38.047461-05:00 nidhoggur kernel: vdev_indirect_mark_obsolete+0xe3/0xf0 [zfs]
2025-12-31T00:17:38.047470-05:00 nidhoggur kernel: spa_vdev_indirect_mark_obsolete+0x25/0x40 [zfs]
2025-12-31T00:17:38.047472-05:00 nidhoggur kernel: dsl_scan_obsolete_block_cb+0x5c/0xc0 [zfs]
2025-12-31T00:17:38.048564-05:00 nidhoggur kernel: bpobj_iterate_blkptrs+0x1bb/0x480 [zfs]
2025-12-31T00:17:38.048572-05:00 nidhoggur kernel: ? list_is_empty+0xd/0x20 [zfs]
2025-12-31T00:17:38.048574-05:00 nidhoggur kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T00:17:38.049326-05:00 nidhoggur kernel: bpobj_iterate_impl+0x246/0x3f0 [zfs]
2025-12-31T00:17:38.049336-05:00 nidhoggur kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T00:17:38.050447-05:00 nidhoggur kernel: bpobj_iterate+0x17/0x30 [zfs]
2025-12-31T00:17:38.050460-05:00 nidhoggur kernel: dsl_process_async_destroys+0x2e9/0x5c0 [zfs]
2025-12-31T00:17:38.050463-05:00 nidhoggur kernel: dsl_scan_sync+0x17a/0xa10 [zfs]
2025-12-31T00:17:38.051214-05:00 nidhoggur kernel: ? ddt_sync+0xb6/0xf0 [zfs]
2025-12-31T00:17:38.051220-05:00 nidhoggur kernel: spa_sync_iterate_to_convergence+0x129/0x220 [zfs]
2025-12-31T00:17:38.052411-05:00 nidhoggur kernel: spa_sync+0x321/0x640 [zfs]
2025-12-31T00:17:38.052417-05:00 nidhoggur kernel: txg_sync_thread+0x1f5/0x270 [zfs]
2025-12-31T00:17:38.052419-05:00 nidhoggur kernel: ? __pfx_txg_sync_thread+0x10/0x10 [zfs]
2025-12-31T00:19:40.926423-05:00 nidhoggur kernel: ? vdev_indirect_mapping_entry_for_offset_impl+0x5f/0xd0 [zfs]
2025-12-31T00:19:40.926423-05:00 nidhoggur kernel: spl_assert.constprop.0+0x2a/0x40 [zfs]
2025-12-31T00:19:40.926423-05:00 nidhoggur kernel: vdev_indirect_mark_obsolete+0xe3/0xf0 [zfs]
2025-12-31T00:19:40.926424-05:00 nidhoggur kernel: spa_vdev_indirect_mark_obsolete+0x25/0x40 [zfs]
2025-12-31T00:19:40.926433-05:00 nidhoggur kernel: dsl_scan_obsolete_block_cb+0x5c/0xc0 [zfs]
2025-12-31T00:19:40.927298-05:00 nidhoggur kernel: bpobj_iterate_blkptrs+0x1bb/0x480 [zfs]
2025-12-31T00:19:40.927300-05:00 nidhoggur kernel: ? list_is_empty+0xd/0x20 [zfs]
2025-12-31T00:19:40.927300-05:00 nidhoggur kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T00:19:40.927301-05:00 nidhoggur kernel: bpobj_iterate_impl+0x246/0x3f0 [zfs]
2025-12-31T00:19:40.927301-05:00 nidhoggur kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T00:19:40.927610-05:00 nidhoggur kernel: bpobj_iterate+0x17/0x30 [zfs]
2025-12-31T00:19:40.928439-05:00 nidhoggur kernel: dsl_process_async_destroys+0x2e9/0x5c0 [zfs]
2025-12-31T00:19:40.928443-05:00 nidhoggur kernel: dsl_scan_sync+0x17a/0xa10 [zfs]
2025-12-31T00:19:40.928445-05:00 nidhoggur kernel: ? ddt_sync+0xb6/0xf0 [zfs]
2025-12-31T00:19:40.928446-05:00 nidhoggur kernel: spa_sync_iterate_to_convergence+0x129/0x220 [zfs]
2025-12-31T00:19:40.928447-05:00 nidhoggur kernel: spa_sync+0x321/0x640 [zfs]
2025-12-31T00:19:40.929223-05:00 nidhoggur kernel: txg_sync_thread+0x1f5/0x270 [zfs]
2025-12-31T00:19:40.929228-05:00 nidhoggur kernel: ? __pfx_txg_sync_thread+0x10/0x10 [zfs]
2025-12-31T00:21:43.806333-05:00 nidhoggur kernel: ? vdev_indirect_mapping_entry_for_offset_impl+0x5f/0xd0 [zfs]
2025-12-31T00:21:43.806334-05:00 nidhoggur kernel: spl_assert.constprop.0+0x2a/0x40 [zfs]
2025-12-31T00:21:43.806334-05:00 nidhoggur kernel: vdev_indirect_mark_obsolete+0xe3/0xf0 [zfs]
2025-12-31T00:21:43.806334-05:00 nidhoggur kernel: spa_vdev_indirect_mark_obsolete+0x25/0x40 [zfs]
2025-12-31T00:21:43.806335-05:00 nidhoggur kernel: dsl_scan_obsolete_block_cb+0x5c/0xc0 [zfs]
2025-12-31T00:21:43.807393-05:00 nidhoggur kernel: bpobj_iterate_blkptrs+0x1bb/0x480 [zfs]
2025-12-31T00:21:43.807411-05:00 nidhoggur kernel: ? list_is_empty+0xd/0x20 [zfs]
2025-12-31T00:21:43.807412-05:00 nidhoggur kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T00:21:43.807413-05:00 nidhoggur kernel: bpobj_iterate_impl+0x246/0x3f0 [zfs]
2025-12-31T00:21:43.807415-05:00 nidhoggur kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T00:21:43.807416-05:00 nidhoggur kernel: bpobj_iterate+0x17/0x30 [zfs]
2025-12-31T00:21:43.808415-05:00 nidhoggur kernel: dsl_process_async_destroys+0x2e9/0x5c0 [zfs]
2025-12-31T00:21:43.808420-05:00 nidhoggur kernel: dsl_scan_sync+0x17a/0xa10 [zfs]
2025-12-31T00:21:43.808420-05:00 nidhoggur kernel: ? ddt_sync+0xb6/0xf0 [zfs]
2025-12-31T00:21:43.809237-05:00 nidhoggur kernel: spa_sync_iterate_to_convergence+0x129/0x220 [zfs]
2025-12-31T00:21:43.809241-05:00 nidhoggur kernel: spa_sync+0x321/0x640 [zfs]
2025-12-31T00:21:43.809243-05:00 nidhoggur kernel: txg_sync_thread+0x1f5/0x270 [zfs]
2025-12-31T00:21:43.810245-05:00 nidhoggur kernel: ? __pfx_txg_sync_thread+0x10/0x10 [zfs]
2025-12-31T00:23:46.686397-05:00 nidhoggur kernel: ? vdev_indirect_mapping_entry_for_offset_impl+0x5f/0xd0 [zfs]
2025-12-31T00:23:46.686399-05:00 nidhoggur kernel: spl_assert.constprop.0+0x2a/0x40 [zfs]
2025-12-31T00:23:46.686400-05:00 nidhoggur kernel: vdev_indirect_mark_obsolete+0xe3/0xf0 [zfs]
2025-12-31T00:23:46.686401-05:00 nidhoggur kernel: spa_vdev_indirect_mark_obsolete+0x25/0x40 [zfs]
2025-12-31T00:23:46.686402-05:00 nidhoggur kernel: dsl_scan_obsolete_block_cb+0x5c/0xc0 [zfs]
2025-12-31T00:23:46.687356-05:00 nidhoggur kernel: bpobj_iterate_blkptrs+0x1bb/0x480 [zfs]
2025-12-31T00:23:46.687359-05:00 nidhoggur kernel: ? list_is_empty+0xd/0x20 [zfs]
2025-12-31T00:23:46.687360-05:00 nidhoggur kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T00:23:46.687361-05:00 nidhoggur kernel: bpobj_iterate_impl+0x246/0x3f0 [zfs]
2025-12-31T00:23:46.687362-05:00 nidhoggur kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T00:23:46.688373-05:00 nidhoggur kernel: bpobj_iterate+0x17/0x30 [zfs]
2025-12-31T00:23:46.688377-05:00 nidhoggur kernel: dsl_process_async_destroys+0x2e9/0x5c0 [zfs]
2025-12-31T00:23:46.688378-05:00 nidhoggur kernel: dsl_scan_sync+0x17a/0xa10 [zfs]
2025-12-31T00:23:46.688378-05:00 nidhoggur kernel: ? ddt_sync+0xb6/0xf0 [zfs]
2025-12-31T00:23:46.689287-05:00 nidhoggur kernel: spa_sync_iterate_to_convergence+0x129/0x220 [zfs]
2025-12-31T00:23:46.689290-05:00 nidhoggur kernel: spa_sync+0x321/0x640 [zfs]
2025-12-31T00:23:46.689291-05:00 nidhoggur kernel: txg_sync_thread+0x1f5/0x270 [zfs]
2025-12-31T00:23:46.689292-05:00 nidhoggur kernel: ? __pfx_txg_sync_thread+0x10/0x10 [zfs]
2025-12-31T00:25:49.568458-05:00 nidhoggur kernel: ? vdev_indirect_mapping_entry_for_offset_impl+0x5f/0xd0 [zfs]
2025-12-31T00:25:49.568459-05:00 nidhoggur kernel: spl_assert.constprop.0+0x2a/0x40 [zfs]
2025-12-31T00:25:49.568459-05:00 nidhoggur kernel: vdev_indirect_mark_obsolete+0xe3/0xf0 [zfs]
2025-12-31T00:25:49.568460-05:00 nidhoggur kernel: spa_vdev_indirect_mark_obsolete+0x25/0x40 [zfs]
2025-12-31T00:25:49.568468-05:00 nidhoggur kernel: dsl_scan_obsolete_block_cb+0x5c/0xc0 [zfs]
2025-12-31T00:25:49.568469-05:00 nidhoggur kernel: bpobj_iterate_blkptrs+0x1bb/0x480 [zfs]
2025-12-31T00:25:49.568469-05:00 nidhoggur kernel: ? list_is_empty+0xd/0x20 [zfs]
2025-12-31T00:25:49.568470-05:00 nidhoggur kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T00:25:49.568470-05:00 nidhoggur kernel: bpobj_iterate_impl+0x246/0x3f0 [zfs]
2025-12-31T00:25:49.569426-05:00 nidhoggur kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T00:25:49.569431-05:00 nidhoggur kernel: bpobj_iterate+0x17/0x30 [zfs]
2025-12-31T00:25:49.569432-05:00 nidhoggur kernel: dsl_process_async_destroys+0x2e9/0x5c0 [zfs]
2025-12-31T00:25:49.569433-05:00 nidhoggur kernel: dsl_scan_sync+0x17a/0xa10 [zfs]
2025-12-31T00:25:49.570368-05:00 nidhoggur kernel: ? ddt_sync+0xb6/0xf0 [zfs]
2025-12-31T00:25:49.570373-05:00 nidhoggur kernel: spa_sync_iterate_to_convergence+0x129/0x220 [zfs]
2025-12-31T00:25:49.570374-05:00 nidhoggur kernel: spa_sync+0x321/0x640 [zfs]
2025-12-31T00:25:49.570375-05:00 nidhoggur kernel: txg_sync_thread+0x1f5/0x270 [zfs]
2025-12-31T00:25:49.571253-05:00 nidhoggur kernel: ? __pfx_txg_sync_thread+0x10/0x10 [zfs]
2025-12-31T00:27:52.446339-05:00 nidhoggur kernel: ? vdev_indirect_mapping_entry_for_offset_impl+0x5f/0xd0 [zfs]
2025-12-31T00:27:52.446340-05:00 nidhoggur kernel: spl_assert.constprop.0+0x2a/0x40 [zfs]
2025-12-31T00:27:52.446340-05:00 nidhoggur kernel: vdev_indirect_mark_obsolete+0xe3/0xf0 [zfs]
2025-12-31T00:27:52.446340-05:00 nidhoggur kernel: spa_vdev_indirect_mark_obsolete+0x25/0x40 [zfs]
2025-12-31T00:27:52.446341-05:00 nidhoggur kernel: dsl_scan_obsolete_block_cb+0x5c/0xc0 [zfs]
2025-12-31T00:27:52.447208-05:00 nidhoggur kernel: bpobj_iterate_blkptrs+0x1bb/0x480 [zfs]
2025-12-31T00:27:52.447213-05:00 nidhoggur kernel: ? list_is_empty+0xd/0x20 [zfs]
2025-12-31T00:27:52.447213-05:00 nidhoggur kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T00:27:52.447214-05:00 nidhoggur kernel: bpobj_iterate_impl+0x246/0x3f0 [zfs]
2025-12-31T00:27:52.447214-05:00 nidhoggur kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T00:27:52.448310-05:00 nidhoggur kernel: bpobj_iterate+0x17/0x30 [zfs]
2025-12-31T00:27:52.448314-05:00 nidhoggur kernel: dsl_process_async_destroys+0x2e9/0x5c0 [zfs]
2025-12-31T00:27:52.448314-05:00 nidhoggur kernel: dsl_scan_sync+0x17a/0xa10 [zfs]
2025-12-31T00:27:52.448315-05:00 nidhoggur kernel: ? ddt_sync+0xb6/0xf0 [zfs]
2025-12-31T00:27:52.448315-05:00 nidhoggur kernel: spa_sync_iterate_to_convergence+0x129/0x220 [zfs]
2025-12-31T00:27:52.448316-05:00 nidhoggur kernel: spa_sync+0x321/0x640 [zfs]
2025-12-31T00:27:52.449423-05:00 nidhoggur kernel: txg_sync_thread+0x1f5/0x270 [zfs]
2025-12-31T00:27:52.449429-05:00 nidhoggur kernel: ? __pfx_txg_sync_thread+0x10/0x10 [zfs]
2025-12-31T00:27:52.449434-05:00 nidhoggur kernel: INFO: task zfs:3134083 blocked for more than 122 seconds.
2025-12-31T00:27:52.449443-05:00 nidhoggur kernel: task:zfs state:D stack:0 pid:3134083 tgid:3134083 ppid:1 task_flags:0x480100 flags:0x00004006
2025-12-31T00:27:52.449447-05:00 nidhoggur kernel: txg_wait_synced_impl+0x105/0x150 [zfs]
2025-12-31T00:27:52.449447-05:00 nidhoggur kernel: txg_wait_synced+0x10/0x60 [zfs]
2025-12-31T00:27:52.450418-05:00 nidhoggur kernel: dsl_sync_task_common+0x1dc/0x2a0 [zfs]
2025-12-31T00:27:52.450423-05:00 nidhoggur kernel: ? __pfx_dsl_null_checkfunc+0x10/0x10 [zfs]
2025-12-31T00:27:52.450423-05:00 nidhoggur kernel: ? __pfx_zcp_eval_sync+0x10/0x10 [zfs]
2025-12-31T00:27:52.450423-05:00 nidhoggur kernel: ? __pfx_dsl_null_checkfunc+0x10/0x10 [zfs]
2025-12-31T00:27:52.450424-05:00 nidhoggur kernel: ? __pfx_zcp_eval_sync+0x10/0x10 [zfs]
2025-12-31T00:27:52.451416-05:00 nidhoggur kernel: dsl_sync_task_sig+0x14/0x30 [zfs]
2025-12-31T00:27:52.451421-05:00 nidhoggur kernel: zcp_eval+0x4ca/0x850 [zfs]
2025-12-31T00:27:52.451421-05:00 nidhoggur kernel: dsl_destroy_snapshots_nvl+0x127/0x240 [zfs]
2025-12-31T00:27:52.451421-05:00 nidhoggur kernel: zfs_ioc_destroy_snaps+0x169/0x170 [zfs]
2025-12-31T00:27:52.451422-05:00 nidhoggur kernel: zfsdev_ioctl_common+0x2c8/0x6b0 [zfs]
2025-12-31T00:27:52.452230-05:00 nidhoggur kernel: zfsdev_ioctl+0x57/0xf0 [zfs]
2025-12-31T00:29:55.326436-05:00 nidhoggur kernel: ? vdev_indirect_mapping_entry_for_offset_impl+0x5f/0xd0 [zfs]
2025-12-31T00:29:55.326436-05:00 nidhoggur kernel: spl_assert.constprop.0+0x2a/0x40 [zfs]
2025-12-31T00:29:55.326436-05:00 nidhoggur kernel: vdev_indirect_mark_obsolete+0xe3/0xf0 [zfs]
2025-12-31T00:29:55.326437-05:00 nidhoggur kernel: spa_vdev_indirect_mark_obsolete+0x25/0x40 [zfs]
2025-12-31T00:29:55.326437-05:00 nidhoggur kernel: dsl_scan_obsolete_block_cb+0x5c/0xc0 [zfs]
2025-12-31T00:29:55.327312-05:00 nidhoggur kernel: bpobj_iterate_blkptrs+0x1bb/0x480 [zfs]
2025-12-31T00:29:55.327316-05:00 nidhoggur kernel: ? list_is_empty+0xd/0x20 [zfs]
2025-12-31T00:29:55.327317-05:00 nidhoggur kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T00:29:55.327317-05:00 nidhoggur kernel: bpobj_iterate_impl+0x246/0x3f0 [zfs]
2025-12-31T00:29:55.327318-05:00 nidhoggur kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T00:29:55.328415-05:00 nidhoggur kernel: bpobj_iterate+0x17/0x30 [zfs]
2025-12-31T00:29:55.328420-05:00 nidhoggur kernel: dsl_process_async_destroys+0x2e9/0x5c0 [zfs]
2025-12-31T00:29:55.328420-05:00 nidhoggur kernel: dsl_scan_sync+0x17a/0xa10 [zfs]
2025-12-31T00:29:55.328421-05:00 nidhoggur kernel: ? ddt_sync+0xb6/0xf0 [zfs]
2025-12-31T00:29:55.328421-05:00 nidhoggur kernel: spa_sync_iterate_to_convergence+0x129/0x220 [zfs]
2025-12-31T00:29:55.328422-05:00 nidhoggur kernel: spa_sync+0x321/0x640 [zfs]
2025-12-31T00:29:55.329262-05:00 nidhoggur kernel: txg_sync_thread+0x1f5/0x270 [zfs]
2025-12-31T00:29:55.329266-05:00 nidhoggur kernel: ? __pfx_txg_sync_thread+0x10/0x10 [zfs]
2025-12-31T00:29:55.329278-05:00 nidhoggur kernel: INFO: task zfs:3134083 blocked for more than 245 seconds.
2025-12-31T00:29:55.329279-05:00 nidhoggur kernel: task:zfs state:D stack:0 pid:3134083 tgid:3134083 ppid:1 task_flags:0x480100 flags:0x00004006
2025-12-31T00:29:55.329292-05:00 nidhoggur kernel: txg_wait_synced_impl+0x105/0x150 [zfs]
2025-12-31T00:29:55.330377-05:00 nidhoggur kernel: txg_wait_synced+0x10/0x60 [zfs]
2025-12-31T00:29:55.330381-05:00 nidhoggur kernel: dsl_sync_task_common+0x1dc/0x2a0 [zfs]
2025-12-31T00:29:55.330382-05:00 nidhoggur kernel: ? __pfx_dsl_null_checkfunc+0x10/0x10 [zfs]
2025-12-31T00:29:55.330382-05:00 nidhoggur kernel: ? __pfx_zcp_eval_sync+0x10/0x10 [zfs]
2025-12-31T00:29:55.330383-05:00 nidhoggur kernel: ? __pfx_dsl_null_checkfunc+0x10/0x10 [zfs]
2025-12-31T00:29:55.330383-05:00 nidhoggur kernel: ? __pfx_zcp_eval_sync+0x10/0x10 [zfs]
2025-12-31T00:29:55.331308-05:00 nidhoggur kernel: dsl_sync_task_sig+0x14/0x30 [zfs]
2025-12-31T00:29:55.331313-05:00 nidhoggur kernel: zcp_eval+0x4ca/0x850 [zfs]
2025-12-31T00:29:55.331313-05:00 nidhoggur kernel: dsl_destroy_snapshots_nvl+0x127/0x240 [zfs]
2025-12-31T00:29:55.331314-05:00 nidhoggur kernel: zfs_ioc_destroy_snaps+0x169/0x170 [zfs]
2025-12-31T00:29:55.331314-05:00 nidhoggur kernel: zfsdev_ioctl_common+0x2c8/0x6b0 [zfs]
2025-12-31T00:29:55.332261-05:00 nidhoggur kernel: zfsdev_ioctl+0x57/0xf0 [zfs]
2025-12-31T00:30:01.794715-05:00 nidhoggur CRON[3153683]: (root) CMD (/root/scripts/zfs-HIWRITE-syncoid.sh)
2025-12-31T00:30:02.001688-05:00 nidhoggur sanoid[3153680]: INFO: cache expired - updating from zfs list.
2025-12-31T13:17:17.364989+00:00 ubuntu zed: eid=7 class=resilver_start pool='HIWRITE'
2025-12-31T13:17:16.670687+00:00 ubuntu kernel: message repeated 2 times: [ sdb: sdb1 sdb9]
2025-12-31T13:17:17.369639+00:00 ubuntu kernel: VERIFY(vdev_indirect_mapping_entry_for_offset( vd->vdev_indirect_mapping, offset) != NULL) failed
2025-12-31T13:17:17.369661+00:00 ubuntu kernel: PANIC at vdev_indirect.c:331:vdev_indirect_mark_obsolete()
2025-12-31T13:17:17.369664+00:00 ubuntu kernel: Showing stack for process 10081
2025-12-31T13:17:17.369667+00:00 ubuntu kernel: CPU: 2 UID: 0 PID: 10081 Comm: txg_sync Tainted: P O 6.14.0-27-generic #27~24.04.1-Ubuntu
2025-12-31T13:17:17.369669+00:00 ubuntu kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
2025-12-31T13:17:17.369672+00:00 ubuntu kernel: Hardware name: Enginetech EG320W-G20/X11SCA-F, BIOS 2.5 06/14/2024
2025-12-31T13:17:17.369675+00:00 ubuntu kernel: Call Trace:
2025-12-31T13:17:17.369679+00:00 ubuntu kernel: <TASK>
2025-12-31T13:17:17.369682+00:00 ubuntu kernel: dump_stack_lvl+0x76/0xa0
2025-12-31T13:17:17.369684+00:00 ubuntu kernel: dump_stack+0x10/0x20
2025-12-31T13:17:17.369687+00:00 ubuntu kernel: spl_dumpstack+0x28/0x40 [spl]
2025-12-31T13:17:17.369689+00:00 ubuntu kernel: spl_panic+0xfc/0x120 [spl]
2025-12-31T13:17:17.369691+00:00 ubuntu kernel: ? vdev_indirect_mapping_entry_for_offset_impl+0x5f/0xd0 [zfs]
2025-12-31T13:17:17.369700+00:00 ubuntu kernel: spl_assert.constprop.0+0x2a/0x40 [zfs]
2025-12-31T13:17:17.369703+00:00 ubuntu kernel: vdev_indirect_mark_obsolete+0xe3/0xf0 [zfs]
2025-12-31T13:17:17.369705+00:00 ubuntu kernel: spa_vdev_indirect_mark_obsolete+0x25/0x40 [zfs]
2025-12-31T13:17:17.369708+00:00 ubuntu kernel: dsl_scan_obsolete_block_cb+0x5c/0xc0 [zfs]
2025-12-31T13:17:17.369710+00:00 ubuntu kernel: bpobj_iterate_blkptrs+0x1bb/0x480 [zfs]
2025-12-31T13:17:17.369713+00:00 ubuntu kernel: ? dbuf_dirty+0x675/0x720 [zfs]
2025-12-31T13:17:17.369715+00:00 ubuntu kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T13:17:17.369718+00:00 ubuntu kernel: bpobj_iterate_impl+0x246/0x3f0 [zfs]
2025-12-31T13:17:17.369720+00:00 ubuntu kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T13:17:17.369723+00:00 ubuntu kernel: bpobj_iterate+0x17/0x30 [zfs]
2025-12-31T13:17:17.369725+00:00 ubuntu kernel: dsl_process_async_destroys+0x2e9/0x5c0 [zfs]
2025-12-31T13:17:17.369727+00:00 ubuntu kernel: dsl_scan_sync+0x17a/0xa10 [zfs]
2025-12-31T13:17:17.369729+00:00 ubuntu kernel: spa_sync_iterate_to_convergence+0x129/0x220 [zfs]
2025-12-31T13:17:17.369731+00:00 ubuntu kernel: spa_sync+0x321/0x640 [zfs]
2025-12-31T13:17:17.369734+00:00 ubuntu kernel: txg_sync_thread+0x1f5/0x270 [zfs]
2025-12-31T13:17:17.369736+00:00 ubuntu kernel: ? __pfx_txg_sync_thread+0x10/0x10 [zfs]
2025-12-31T13:17:17.369737+00:00 ubuntu kernel: ? __pfx_thread_generic_wrapper+0x10/0x10 [spl]
2025-12-31T13:17:17.369739+00:00 ubuntu kernel: thread_generic_wrapper+0x5b/0x70 [spl]
2025-12-31T13:17:17.369742+00:00 ubuntu kernel: kthread+0xfb/0x230
2025-12-31T13:17:17.369744+00:00 ubuntu kernel: ? __pfx_kthread+0x10/0x10
2025-12-31T13:17:17.369747+00:00 ubuntu kernel: ret_from_fork+0x44/0x70
2025-12-31T13:17:17.369750+00:00 ubuntu kernel: ? __pfx_kthread+0x10/0x10
2025-12-31T13:17:17.369752+00:00 ubuntu kernel: ret_from_fork_asm+0x1a/0x30
2025-12-31T13:17:17.369754+00:00 ubuntu kernel: </TASK>
Attempt to mount on laptop system (fafnir):
2025-12-31T13:32:28.237334-05:00 fafnir kernel: VERIFY(vdev_indirect_mapping_entry_for_offset( vd->vdev_indirect_mapping, offset) != NULL) failed
2025-12-31T13:32:28.237351-05:00 fafnir kernel: PANIC at vdev_indirect.c:331:vdev_indirect_mark_obsolete()
2025-12-31T13:32:28.237352-05:00 fafnir kernel: Showing stack for process 2025310
2025-12-31T13:32:28.237354-05:00 fafnir kernel: CPU: 17 UID: 0 PID: 2025310 Comm: txg_sync Tainted: P O 6.14.0-37-generic #37~24.04.1-Ubuntu
2025-12-31T13:32:28.237355-05:00 fafnir kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
2025-12-31T13:32:28.237356-05:00 fafnir kernel: Hardware name: ASUSTeK COMPUTER INC. ROG Strix G18 G814JVR_G814JVR/G814JVR, BIOS G814JVR.320 10/24/2024
2025-12-31T13:32:28.237357-05:00 fafnir kernel: Call Trace:
2025-12-31T13:32:28.237358-05:00 fafnir kernel: <TASK>
2025-12-31T13:32:28.237359-05:00 fafnir kernel: dump_stack_lvl+0x76/0xa0
2025-12-31T13:32:28.237359-05:00 fafnir kernel: dump_stack+0x10/0x20
2025-12-31T13:32:28.237360-05:00 fafnir kernel: spl_dumpstack+0x28/0x40 [spl]
2025-12-31T13:32:28.240115-05:00 fafnir kernel: spl_panic+0xfc/0x120 [spl]
2025-12-31T13:32:28.240130-05:00 fafnir kernel: ? vdev_indirect_mapping_entry_for_offset_impl+0x5f/0xd0 [zfs]
2025-12-31T13:32:28.240133-05:00 fafnir kernel: spl_assert.constprop.0+0x2a/0x40 [zfs]
2025-12-31T13:32:28.240134-05:00 fafnir kernel: vdev_indirect_mark_obsolete+0xe3/0xf0 [zfs]
2025-12-31T13:32:28.240146-05:00 fafnir kernel: spa_vdev_indirect_mark_obsolete+0x25/0x40 [zfs]
2025-12-31T13:32:28.240148-05:00 fafnir kernel: dsl_scan_obsolete_block_cb+0x5c/0xc0 [zfs]
2025-12-31T13:32:28.240149-05:00 fafnir kernel: bpobj_iterate_blkptrs+0x1bb/0x480 [zfs]
2025-12-31T13:32:28.240150-05:00 fafnir kernel: ? dbuf_dirty+0x675/0x720 [zfs]
2025-12-31T13:32:28.240151-05:00 fafnir kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T13:32:28.240153-05:00 fafnir kernel: bpobj_iterate_impl+0x246/0x3f0 [zfs]
2025-12-31T13:32:28.240154-05:00 fafnir kernel: ? __pfx_dsl_scan_obsolete_block_cb+0x10/0x10 [zfs]
2025-12-31T13:32:28.240341-05:00 fafnir kernel: ? dmu_tx_commit+0xeb/0x130 [zfs]
2025-12-31T13:32:28.240342-05:00 fafnir kernel: bpobj_iterate+0x17/0x30 [zfs]
2025-12-31T13:32:28.240343-05:00 fafnir kernel: dsl_process_async_destroys+0x2e9/0x5c0 [zfs]
2025-12-31T13:32:28.240343-05:00 fafnir kernel: dsl_scan_sync+0x17a/0xa10 [zfs]
2025-12-31T13:32:28.240346-05:00 fafnir kernel: ? ddt_sync+0xb6/0xf0 [zfs]
2025-12-31T13:32:28.240347-05:00 fafnir kernel: spa_sync_iterate_to_convergence+0x129/0x220 [zfs]
2025-12-31T13:32:28.240347-05:00 fafnir kernel: spa_sync+0x321/0x640 [zfs]
2025-12-31T13:32:28.240348-05:00 fafnir kernel: txg_sync_thread+0x1f5/0x270 [zfs]
2025-12-31T13:32:28.240349-05:00 fafnir kernel: ? __pfx_txg_sync_thread+0x10/0x10 [zfs]
2025-12-31T13:32:28.240349-05:00 fafnir kernel: ? __pfx_thread_generic_wrapper+0x10/0x10 [spl]
2025-12-31T13:32:28.240350-05:00 fafnir kernel: thread_generic_wrapper+0x5b/0x70 [spl]
2025-12-31T13:32:28.240351-05:00 fafnir kernel: kthread+0xfb/0x230
2025-12-31T13:32:28.240351-05:00 fafnir kernel: ? __pfx_kthread+0x10/0x10
2025-12-31T13:32:28.240352-05:00 fafnir kernel: ret_from_fork+0x44/0x70
2025-12-31T13:32:28.240730-05:00 fafnir kernel: ? __pfx_kthread+0x10/0x10
2025-12-31T13:32:28.240739-05:00 fafnir kernel: ret_from_fork_asm+0x1a/0x30
2025-12-31T13:32:28.240740-05:00 fafnir kernel: </TASK>