zpool import on boot takes forever, causes OOM, some sort of memory deadlock

[darkpixel]

System information

Type	Version/Name
Distribution Name	Debian Trixie
Distribution Version	13
Kernel Version	6.12.45+deb13-amd64
Architecture	amd64
OpenZFS Version	2.3.2-2

Describe the problem you're observing

This box has been running for ~5 years now. It's been upgraded over the years from Debian Bullseye on up through Trixie. There are 6x HGST 12 TB drives in a RAIDZ1 which gives around 60 TB of storage. It has 32 GB RAM and pretty much nothing running on it. Just the non-GUI Debian install. It holds backups. A lot of backups. It spends its days automatically SSHing out to various client sites and doing zfs receives on ~10 datasets per client site.

We have a script that loops through each backed up dataset that keeps the latest ~350 snapshots and deletes anything older. This usually keeps us around ~2 TB of free space. It's in my morning procedures to check the backup server and run the script every morning. That 2 TB of free space is more than enough in case I forget to check it for 2-3 days.

The box has been running Trixie for about a month now.

Recently it decided to do a scrub. I think (not certain) that this is the first scrub under Trixie.
The scrub got to ~90% complete over ~5 days...and then I did something stupid.

I got busy, distracted, and then sick for a few days...and I completely forgot to run the script.
The pool ran completely out of space.
When I checked it the scrub was hanged at ~90%, as well as all the automatic backup jobs.

I ran my script from the console, and it slowly (very slowly) started freeing up space in the pool.
The scrub crawled back to life, and the backup jobs un-blocked and failed because they had been hanged for a while.

I watched the pool crawl up to ~980 GB free and the scrub climb to 90.89% via my SSH connection...and then I heard the "beep" of a server rebooting...and my SSH connection dropped.

I waited ~10 minutes and the box still hadn't come back up, so I connected a keyboard and monitor to it and saw it at the 'zpool import' step of the boot process and messages that zpool import was killed by OOM.

I rebooted and set the "init" boot arg to /bin/bash and got to a command line.
zpool import showed my pool.
zpool import tank started churning away.
It ran for a while...maybe 15 minutes (with intermittent messages from the kernel about a hanged task)...and then spit out OOM messages again.

I hit CTRL ALT DEL...and got a spew of messages about memory deadlocks.

I tried again. Same issue.

I can't seem to get the pool imported.

Describe how to reproduce the problem

zpool import tank
wait ~10 minutes

Include any warning/errors/backtraces from the system logs

I can't get my console buffer to scroll back, but I will attach a snap from my cell phone.

[Lady-Galadriel]

I don't have any suggestions other than stopping all other services, (temporarily until you can import the pool). Basically anything that takes up memory and is not critical to the pool import.

The one time I saw this, (on the TrueNAS forums), was when a user had a fullish pool using De-Dup. On import the old De-Dup needs to pull in the De-Dup table into memory. If their is not enough free memory, the import will fail. (As I recall, could be wrong since I don't use De-Dup.)

[darkpixel]

I booted using init=/bin/bash...so the init system didn't come up and no services were started--just dropped me straight to a shell.

No dedup on this pool.

We've been importing this pool for the better part of 5 years (and have on a few occasions run out of space before) and had no trouble recovering things.

Keep in mind, the pool had nearly 1 TB of free space (and was still in the process of freeing more) when the box suddenly rebooted.

[amotin]

I can't say if that is the cause, but I've recently made a couple of changes to throttle async destroys and to delay them a bit on pool import: #18043 , #18033. You may try to set zfs_async_block_max_blocks parameter to some 100000.

If nothing else, you should always be able to import the pool real-only, that does not scrub, does not delete, etc.

[darkpixel]

Testing with that now.
The previous value of my zfs_async_block_max_blocks was set to 18,446,744,073,709,551,615.

[darkpixel]

I'm still running into the memory deadlock part of the problem:

[darkpixel]

I'm starting to getting a bit twitchy after 3 days with no backups.

Should I just destroy and re-create the pool?
I really don't want to re-download all the backup data and lose the history. I think the base set of backups is ~35 TB.

Then again, I guess this is the right time to do it. No one will be at work for the next 10 days...

[mabod]

from: @amotin : If nothing else, you should always be able to import the pool real-only, that does not scrub, does not delete, etc.

So have you tried this:

zpool import -o readonly=on poolname

[darkpixel]

It imports, but zpool status gives a strange "permission denied" when listing "errors":

[mabod]

What is the space situation of the pool now?

[darkpixel]

Over half a TB.
Percentage-wise it's not a lot. :)

[darkpixel]

Weird....I just noticed the output of zpool list -o name,free,freeing....it's basically saying my entire pool is "free"...

[mabod]

Weird....I just noticed the output of zpool list -o name,free,freeing....it's basically saying my entire pool is "free"...

That is really weird. I leave this for the experts. I am out.

[darkpixel]

I'm tempted to roll back to the previous version and run a few tests or just blow away the pool and back everything up again over the holidays (losing ~6 months of history)...but the moment I do that, there's pretty much zero chance to figure out what bug in ZFS made the data inaccessible.

Unfortunately most of the expert volunteers are probably taking some well-deserved vacation this week. :)

[darkpixel]

From that previous read-only import, I decided to blow away the pool.
I did a zpool destroy tank and rm /etc/zfs/zpool.cache and rebooted.

The system booted cleanly (minus tank being imported).
I pulled up top /var/log/syslog and /var/log/messages.
I ran zpool import tank and watched.

I got this in syslog:

2025-12-31T11:41:47.773044-08:00 uswuxsdbkp00 zed: eid=2 class=config_sync pool='tank'                                                                                                                                                                                                                                                                                                                                            2025-12-31T11:41:47.773233-08:00 uswuxsdbkp00 zed: eid=3 class=pool_import pool='tank'  

Followed by a bunch of hung task timeouts and watched as top slowly showed my memory being eaten away until OOM killer kicked in and killed pretty much everything and hanged the box.

I tried downgrading to the ZFS included with Bookworm, but it's too far out of date.