[2.4] TXG timestamp DB sync if idle causes unnecessary disk access/prevent spin down

[PapaNappa]

System information

Type	Version/Name
Distribution Name	Arch Linux
Distribution Version	latest
Kernel Version	6.12.63
Architecture	x64
OpenZFS Version	2.4.0

Describe the problem you're observing

Even when the pool/dataset is idle, my hard disks are not spinning down.
They should spin down after being idle for 20 minutes.

Alternatively, when I set hd-idle timeout to <10 minutes, they spin down, but will spin up exactly every 10 minutes.

This happened after upgrading to ZFS 2.4, even before upgrading the pool to the latest ZFS features.

I can make the additional observations:

• Checking with fatrace, I do not see any access to files on the pool
• disk spin down is still not happening when all datasets are unmounted
• exporting the pool will make the HDDs spin down and no longer spin up
• importing the pool readonly will also make the disks spin down

When the pool and all datasets are mounted normally (rw), and I set spa_flush_txg_time to 1 week, the disks are also spinning down as they should.

Thus, it very much seems as it is related to spa_sync_time_logger and the features implemented in #16853.

It seems like spa_sync_time_logger is called as part of spa_sync in the txg_sync_thread every 5s.

It is probably not as simple as a missing else return; after

zfs/module/zfs/spa.c

Lines 2158 to 2165 in 962e688

	if (txg > spa->spa_last_noted_txg) {
	spa->spa_last_noted_txg_time = curtime;
	spa->spa_last_noted_txg = txg;
	mutex_enter(&spa->spa_txg_log_time_lock);
	dbrrd_add(&spa->spa_txg_log_time, curtime, txg);
	mutex_exit(&spa->spa_txg_log_time_lock);
	}

(i.e. if there is no new TXG), but there are not many places where spa_flush_txg_time is used.

Preventing disks to spin down is not an issue in a professional setting. This is a home NAS use-case.
However, this issue also indicates potential unnecessary disk access and might thus be still worth fixing.

Describe how to reproduce the problem

• Configure HDDs to spin down after a certain amount of inactivity.
• Import a pool on the HDDs, but unmount all datasets to ensure there is no other activity on the disks
• Notice that the disks are not spinning down.
  • Alternatively, if HDD idle-timeout is < 10 minutes, disks will spin up and down every 10 minutes.

Include any warning/errors/backtraces from the system logs

I do not see any errors or other log messages related to ZFS or this problem.

[RzTen1]

I think I've also run into this issue. I run my machines on solar + battery power so spinning the disks down is a little more important than for the average system. The array does have a special device on ssd to handle metadata so the spinning disks should only be handling pure data.

Every 10 minutes plus 14-15 seconds I see z_null_iss kick off and write a couple sectors, which resets the spindown counter. I wonder if it would be possible to direct these writes to the special device only?

If it helps I've included a trace from one of the disks that won't spin down below:

      z_null_iss-2144    [004] d..2.  3064.455117:   8,32   A   W 2488 + 8 <- (8,33) 440
      z_null_iss-2144    [004] d..2.  3064.455121:   8,32   Q   W 2488 + 8 [z_null_iss]
      z_null_iss-2144    [004] d..2.  3064.455123:   8,32   G   W 2488 + 8 [z_null_iss]
      z_null_iss-2144    [004] d..2.  3064.455124:   8,32   P   N [z_null_iss]
      z_null_iss-2144    [004] d..2.  3064.455124:   8,32   U   N [z_null_iss] 1
      z_null_iss-2144    [004] d..3.  3064.455126:   8,32   D   W 2488 + 8 [z_null_iss]
      z_null_iss-2144    [004] d..2.  3064.455136:   8,32   A   W 3000 + 8 <- (8,33) 952
      z_null_iss-2144    [004] d..2.  3064.455137:   8,32   Q   W 3000 + 8 [z_null_iss]
      z_null_iss-2144    [004] d..2.  3064.455137:   8,32   G   W 3000 + 8 [z_null_iss]
      z_null_iss-2144    [004] d..2.  3064.455138:   8,32   P   N [z_null_iss]
      z_null_iss-2144    [004] d..2.  3064.455139:   8,32   U   N [z_null_iss] 1
      z_null_iss-2144    [004] d..3.  3064.455140:   8,32   D   W 3000 + 8 [z_null_iss]
      z_null_iss-2144    [004] d..2.  3064.455148:   8,32   A   W 31251740088 + 8 <- (8,33) 31251738040
      z_null_iss-2144    [004] d..2.  3064.455148:   8,32   Q   W 31251740088 + 8 [z_null_iss]
      z_null_iss-2144    [004] d..2.  3064.455149:   8,32   G   W 31251740088 + 8 [z_null_iss]
      z_null_iss-2144    [004] d..2.  3064.455150:   8,32   P   N [z_null_iss]
      z_null_iss-2144    [004] d..2.  3064.455150:   8,32   U   N [z_null_iss] 1
      z_null_iss-2144    [004] d..3.  3064.455151:   8,32   D   W 31251740088 + 8 [z_null_iss]
      z_null_iss-2144    [004] d..2.  3064.455160:   8,32   A   W 31251740600 + 8 <- (8,33) 31251738552
      z_null_iss-2144    [004] d..2.  3064.455161:   8,32   Q   W 31251740600 + 8 [z_null_iss]
      z_null_iss-2144    [004] d..2.  3064.455161:   8,32   G   W 31251740600 + 8 [z_null_iss]
      z_null_iss-2144    [004] d..2.  3064.455162:   8,32   P   N [z_null_iss]
      z_null_iss-2144    [004] d..2.  3064.455163:   8,32   U   N [z_null_iss] 1
      z_null_iss-2144    [004] d..3.  3064.455164:   8,32   D   W 31251740600 + 8 [z_null_iss]
          <idle>-0       [004] dNs3.  3064.455861:   8,32   C   W 2488 + 8 [0]
          <idle>-0       [004] dNs3.  3064.455881:   8,32   C   W 3000 + 8 [0]
          <idle>-0       [004] dNs3.  3064.455885:   8,32   C   W 31251740088 + 8 [0]
          <idle>-0       [004] d.s3.  3064.455897:   8,32   C   W 31251740600 + 8 [0]
        txg_sync-2412    [007] dl.2.  3064.457097:   8,32   Q  FW [txg_sync]
        txg_sync-2412    [007] d..2.  3064.457167:   8,32   G  FW [txg_sync]
           <...>-1455    [007] d..3.  3064.457174:   8,32   D  FN [<...>]
          <idle>-0       [007] d.s3.  3064.469616:   8,32   C  FN 0 [0]
          <idle>-0       [007] d.s4.  3064.469620:   8,32   C  WS 0 [0]

[PapaNappa]

Special VDEVs are a good point.

• If there are no special VDEVs, and no data has been written, the timestamp DB should not be written to disk, and disks should be able to spin down
• If there is a special VDEV, should the timestamp DB be written to it? In particular if the TXG only hits the special VDEV, e.g. because of special_small_blocks. Otherwise, HDDs from the regular VDEVs would spin up just for the timestamp DB.

[BrainOfJ]

I'm experiencing this as well, for my uses (mostly WORM) the timestamp database is not useful and it wakes up my disks every 10 minutes when I'd rather they stay spun down when not in use for extended periods.

My current workaround is to set the spa_note_txg_time module parameter to a high value like 86400 (24h seems like an acceptable mitigation, since there's supposed to be a daily event). Even with spa_flush_txg_time set to the default 600 no write will occur (according to zpool iostat) since, I guess, the database hasn't changed. Unfortunately this obviously affects all pools so it may not always be appropriate.

Frankly this feature seems a bit half-baked, if the pool and its vdev(s) are unchanged, why force continuous recording of anything at all? Maybe the timer should activate on demand and deactivate after doing its job if/when the pool goes idle?