Unexpected results when testing Write Rand 4K T1Q1 fsync=1 on different io engines

[Kamillaova]

TLDR: When using fsync=1 why is the write io_uring result with lower than that of libaio (top right test in red)?

Tested on NVMe SSD with PLP, fio settings: fio -name=randwrite_fsync -filename=/dev/nvme2n1 -ioengine=[libaio/io_uring] -direct=1 -randrepeat=0 -numa_cpu_nodes=0 -rw=randwrite -bs=4k -numjobs=1 -iodepth=1 -fsync=1 -size=100G -runtime=60 -time_based=1

---

Some observations

Also, interestingly, with the io_uring_cmd engine and -fsync=1 the write speed was 2x slower than read! And with -writefua=1 (instead of -fsync=1) it reached even more IOPS on write than with libaio.

Results

The thing that confuses me is that -ioengine=io_uring WITHOUT -fsync=1 has ~the same performance as -ioengine=libaio WITH -fsync=1, e.g. 136k IOPS, only ~4k less than with -ioengine=io_uring_cmd -writefua=1

[sitsofe]

@Kamillaova:
(Please help those of us with poor reading skills: can you state each expectation + question separately? Also please state your system and fio version as requested in https://github.com/axboe/fio/blob/master/.github/SUPPORT.md#reporting-bugs . Thanks in advance!)

Is your main question "When using fsync=1 why is the io_uring result with lower than that of libaio as seen in the top right most graph"? I don't know the answer to that but could you can gather more data: what is the results like when you use a synchronous ioengine like psync both with and without fsync=1? This should be comparable given the top right graph is using an iodepth of 1...

[Kamillaova]

@sitsofe Linux 6.16, fio 3.40

Is your main question "When using fsync=1 why is the io_uring result with lower than that of libaio as seen in the top right most graph"?

Yes.

---

Test results with psync:

[sitsofe]

@Kamillaova OK your latest run makes the io_uring write at iodepth=1 + fsync=1 run look suspicious and I can't just say its libaio taking shortcuts because the psync ioengine sees similar speeds to libaio. Unfortunately I don't have a disk fast enough to reproduce the problem myself. I've glanced at fsync in the io_uring ioengine code and I don't see any problems...

@axboe do you know if the low io_uring performance versus the libaio/psync ioengines is expected?

[Kamillaova]

Also, I looked at the implementation of the io_submit fsync operation, and it just calls the vfs_fsync function, which essentially calls blkdev_fsync.
AFAIK, the fsync implementation in io_uring is implemented differently in modern kernels.

[sitsofe]

Turns out I CAN reproduce the problem using synthetic devices (6.11 kernel, fio-3.40-50-gb1b0):

sudo modprobe null_blk
sudo ./fio --stonewall --filename /dev/nullb0 --time_based=1 --runtime=5s --direct=1 --rw=write \
  --name=io_uring-no-fsync --ioengine=io_uring \
  --name=io_uring-fsync --ioengine=io_uring --fsync=1 \
  --name=libaio-no-fsync --ioengine=libaio \
  --name=libaio-fsync --ioengine=libaio --fsync=1

Key parts of the output:

[...]
io_uring-no-fsync: (groupid=0, jobs=1): err= 0: pid=100742: Sun Aug 17 08:32:27 2025
  write: IOPS=260k, BW=1015MiB/s (1064MB/s)(5076MiB/5001msec); 0 zone resets
    slat (usec): min=3, max=1196, avg= 3.39, stdev= 1.39
    clat (nsec): min=120, max=52268, avg=139.24, stdev=154.72
     lat (usec): min=3, max=1199, avg= 3.53, stdev= 1.40
[...]
  cpu          : usr=16.80%, sys=83.10%, ctx=71, majf=0, minf=30
[...]
io_uring-fsync: (groupid=1, jobs=1): err= 0: pid=100744: Sun Aug 17 08:32:27 2025
  write: IOPS=32.4k, BW=127MiB/s (133MB/s)(633MiB/5000msec); 0 zone resets
    slat (nsec): min=3446, max=63179, avg=3943.92, stdev=1226.58
    clat (nsec): min=120, max=21320, avg=159.23, stdev=225.35
     lat (nsec): min=3586, max=63550, avg=4103.15, stdev=1254.10
[...]
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=160, max=295746, avg=22728.42, stdev=4365.70
[...]
  cpu          : usr=6.28%, sys=48.86%, ctx=161530, majf=0, minf=32
[...]
libaio-no-fsync: (groupid=2, jobs=1): err= 0: pid=100748: Sun Aug 17 08:32:27 2025
  write: IOPS=218k, BW=853MiB/s (894MB/s)(4263MiB/5000msec); 0 zone resets
    slat (nsec): min=3356, max=114324, avg=3467.20, stdev=786.03
    clat (nsec): min=750, max=93425, avg=791.65, stdev=372.24
     lat (nsec): min=4127, max=115647, avg=4258.85, stdev=873.55
[...]
  cpu          : usr=17.90%, sys=82.02%, ctx=85, majf=0, minf=27
[...]
libaio-fsync: (groupid=3, jobs=1): err= 0: pid=100749: Sun Aug 17 08:32:27 2025
  write: IOPS=175k, BW=684MiB/s (717MB/s)(3418MiB/5001msec); 0 zone resets
    slat (nsec): min=3355, max=127770, avg=3491.13, stdev=862.23
    clat (nsec): min=761, max=60394, avg=798.42, stdev=387.45
     lat (nsec): min=4127, max=128962, avg=4289.55, stdev=954.55
[...]
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=1883, max=121578, avg=1970.20, stdev=638.50
[...]
  cpu          : usr=20.38%, sys=79.56%, ctx=81, majf=0, minf=29
[...]

Key takeaways:

• As the I/O is to the null block device and is going direct there should be nothing in the page cache to flush so the sync timing should be measuring the the time to check that the page cache is empty which is the best case.
• The average fsync latency for libaio is 1970 nsec versus io_uring's 22728 nsec versus (so io_uring is 11 times slower)
• When not doing fsync both libaio and io_uring use up near to 100% of a core. When doing fsync libaio continues using near to 100% of a core but io_uring drops to using 55% of a core.

It would be interesting to have a raw program that just times repeatedly sending fsync using io_uring as that would definitely prove whether this is a kernel issue on a fio issue.

[Kamillaova]

But why is the test with the psync engine and fsync is much faster? Or even with just fsync(2), which is currently used in the libaio engine.
That seems strange to me, since this is a T1Q1 test...

[axboe]

Mostly because neither the libaio nor the io_uring implementation is that great, as they both always need to offload to a different context to perform the operation. It would certainly be possible to improve it, by having them check if anything is required at all and then just return that status inline rather than need a thread punt. But that would also be optimizing for a pretty uninteresting case where indeed no work is required and the fsync wasn't needed in the first place.

If someone is interested in doing MORE work, for the bdev case, you could certainly attempt an inline write-out-and-wait and then have the bdev sync cache be done in an async fashion (eg issue the command, have the completion callback post the CQE). But that would be a lot of work on the kernel side...

[sitsofe]

@Kamillaova would you agree that this isn't a bug in fio? I guess there's an argument that the libaio ioengine should be made to send down a true AIO fsync(!) but it's complicated because you have to still deal with the case where you have an old kernel that lacks a working AIO fsync and do the existing fallback...

[axboe]

I already made that libaio.c engine change earlier. If you're on an old kernel, then don't use fsync with libaio... It's a fairly nonsensical use case anyway, as libaio strictly works with O_DIRECT, and generally you'd use fsync with buffered IO obviously. As far as I can tell, libaio got fsync support in 4.17/18.

You could just have a fallback for it, if we really cared. I'm finding it hard enough to care that deeply, however!

[vitalif]

the libaio engine is not actually using libaio fsync

Wow, I never knew that %) thanks for fixing it!