zbd: when performing multi-threaded append writes to a zoned block device, write operations can occasionally fail

[ywang-wnlo]

Please acknowledge the following before creating a ticket

• I have read the GitHub issues section of REPORTING-BUGS.

Description of the bug:
The root cause is suspected to be that the zbd_info->num_write_zones counter is decremented too early at zbd_write_zone_put, before the opend-zone finish operation is fully complete at zbd_finish_zone

zbd.c:1523

	if (zbd_zone_remainder(zb) > 0 && zbd_zone_remainder(zb) < min_bs) {
		pthread_mutex_lock(&f->zbd_info->mutex);
		zbd_write_zone_put(td, f, zb);
		pthread_mutex_unlock(&f->zbd_info->mutex);
		dprint(FD_ZBD, "%s: finish zone %d\n",
		       f->file_name, zbd_zone_idx(f, zb));
		io_u_quiesce(td);
		zbd_finish_zone(td, f, zb);
		zone_unlock(zb);

fio version: branch master

Reproduction steps
maybe this job, bs should be larger than 4k to make sure into branch
zbd_zone_remainder(zb) > 0 && zbd_zone_remainder(zb) < min_bs

[global]
ioengine=XXX
zonemode=zbd
direct=1
thread=1
iodepth=64

[append1]
bs=32k
rw=randwrite        ; random zone
io_size=1%

[append2]
bs=32k
rw=randwrite        ; random zone
io_size=1%
offset=10%

[ywang-wnlo]

Hi @kawasaki , could you please take a look at this issue when you have a moment? I'd appreciate your insight on the zbd module. Thanks!

[kawasaki]

Hi @ywang-wnlo. Your workload specifies io_size=1% and offset=10%. The percentage numbers can often be unaligned with block size or zone size. This tends to leave small write remainders at zone ends which require the zone finish operation. If you replace the percentage with byte numbers aligned to the block size and zone size, you may be able to avoid the failure.

Actually, I faced other use cases which have troubles caused by the zone finish operation. So now I think the zone finish operation should be and can be totally removed from fio. I have just posted a patch series to propose this change. You may want to try out this series to see if it avoids your failure.

[ywang-wnlo]

Hi @kawasaki, I am unsure which commit these patches are based on. However, after reviewing the code, I believe that removing finish_zone should resolve the issue.

After the patches, each zone will now be fully filled, eliminating the need to take the branch I previously provided in zbd_adjust_block which has since been deleted in the patches. Consequently, the write failures caused by a discrepancy between the number of open zones modified prematurely by fio and the number recognized by the underlying hardware will no longer occur.

I look forward to seeing these patches merged into the master branch soon. Best regards.