How does latency_target works when numjobs > 1?

[hghai14]

For the script below

[read_latency_test]

ioengine=libaio
direct=1
size=100%        
numjobs=2

rw=randread  
bs=4K            
iodepth=256
runtime=300s    
time_based=1

latency_target=1ms
latency_percentile=99.0 
latency_window=5s
latency_run=0 
filename=/dev/nvme0n1

Got following output

read_latency_test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.33
Starting 2 processes
Jobs: 1 (f=1): [r(1),_(1)][18.0%][r=703MiB/s][r=180k IOPS][eta 05m:00s]  
read_latency_test: (groupid=0, jobs=1): err= 0: pid=783279: Mon Oct 27 17:26:04 2025
  read: IOPS=180k, BW=703MiB/s (737MB/s)(3516MiB/5001msec)
    slat (nsec): min=1122, max=74601, avg=1813.01, stdev=1164.68
    clat (usec): min=96, max=963, avg=258.87, stdev=145.65
     lat (usec): min=97, max=965, avg=260.68, stdev=145.64
    clat percentiles (usec):
     |  1.00th=[  124],  5.00th=[  137], 10.00th=[  145], 20.00th=[  155],
     | 30.00th=[  165], 40.00th=[  176], 50.00th=[  188], 60.00th=[  202],
     | 70.00th=[  231], 80.00th=[  453], 90.00th=[  510], 95.00th=[  545],
     | 99.00th=[  603], 99.50th=[  619], 99.90th=[  652], 99.95th=[  668],
     | 99.99th=[  758]
   bw (  KiB/s): min=720104, max=720697, per=66.72%, avg=720394.60, stdev=231.86, samples=10
   iops        : min=180026, max=180176, avg=180098.80, stdev=58.20, samples=10
  lat (usec)   : 100=0.01%, 250=72.44%, 500=15.42%, 750=12.12%, 1000=0.01%
  cpu          : usr=23.56%, sys=35.22%, ctx=39052, majf=0, minf=34
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=900072,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=1000000, window=5000000, percentile=99.00%, depth=47
read_latency_test: (groupid=0, jobs=1): err= 0: pid=783280: Mon Oct 27 17:26:04 2025
  read: IOPS=90.0k, BW=351MiB/s (368MB/s)(1757MiB/5001msec)
    slat (nsec): min=1154, max=81298, avg=2157.46, stdev=1024.27
    clat (usec): min=511, max=4907, avg=2841.77, stdev=349.27
     lat (usec): min=512, max=4908, avg=2843.93, stdev=349.25
    clat percentiles (usec):
     |  1.00th=[ 2057],  5.00th=[ 2147], 10.00th=[ 2212], 20.00th=[ 2671],
     | 30.00th=[ 2802], 40.00th=[ 2900], 50.00th=[ 2966], 60.00th=[ 2999],
     | 70.00th=[ 3032], 80.00th=[ 3064], 90.00th=[ 3130], 95.00th=[ 3195],
     | 99.00th=[ 3556], 99.50th=[ 3720], 99.90th=[ 3982], 99.95th=[ 4047],
     | 99.99th=[ 4178]
   bw (  KiB/s): min=358760, max=360912, per=33.35%, avg=360071.00, stdev=627.39, samples=9
   iops        : min=89690, max=90228, avg=90017.67, stdev=156.88, samples=9
  lat (usec)   : 750=0.01%, 1000=0.02%
  lat (msec)   : 2=0.18%, 4=99.64%, 10=0.08%
  cpu          : usr=12.68%, sys=19.02%, ctx=7875, majf=0, minf=40
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=450100,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=1000000000, window=5000000, percentile=99.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=1054MiB/s (1106MB/s), 351MiB/s-703MiB/s (368MB/s-737MB/s), io=5273MiB (5529MB), run=5001-5001msec

Disk stats (read/write):
  nvme0n1: ios=9177179/0, merge=0/0, ticks=5407021/0, in_queue=5407021, util=99.77%

From the output of second job, we can see the latency target is set to be 1s but I specified it as 1ms in fio job file.
Why the same value is not used for both the jobs here?

Also is there a way in fio in which we can set target latency value for aggregated performance of all jobs combined?

[vincentkfu]

The latency target discrepancy is due to an option parsing problem which has already been fixed. Please update your fio.

Regarding your second question, each fio job maintains its own latency percentile data structure. So the helper thread would have to merge the latency bins across all relevant jobs, check the relevant latency percentile, and then adjust iodepth and communicate the new value to the relevant jobs. I think it would be possible to implement this feature but doing so would not be trivial.