seqread: (g=0): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=psync, iodepth=1 ... fio-3.19 Starting 8 processes seqread: Laying out IO file (1 file / 1024MiB) seqread: Laying out IO file (1 file / 1024MiB) seqread: Laying out IO file (1 file / 1024MiB) seqread: Laying out IO file (1 file / 1024MiB) seqread: Laying out IO file (1 file / 1024MiB) seqread: Laying out IO file (1 file / 1024MiB) seqread: Laying out IO file (1 file / 1024MiB) seqread: Laying out IO file (1 file / 1024MiB) seqread: (groupid=0, jobs=8): err= 0: pid=2344: Fri May 22 19:04:32 2020 read: IOPS=528k, BW=4125MiB/s (4325MB/s)(8192MiB/1986msec) clat (usec): min=3, max=8882, avg=12.35, stdev=61.21 lat (usec): min=3, max=8882, avg=12.51, stdev=61.64 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 13], 95.00th=[ 26], | 99.00th=[ 97], 99.50th=[ 115], 99.90th=[ 165], 99.95th=[ 247], | 99.99th=[ 3458] bw ( MiB/s): min= 4008, max= 4761, per=100.00%, avg=4320.19, stdev=49.03, samples=23 iops : min=513138, max=609515, avg=552980.33, stdev=6276.00, samples=23 lat (usec) : 4=0.65%, 10=71.43%, 20=22.77%, 50=0.45%, 100=3.81% lat (usec) : 250=0.84%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=14.13%, sys=78.23%, ctx=7640, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=4125MiB/s (4325MB/s), 4125MiB/s-4125MiB/s (4325MB/s-4325MB/s), io=8192MiB (8590MB), run=1986-1986msec seqwrite: (g=0): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1 ... fio-3.19 Starting 4 processes seqwrite: Laying out IO file (1 file / 2048MiB) seqwrite: Laying out IO file (1 file / 2048MiB) seqwrite: Laying out IO file (1 file / 2048MiB) seqwrite: Laying out IO file (1 file / 2048MiB) seqwrite: (groupid=0, jobs=4): err= 0: pid=2360: Fri May 22 19:05:38 2020 write: IOPS=16.7k, BW=523MiB/s (549MB/s)(8192MiB/15656msec) clat (usec): min=15, max=1189.0k, avg=232.20, stdev=4565.31 lat (usec): min=16, max=1189.0k, avg=233.12, stdev=4565.31 clat percentiles (usec): | 1.00th=[ 18], 5.00th=[ 20], 10.00th=[ 21], 20.00th=[ 29], | 30.00th=[ 63], 40.00th=[ 80], 50.00th=[ 93], 60.00th=[ 98], | 70.00th=[ 110], 80.00th=[ 124], 90.00th=[ 281], 95.00th=[ 627], | 99.00th=[ 3720], 99.50th=[ 5473], 99.90th=[ 9110], 99.95th=[10814], | 99.99th=[16909] bw ( KiB/s): min=24300, max=3032719, per=99.74%, avg=534392.53, stdev=156082.82, samples=115 iops : min= 756, max=94771, avg=16697.86, stdev=4877.60, samples=115 lat (usec) : 20=8.63%, 50=19.01%, 100=37.31%, 250=24.19%, 500=4.67% lat (usec) : 750=2.00%, 1000=1.00% lat (msec) : 2=1.27%, 4=1.05%, 10=0.81%, 20=0.06%, 50=0.01% lat (msec) : 100=0.01%, 2000=0.01% cpu : usr=2.09%, sys=19.42%, ctx=232205, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=523MiB/s (549MB/s), 523MiB/s-523MiB/s (549MB/s-549MB/s), io=8192MiB (8590MB), run=15656-15656msec randread: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=psync, iodepth=1 ... fio-3.19 Starting 16 processes randread: Laying out IO file (1 file / 1024MiB) randread: Laying out IO file (1 file / 1024MiB) randread: Laying out IO file (1 file / 1024MiB) randread: Laying out IO file (1 file / 1024MiB) randread: Laying out IO file (1 file / 1024MiB) randread: Laying out IO file (1 file / 1024MiB) randread: Laying out IO file (1 file / 1024MiB) randread: Laying out IO file (1 file / 1024MiB) randread: Laying out IO file (1 file / 1024MiB) randread: Laying out IO file (1 file / 1024MiB) randread: Laying out IO file (1 file / 1024MiB) randread: Laying out IO file (1 file / 1024MiB) randread: Laying out IO file (1 file / 1024MiB) randread: Laying out IO file (1 file / 1024MiB) randread: Laying out IO file (1 file / 1024MiB) randread: Laying out IO file (1 file / 1024MiB) randread: (groupid=0, jobs=16): err= 0: pid=2365: Fri May 22 19:08:17 2020 read: IOPS=92.8k, BW=725MiB/s (760MB/s)(16.0GiB/22605msec) clat (usec): min=3, max=114389, avg=155.12, stdev=347.85 lat (usec): min=4, max=114390, avg=155.54, stdev=347.97 clat percentiles (usec): | 1.00th=[ 8], 5.00th=[ 12], 10.00th=[ 15], 20.00th=[ 20], | 30.00th=[ 43], 40.00th=[ 137], 50.00th=[ 186], 60.00th=[ 200], | 70.00th=[ 210], 80.00th=[ 219], 90.00th=[ 231], 95.00th=[ 273], | 99.00th=[ 457], 99.50th=[ 824], 99.90th=[ 2474], 99.95th=[ 3720], | 99.99th=[ 6652] bw ( KiB/s): min=742926, max=1943976, per=100.00%, avg=959115.73, stdev=13180.01, samples=656 iops : min=92859, max=242989, avg=119882.28, stdev=1647.50, samples=656 lat (usec) : 4=0.01%, 10=3.52%, 20=17.01%, 50=9.64%, 100=1.97% lat (usec) : 250=61.70%, 500=5.31%, 750=0.31%, 1000=0.17% lat (msec) : 2=0.24%, 4=0.09%, 10=0.04%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=4.10%, sys=86.78%, ctx=376994, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2097152,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=725MiB/s (760MB/s), 725MiB/s-725MiB/s (760MB/s-760MB/s), io=16.0GiB (17.2GB), run=22605-22605msec randwrite: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 ... fio-3.19 Starting 8 processes randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: Laying out IO file (1 file / 512MiB) randwrite: (groupid=0, jobs=8): err= 0: pid=2382: Fri May 22 19:09:19 2020 write: IOPS=12.9k, BW=807MiB/s (846MB/s)(4096MiB/5074msec) clat (usec): min=22, max=77872, avg=495.62, stdev=1571.12 lat (usec): min=23, max=77875, avg=497.48, stdev=1571.28 clat percentiles (usec): | 1.00th=[ 33], 5.00th=[ 47], 10.00th=[ 71], 20.00th=[ 86], | 30.00th=[ 95], 40.00th=[ 101], 50.00th=[ 111], 60.00th=[ 124], | 70.00th=[ 151], 80.00th=[ 269], 90.00th=[ 1004], 95.00th=[ 2040], | 99.00th=[ 7504], 99.50th=[ 9110], 99.90th=[13829], 99.95th=[16450], | 99.99th=[77071] bw ( MiB/s): min= 250, max= 2880, per=100.00%, avg=1039.99, stdev=132.70, samples=58 iops : min= 4009, max=46079, avg=16636.19, stdev=2123.17, samples=58 lat (usec) : 50=5.70%, 100=33.24%, 250=40.71%, 500=3.39%, 750=3.66% lat (usec) : 1000=3.25% lat (msec) : 2=4.99%, 4=2.00%, 10=2.70%, 20=0.34%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.38%, sys=27.54%, ctx=58692, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=807MiB/s (846MB/s), 807MiB/s-807MiB/s (846MB/s-846MB/s), io=4096MiB (4295MB), run=5074-5074msec randrw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 ... fio-3.19 Starting 8 processes randrw: Laying out IO file (1 file / 1024MiB) randrw: Laying out IO file (1 file / 1024MiB) randrw: Laying out IO file (1 file / 1024MiB) randrw: Laying out IO file (1 file / 1024MiB) randrw: Laying out IO file (1 file / 1024MiB) randrw: Laying out IO file (1 file / 1024MiB) randrw: Laying out IO file (1 file / 1024MiB) randrw: Laying out IO file (1 file / 1024MiB) randrw: (groupid=0, jobs=8): err= 0: pid=2393: Fri May 22 19:10:53 2020 read: IOPS=48.3k, BW=755MiB/s (792MB/s)(7370MiB/9763msec) clat (usec): min=5, max=11084, avg=32.93, stdev=94.38 lat (usec): min=5, max=11084, avg=33.17, stdev=95.17 clat percentiles (usec): | 1.00th=[ 9], 5.00th=[ 10], 10.00th=[ 10], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 15], 60.00th=[ 17], | 70.00th=[ 21], 80.00th=[ 55], 90.00th=[ 87], 95.00th=[ 104], | 99.00th=[ 153], 99.50th=[ 184], 99.90th=[ 709], 99.95th=[ 1500], | 99.99th=[ 4948] bw ( MiB/s): min= 1315, max= 3390, per=100.00%, avg=1878.05, stdev=89.61, samples=99 iops : min=84206, max=216985, avg=120191.42, stdev=5735.03, samples=99 write: IOPS=5388, BW=84.2MiB/s (88.3MB/s)(822MiB/9763msec) clat (usec): min=18, max=1529.7k, avg=793.14, stdev=16055.05 lat (usec): min=18, max=1529.7k, avg=793.74, stdev=16055.05 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 32], 10.00th=[ 35], | 20.00th=[ 41], 30.00th=[ 49], 40.00th=[ 65], | 50.00th=[ 90], 60.00th=[ 115], 70.00th=[ 190], | 80.00th=[ 570], 90.00th=[ 1450], 95.00th=[ 3261], | 99.00th=[ 9110], 99.50th=[ 10945], 99.90th=[ 17433], | 99.95th=[ 20579], 99.99th=[1468007] bw ( KiB/s): min=144935, max=391526, per=100.00%, avg=213287.26, stdev=10421.85, samples=100 iops : min= 9054, max=24466, avg=13326.54, stdev=651.34, samples=100 lat (usec) : 10=9.51%, 20=53.01%, 50=12.11%, 100=15.55%, 250=6.80% lat (usec) : 500=0.71%, 750=0.53%, 1000=0.33% lat (msec) : 2=0.65%, 4=0.37%, 10=0.35%, 20=0.07%, 50=0.01% lat (msec) : 2000=0.01% cpu : usr=2.80%, sys=30.05%, ctx=84027, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=471684,52604,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=755MiB/s (792MB/s), 755MiB/s-755MiB/s (792MB/s-792MB/s), io=7370MiB (7728MB), run=9763-9763msec WRITE: bw=84.2MiB/s (88.3MB/s), 84.2MiB/s-84.2MiB/s (88.3MB/s-88.3MB/s), io=822MiB (862MB), run=9763-9763msec