From Trifecta, 1 Week ago, written in Plain Text.
  1.  
  2. Freenas host (pool with 2 nvme 970 pro 500gb in mirror 1)
  3. # fio --randrepeat=1 --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
  4. test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=64
  5. fio-3.5
  6. Starting 1 process
  7. test: Laying out IO file (1 file / 4096MiB)
  8. Jobs: 1 (f=1): [m(1)][98.8%][r=60.6MiB/s,w=19.4MiB/s][r=15.5k,w=4960 IOPS][eta 00m:01s]
  9. test: (groupid=0, jobs=1): err= 0: pid=10082: Wed Nov 27 17:26:19 2019
  10.    read: IOPS=9735, BW=38.0MiB/s (39.9MB/s)(3070MiB/80730msec)
  11.    bw (  KiB/s): min= 1328, max=128697, per=100.00%, avg=39530.92, stdev=31890.84, samples=157
  12.    iops        : min=  332, max=32174, avg=9882.38, stdev=7972.70, samples=157
  13.   write: IOPS=3253, BW=12.7MiB/s (13.3MB/s)(1026MiB/80730msec)
  14.    bw (  KiB/s): min=  363, max=42518, per=100.00%, avg=13209.90, stdev=10638.72, samples=157
  15.    iops        : min=   90, max=10629, avg=3302.09, stdev=2659.66, samples=157
  16.   cpu          : usr=1.53%, sys=12.69%, ctx=306210, majf=0, minf=0
  17.   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  18.      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  19.      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  20.      issued rwts: total=785920,262656,0,0 short=0,0,0,0 dropped=0,0,0,0
  21.      latency   : target=0, window=0, percentile=100.00%, depth=64
  22.  
  23. Run status group 0 (all jobs):
  24.    READ: bw=38.0MiB/s (39.9MB/s), 38.0MiB/s-38.0MiB/s (39.9MB/s-39.9MB/s), io=3070MiB (3219MB), run=80730-80730msec
  25.   WRITE: bw=12.7MiB/s (13.3MB/s), 12.7MiB/s-12.7MiB/s (13.3MB/s-13.3MB/s), io=1026MiB (1076MB), run=80730-80730msec
  26.  
  27.  
  28.  
  29.   Vm on same pool:
  30.  
  31.   # fio --randrepeat=1 --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
  32. test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=64
  33. fio-3.1
  34. Starting 1 process
  35. test: Laying out IO file (1 file / 4096MiB)
  36. Jobs: 1 (f=1): [m(1)][100.0%][r=14.9MiB/s,w=5144KiB/s][r=3804,w=1286 IOPS][eta 00m:00s]
  37. test: (groupid=0, jobs=1): err= 0: pid=11428: Wed Nov 27 19:13:46 2019
  38.    read: IOPS=3689, BW=14.4MiB/s (15.1MB/s)(3070MiB/213035msec)
  39.    bw (  KiB/s): min= 7088, max=16600, per=100.00%, avg=14792.69, stdev=952.58, samples=426
  40.    iops        : min= 1772, max= 4150, avg=3698.05, stdev=238.14, samples=426
  41.   write: IOPS=1232, BW=4932KiB/s (5050kB/s)(1026MiB/213035msec)
  42.    bw (  KiB/s): min= 2264, max= 5984, per=100.00%, avg=4943.89, stdev=362.96, samples=426
  43.    iops        : min=  566, max= 1496, avg=1235.87, stdev=90.74, samples=426
  44.   cpu          : usr=0.86%, sys=5.57%, ctx=1049883, majf=0, minf=8
  45.   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  46.      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  47.      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  48.      issued rwt: total=785920,262656,0, short=0,0,0, dropped=0,0,0
  49.      latency   : target=0, window=0, percentile=100.00%, depth=64
  50.  
  51. Run status group 0 (all jobs):
  52.    READ: bw=14.4MiB/s (15.1MB/s), 14.4MiB/s-14.4MiB/s (15.1MB/s-15.1MB/s), io=3070MiB (3219MB), run=213035-213035msec
  53.   WRITE: bw=4932KiB/s (5050kB/s), 4932KiB/s-4932KiB/s (5050kB/s-5050kB/s), io=1026MiB (1076MB), run=213035-213035msec
  54.  
captcha