Read | Write | Random write | Random Read | |
---|---|---|---|---|
RAIDZ1 (LZ) | 894MiB/s-894MiB/s | 568MiB/s-568MiB/s | 82.5MiB/s-82.5MiB/s | 99.2MiB/s-99.2MiB/s |
Commands used
cd /mnt/pool/shared/user sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=write --ramp_time=4 sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=write --ramp_time=4 sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=randwrite --ramp_time=4 sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=randread --ramp_time=4
Check Power Limits
root@truenas:/mnt/pool/shared/user# turbostat 2>&1|grep Watt RAPL: 43691 sec. Joule Counter Range, at 6 Watts cpu0: MSR_RAPL_POWER_UNIT: 0x000a0e03 (0.125000 Watts, 0.000061 Joules, 0.000977 sec.) cpu0: PKG Limit #1: ENabled (15.000 Watts, 8.000000 sec, clamp ENabled) cpu0: PKG Limit #2: ENabled (15.000 Watts, 0.002441* sec, clamp DISabled) cpu0: PKG Limit #4: 78.000000 Watts (UNlocked) cpu0: DRAM Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled) cpu0: Cores Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled) cpu0: GFX Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled)
root@truenas:/mnt/pool/shared/user# zfs get all pool |grep comp pool compressratio 1.61x - pool compression lz4 local pool refcompressratio 1.00x -
root@truenas:/mnt/pool/shared/user# zfs get all pool/shared |grep comp pool/shared compressratio 1.41x - pool/shared compression lz4 inherited from pool pool/shared refcompressratio 1.41x -
RAIDZ1 (LZ) commands output
root@truenas:/mnt/pool/shared/user# sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=randwrite --ramp_time=4 test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.33 Starting 1 process test: Laying out IO file (1 file / 4096MiB) Jobs: 1 (f=0): [f(1)][100.0%][w=147MiB/s][w=37.6k IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=41307: Thu Apr 10 12:49:43 2025 write: IOPS=21.1k, BW=82.5MiB/s (86.5MB/s)(3672MiB/44503msec); 0 zone resets slat (usec): min=4, max=80062, avg=45.37, stdev=167.42 clat (nsec): min=614, max=4072.6k, avg=1133.86, stdev=6985.89 lat (usec): min=5, max=80072, avg=46.50, stdev=167.70 clat percentiles (nsec): | 1.00th=[ 740], 5.00th=[ 836], 10.00th=[ 884], 20.00th=[ 924], | 30.00th=[ 956], 40.00th=[ 972], 50.00th=[ 996], 60.00th=[ 1020], | 70.00th=[ 1048], 80.00th=[ 1080], 90.00th=[ 1192], 95.00th=[ 1384], | 99.00th=[ 2008], 99.50th=[ 3312], 99.90th=[24704], 99.95th=[41728], | 99.99th=[72192] bw ( KiB/s): min=48766, max=137119, per=98.81%, avg=83487.14, stdev=11581.15, samples=87 iops : min=12191, max=34279, avg=20872.08, stdev=2895.09, samples=87 lat (nsec) : 750=1.20%, 1000=50.42% lat (usec) : 2=47.36%, 4=0.59%, 10=0.14%, 20=0.16%, 50=0.10% lat (usec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=3.69%, sys=85.07%, ctx=107206, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,940003,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=82.5MiB/s (86.5MB/s), 82.5MiB/s-82.5MiB/s (86.5MB/s-86.5MB/s), io=3672MiB (3850MB), run=44503-44503msec root@truenas:/mnt/pool/shared/user# sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=randread --ramp_time=4 test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.33 Starting 1 process Jobs: 1 (f=1): [r(1)][91.3%][r=160MiB/s][r=41.0k IOPS][eta 00m:04s] test: (groupid=0, jobs=1): err= 0: pid=41962: Thu Apr 10 12:50:58 2025 read: IOPS=25.4k, BW=99.2MiB/s (104MB/s)(3708MiB/37391msec) slat (usec): min=2, max=467, avg=38.12, stdev=11.50 clat (nsec): min=587, max=54938, avg=738.69, stdev=251.78 lat (usec): min=3, max=470, avg=38.86, stdev=11.53 clat percentiles (nsec): | 1.00th=[ 636], 5.00th=[ 668], 10.00th=[ 684], 20.00th=[ 692], | 30.00th=[ 700], 40.00th=[ 708], 50.00th=[ 716], 60.00th=[ 724], | 70.00th=[ 740], 80.00th=[ 756], 90.00th=[ 804], 95.00th=[ 876], | 99.00th=[ 1012], 99.50th=[ 1096], 99.90th=[ 2416], 99.95th=[ 3056], | 99.99th=[14144] bw ( KiB/s): min=91048, max=157648, per=98.65%, avg=100191.19, stdev=8691.21, samples=74 iops : min=22762, max=39412, avg=25047.76, stdev=2172.79, samples=74 lat (nsec) : 750=76.93%, 1000=21.93% lat (usec) : 2=1.03%, 4=0.08%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=0.01% cpu : usr=2.95%, sys=96.97%, ctx=420, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=949364,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=99.2MiB/s (104MB/s), 99.2MiB/s-99.2MiB/s (104MB/s-104MB/s), io=3708MiB (3889MB), run=37391-37391msec root@truenas:/mnt/pool/shared/user# sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=write --ramp_time=4 test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.33 Starting 1 process Jobs: 1 (f=1): [W(1)][53.8%][w=555MiB/s][w=142k IOPS][eta 00m:06s] test: (groupid=0, jobs=1): err= 0: pid=43581: Thu Apr 10 12:53:42 2025 write: IOPS=145k, BW=568MiB/s (595MB/s)(1808MiB/3184msec); 0 zone resets slat (usec): min=3, max=7144, avg= 5.85, stdev=13.87 clat (nsec): min=581, max=149242, avg=722.02, stdev=731.07 lat (usec): min=4, max=7154, avg= 6.57, stdev=13.94 clat percentiles (nsec): | 1.00th=[ 636], 5.00th=[ 644], 10.00th=[ 652], 20.00th=[ 660], | 30.00th=[ 668], 40.00th=[ 676], 50.00th=[ 684], 60.00th=[ 692], | 70.00th=[ 700], 80.00th=[ 732], 90.00th=[ 844], 95.00th=[ 892], | 99.00th=[ 972], 99.50th=[ 1144], 99.90th=[ 1576], 99.95th=[ 4384], | 99.99th=[47360] bw ( KiB/s): min=565370, max=595054, per=99.72%, avg=579799.00, stdev=12657.07, samples=6 iops : min=141342, max=148763, avg=144949.50, stdev=3164.21, samples=6 lat (nsec) : 750=82.16%, 1000=17.03% lat (usec) : 2=0.73%, 4=0.02%, 10=0.02%, 20=0.01%, 50=0.01% lat (usec) : 100=0.01%, 250=0.01% cpu : usr=13.51%, sys=84.32%, ctx=1523, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,462792,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=568MiB/s (595MB/s), 568MiB/s-568MiB/s (595MB/s-595MB/s), io=1808MiB (1896MB), run=3184-3184msec