Versions Compared

Old Version 29

changes.mady.by.user Oleksandr Liutyi

Saved on

compared with

New Version Current

changes.mady.by.user Oleksandr Liutyi

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

OS: TrueNAS Scale 25.10 RC1,

Power Limits: 15W/15W (BIOS: Performance mode)

Disk pool record size: 128K

...

4xSamsung 990 Pro
GMKtec NucBox G9
TrueNAS and Synology		SequentalRandom
Read (MiB/s)Write (MiB/s)Read (MiB/s)Write (MiB/s)
RAIDcompressionbs = 1Mbs = 4Kbs = 1Mbs = 4Kbs = 1Mbs = 4Kbs = 1Mbs = 4K
MirrorGZIP-1/4K1261635102107126948910296.9
MirrorGZIP-1/16K2876360

147

224286611514870.4
MirrorGZIP-1/128K44962870529

93.2

348439028.091.119.8
MirrorGZIP-1/1M5082560

93.0

375492929.792.50.4
MirrorLZ4/4K1197641

628

3401272484567241
MirrorLZ4/16K2790882

1168

48928213771147186
MirrorLZ4/128K42011047

1517

617409666.7144649.4
MirrorLZ4/1M45061064

1428

652476814.815856.7
MirrorOFF/4K1219665

646

3351274598526246
MirrorOFF/16K2872890

1550

50227964441454223
MirrorOFF/128K44911076

2015

5954443173201260.2
MirrorOFF/1M50571100

2054

636505129.920527.5
MirrorOFF/4M42271028

1925

47716227.45411.9
MirrorAES-256-GCM4462104619314611896170187634.5
Mirror
DS220+		OFF
2xHDD		17962.716820.918261.51699.6

Mirror

DS220+

AES-256-CBC

(not direct)

139

178

129

138

32.43.5142

5.6

...


RAID0 on Ubuntu 24.04 using LVM, BTRFS and MDADM 

4xSamsung 990 Pro
GMKtec NucBox G9
Ubuntu 24.04 RAID0		SequentalRandom
Read (MiB/s)Write (MiB/s)Read (MiB/s)Write (MiB/s)
deviceRAIDdisk typecompressionbs = 1Mbs = 4Kbs = 1Mbs = 4Kbs = 1Mbs = 4Kbs = 1Mbs = 4K
R100N1xNVMe13021231047104121149.11056103
R100S1xSSD40557.434749.932120.034148.5
N200N1xNVMe129389.91212202125574.01152193
I1071N1xNVMe27202782691280246275.02414216
I1355N1xNVMe175628916.0 2.5134857.415.92.6
DS220+2xHDD17962.716820.918261.51699.6

For disk performance comparison next system were use,

...

1 Year is 8766 hours (37877 hours = 4.3 Years)

Commands used to test disk speed

Code Block
cd /mnt/pool/shared/user
# each command executed 3 times
# 1M
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=1M --size=4G --readwrite=write --ramp_time=4 # Col 3
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=1M --size=4G --readwrite=read --ramp_time=4 # Col 1
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=1M --size=4G --readwrite=randwrite --ramp_time=4  # Col  7
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=1M --size=4G --readwrite=randread --ramp_time=4 # Col 5
# 4K
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=4K --size=4G --readwrite=write --ramp_time=4 # Col 4
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=4K --size=4G --readwrite=read --ramp_time=4 # Col 2
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=4k --size=4G --readwrite=randwrite --ramp_time=4 # Col 8
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=4k --size=4G --readwrite=randread --ramp_time=4 # Col 6

Check Power Limits

Code Block
root@truenas:/mnt/pool/shared/user# turbostat 2>&1|grep Watt
RAPL: 43691 sec. Joule Counter Range, at 6 Watts
cpu0: MSR_RAPL_POWER_UNIT: 0x000a0e03 (0.125000 Watts, 0.000061 Joules, 0.000977 sec.)
cpu0: PKG Limit #1: ENabled (15.000 Watts, 8.000000 sec, clamp ENabled)
cpu0: PKG Limit #2: ENabled (15.000 Watts, 0.002441* sec, clamp DISabled)
cpu0: PKG Limit #4: 78.000000 Watts (UNlocked)
cpu0: DRAM Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled)
cpu0: Cores Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled)
cpu0: GFX Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled)

RAIDZ1 (LZ4) commands output

Code Block
root@truenas:/mnt/pool/shared/user# zfs get all pool/shared |grep comp
pool/shared  compressratio            1.41x                    -
pool/shared  compression              lz4                      inherited from pool
pool/shared  refcompressratio         1.41x                    -
Code Block
root@truenas:/mnt/pool/shared/user# sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=randwrite --ramp_time=4
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=0): [f(1)][100.0%][w=147MiB/s][w=37.6k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=41307: Thu Apr 10 12:49:43 2025
  write: IOPS=21.1k, BW=82.5MiB/s (86.5MB/s)(3672MiB/44503msec); 0 zone resets
    slat (usec): min=4, max=80062, avg=45.37, stdev=167.42
    clat (nsec): min=614, max=4072.6k, avg=1133.86, stdev=6985.89
     lat (usec): min=5, max=80072, avg=46.50, stdev=167.70
    clat percentiles (nsec):
     |  1.00th=[  740],  5.00th=[  836], 10.00th=[  884], 20.00th=[  924],
     | 30.00th=[  956], 40.00th=[  972], 50.00th=[  996], 60.00th=[ 1020],
     | 70.00th=[ 1048], 80.00th=[ 1080], 90.00th=[ 1192], 95.00th=[ 1384],
     | 99.00th=[ 2008], 99.50th=[ 3312], 99.90th=[24704], 99.95th=[41728],
     | 99.99th=[72192]
   bw (  KiB/s): min=48766, max=137119, per=98.81%, avg=83487.14, stdev=11581.15, samples=87
   iops        : min=12191, max=34279, avg=20872.08, stdev=2895.09, samples=87
  lat (nsec)   : 750=1.20%, 1000=50.42%
  lat (usec)   : 2=47.36%, 4=0.59%, 10=0.14%, 20=0.16%, 50=0.10%
  lat (usec)   : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=3.69%, sys=85.07%, ctx=107206, majf=0, minf=37
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,940003,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=82.5MiB/s (86.5MB/s), 82.5MiB/s-82.5MiB/s (86.5MB/s-86.5MB/s), io=3672MiB (3850MB), run=44503-44503msec


root@truenas:/mnt/pool/shared/user# sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=randread --ramp_time=4
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [r(1)][91.3%][r=160MiB/s][r=41.0k IOPS][eta 00m:04s]
test: (groupid=0, jobs=1): err= 0: pid=41962: Thu Apr 10 12:50:58 2025
  read: IOPS=25.4k, BW=99.2MiB/s (104MB/s)(3708MiB/37391msec)
    slat (usec): min=2, max=467, avg=38.12, stdev=11.50
    clat (nsec): min=587, max=54938, avg=738.69, stdev=251.78
     lat (usec): min=3, max=470, avg=38.86, stdev=11.53
    clat percentiles (nsec):
     |  1.00th=[  636],  5.00th=[  668], 10.00th=[  684], 20.00th=[  692],
     | 30.00th=[  700], 40.00th=[  708], 50.00th=[  716], 60.00th=[  724],
     | 70.00th=[  740], 80.00th=[  756], 90.00th=[  804], 95.00th=[  876],
     | 99.00th=[ 1012], 99.50th=[ 1096], 99.90th=[ 2416], 99.95th=[ 3056],
     | 99.99th=[14144]
   bw (  KiB/s): min=91048, max=157648, per=98.65%, avg=100191.19, stdev=8691.21, samples=74
   iops        : min=22762, max=39412, avg=25047.76, stdev=2172.79, samples=74
  lat (nsec)   : 750=76.93%, 1000=21.93%
  lat (usec)   : 2=1.03%, 4=0.08%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.01%
  cpu          : usr=2.95%, sys=96.97%, ctx=420, majf=0, minf=37
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=949364,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=99.2MiB/s (104MB/s), 99.2MiB/s-99.2MiB/s (104MB/s-104MB/s), io=3708MiB (3889MB), run=37391-37391msec

root@truenas:/mnt/pool/shared/user# sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=write --ramp_time=4
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [W(1)][53.8%][w=555MiB/s][w=142k IOPS][eta 00m:06s]
test: (groupid=0, jobs=1): err= 0: pid=43581: Thu Apr 10 12:53:42 2025
  write: IOPS=145k, BW=568MiB/s (595MB/s)(1808MiB/3184msec); 0 zone resets
    slat (usec): min=3, max=7144, avg= 5.85, stdev=13.87
    clat (nsec): min=581, max=149242, avg=722.02, stdev=731.07
     lat (usec): min=4, max=7154, avg= 6.57, stdev=13.94
    clat percentiles (nsec):
     |  1.00th=[  636],  5.00th=[  644], 10.00th=[  652], 20.00th=[  660],
     | 30.00th=[  668], 40.00th=[  676], 50.00th=[  684], 60.00th=[  692],
     | 70.00th=[  700], 80.00th=[  732], 90.00th=[  844], 95.00th=[  892],
     | 99.00th=[  972], 99.50th=[ 1144], 99.90th=[ 1576], 99.95th=[ 4384],
     | 99.99th=[47360]
   bw (  KiB/s): min=565370, max=595054, per=99.72%, avg=579799.00, stdev=12657.07, samples=6
   iops        : min=141342, max=148763, avg=144949.50, stdev=3164.21, samples=6
  lat (nsec)   : 750=82.16%, 1000=17.03%
  lat (usec)   : 2=0.73%, 4=0.02%, 10=0.02%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%
  cpu          : usr=13.51%, sys=84.32%, ctx=1523, majf=0, minf=37
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,462792,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=568MiB/s (595MB/s), 568MiB/s-568MiB/s (595MB/s-595MB/s), io=1808MiB (1896MB), run=3184-3184msec

RAIDZ1 (OFF) commands output

Code Block
root@truenas:/mnt/pool/shared/user# zfs get all pool/shared |grep comp
pool/shared  compressratio            1.41x                    -
pool/shared  compression              off                      local
pool/shared  refcompressratio         1.41x                    -

...

ZFS MirrorOFF/4K1219665

646

3351274598526246
ZFS StripeOFF/128K4357986

3405

5084258168300780.3
LVM Stripe (ext4)OFF/4K2596213

2434

165230265.32437162
LVM Stripe (ext4)OFF/16K2574220

2357

167216364.92362157
LVM Stripe (ext4)OFF/128K261951

2369

148232265.62372143
btrfs raid 0OFF/4K1947126

2017

134194342.51988118
btrfs raid 0ON/4K196044.7

2036

129193040.01993117
mdraid (ext4)OFF/4K2292207

2132

157205165.42037148
mdraid (ext4)OFF/16K2271211

2080

153204864.52067155


Code Block
mkfs.ext4 -C 16384 -O  bigalloc -L EXT4-16K /dev/storage/fs
mkfs.btrfs --data raid0 --sectorsize 4k /dev/nvme0n1p1 /dev/nvme1n1p1 /dev/nvme2n1p1 /dev/nvme3n1p1

Table for comparison with non-NAS devices


SequentalRandom
Read (MiB/s)Write (MiB/s)Read (MiB/s)Write (MiB/s)
devicedisk typebs = 1Mbs = 4Kbs = 1Mbs = 4Kbs = 1Mbs = 4Kbs = 1Mbs = 4K
R100N1xNVMe13021231047104121149.11056103
R100S1xSSD40557.434749.932120.034148.5
N200N1xNVMe129389.91212202125574.01152193
I1071N1xNVMe27202782691280246275.02414216
I1355N1xNVMe175628916.0 2.5134857.415.92.6
U185HL2xNVME104313610591781486112592142
U185HH2xNVME5936 131 4245188 6032 1204452 75-166 
DS220+2xHDD17962.716820.918261.51699.6

For disk performance comparison next system were use,

codeHardwareDiskspace used,%Power On, hoursLife Time Remain, %
N150RIntel N150, 12GB LPDDR5, Debian, TrueNAS 25.04 RC1, ZFSSamsung SSD 990 PRO with Heatsink 4TB (4 in RAID)156100
R100NIntel N100, 16GB DDR4, Ubuntu 24.04, CasaOS, ext4Movespeed YSSDHB-4TN7000 4TB NVMe84822100
R100SIntel N100, 16GB DDR4, Ubuntu 24.04, CasaOS, ext4Transcend SSD230S 4 TB SATA SSD491522100
N200NIntel N200, 32GB DDR5, Ubuntu 24.04, ext4Fanxiang S500PRO 2TB978343100
I1071NIntel Core i7-10710U, 64GB DDR4, Ubuntu 24.0Lexar SSD ARES 4TB741223899
I1355NIntel Core i7-1355U (15W), 96GB DDR5, Ubuntu 24.04XPG GAMMIX S70 BLADE 8TB856823100
U185HLIntel Core Ultra 9 (95W), 128GB DDR5, Ubuntu 24, btrfs RAID0WD_BLACK SN850X 8TB (2 in btrfs RAID0) - 3.40W48336100
U185HHIntel Core Ultra 9 (95W), 128GB DDR5, Ubuntu 24, btrfs RAID0WD_BLACK SN850X 8TB (2 in btrfs RAID0) - 8.25W48336100
DS220+Synology DS220+, J4025, 2GB, RAID1, btrfsWD Gold Enterprise Class 18 TB 7200 RPM(2 in RAID1)7737877N/A

1 Year is 8766 hours (37877 hours = 4.3 Years)


Commands used to test disk speed

Code Block
cd /mnt/pool/shared/user
# each command executed 3 times
# 1M
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=1M --size=4G --readwrite=write --ramp_time=4 # Col 3
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=1M --size=4G --readwrite=read --ramp_time=4 # Col 1
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=1M --size=4G --readwrite=randwrite --ramp_time=4  # Col  7
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=1M --size=4G --readwrite=randread --ramp_time=4 # Col 5
# 4K
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=4K --size=4G --readwrite=write --ramp_time=4 # Col 4
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=4K --size=4G --readwrite=read --ramp_time=4 # Col 2
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=4k --size=4G --readwrite=randwrite --ramp_time=4 # Col 8
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=4k --size=4G --readwrite=randread --ramp_time=4 # Col 6


Table of Contents