Versions Compared

Old Version 1

changes.mady.by.user Oleksandr Liutyi

Saved on

compared with

New Version Current

changes.mady.by.user Oleksandr Liutyi

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

HW Setup

GMKtec NucBox G9

CPU: Intel N150,

RAM: 12GB LPDDR5,

DISK: 4x4TB Samsung 990 Pro with heatsink (Gen4x4 inserted in Gen3x2 slots), 

OS: TrueNAS Scale 25.10 RC1,

Power Limits: 15W/15W (BIOS: Performance mode)

Disk pool record size: 128K

TrueNAS 4x4TB NVMe RAID disk speed comparison 

By RAID Mode RAIDZ1 (RAID5), Stripe (RAID0), Mirror (RAID10) and compression (OFF, LZ4, GZIP-1)

4xSamsung 990 Pro
GMKtec NucBox G9
TrueNAS		SequentalRandom
Read (MiB/s)Write (MiB/s)Read (MiB/s)Write (MiB/s)
RAIDcompressionbs = 1Mbs = 4Kbs = 1Mbs = 4Kbs = 1Mbs = 4Kbs = 1Mbs = 4K
RAIDZ1GZIP-1305051198.7478314676.396.321.9
RAIDZ1LZ439968941334568317399.2116782.5
RAIDZ1OFF

4117

998

2513

589

2059

167

1577

62.4

StripeGZIP-1405551789.0333403926.689.317.9
StripeLZ4429410461527617418097.7155165.4
StripeOFF4357986

3405

5084258168300780.3
MirrorGZIP-14496529

93.2

348439028.091.119.8
MirrorLZ442011047

1517

617409666.7144649.4
MirrorOFF44911076

2015

5954443173201260.2
Mirror
DS220+		OFF
2xHDD		17962.716820.918261.51699.6

In RAID10 mode by dataset write size (4K, 16K, 128K) and compression (OFF, LZ4, GZIP-1)

With additional test for Crypto dataset OFF/128K/AES-256-GCM compared to AES-256-CBC of Synology DS220+

4xSamsung 990 Pro
GMKtec NucBox G9
TrueNAS and Synology		SequentalRandom
Read (MiB/s)Write (MiB/s)Read (MiB/s)Write (MiB/s)
RAIDcompressionbs = 1Mbs = 4Kbs = 1Mbs = 4Kbs = 1Mbs = 4Kbs = 1Mbs = 4K
MirrorGZIP-1/4K1261635102107126948910296.9
MirrorGZIP-1/16K2876360

147

224286611514870.4
MirrorGZIP-1/128K2870529

93.2

348439028.091.119.8
MirrorGZIP-1/1M5082560

93.0

375492929.792.50.4
MirrorLZ4/4K1197641

628

3401272484567241
MirrorLZ4/16K2790882

1168

48928213771147186
MirrorLZ4/128K42011047

1517

617409666.7144649.4
MirrorLZ4/1M45061064

1428

652476814.815856.7
MirrorOFF/4K1219665

646

3351274598526246
MirrorOFF/16K2872890

1550

50227964441454223
MirrorOFF/128K44911076

2015

5954443173201260.2
MirrorOFF/1M50571100

2054

636505129.920527.5
MirrorOFF/4M42271028

1925

47716227.45411.9
MirrorAES-256-GCM4462104619314611896170187634.5
Mirror
DS220+		OFF
2xHDD		17962.716820.918261.51699.6

Mirror

DS220+

AES-256-CBC

(not direct)

139

178

129

138

32.43.5142

5.6


RAID0 on Ubuntu 24.04 using LVM, BTRFS and MDADM 

4xSamsung 990 Pro
GMKtec NucBox G9
Ubuntu 24.04 RAID0		SequentalRandom
Read (MiB/s)Write (MiB/s)Read (MiB/s)Write (MiB/s)
RAIDcompressionbs = 1Mbs = 4Kbs = 1Mbs = 4Kbs = 1Mbs = 4Kbs = 1Mbs = 4K
ZFS MirrorOFF/4K1219665

646

3351274598526246
ZFS StripeOFF/128K4357986

3405

5084258168300780.3
LVM Stripe (ext4)OFF/4K2596213

2434

165230265.32437162
LVM Stripe (ext4)OFF/16K2574220

2357

167216364.92362157
LVM Stripe (ext4)OFF/128K261951

2369

148232265.62372143
btrfs raid 0OFF/4K1947126

2017

134194342.51988118
btrfs raid 0ON/4K196044.7

2036

129193040.01993117
mdraid (ext4)OFF/4K2292207

2132

157205165.42037148
mdraid (ext4)OFF/16K2271211

2080

153204864.52067155


Code Block
mkfs.ext4 -C 16384 -O  bigalloc -L EXT4-16K /dev/storage/fs
mkfs.btrfs --data raid0 --sectorsize 4k /dev/nvme0n1p1 /dev/nvme1n1p1 /dev/nvme2n1p1 /dev/nvme3n1p1

Table for comparison with non-NAS devices


SequentalRandom
Read (MiB/s)Write (MiB/s)Read (MiB/s)Write (MiB/s)
devicedisk typebs = 1Mbs = 4Kbs = 1Mbs = 4Kbs = 1Mbs = 4Kbs = 1Mbs = 4K
R100N1xNVMe13021231047104121149.11056103
R100S1xSSD40557.434749.932120.034148.5
N200N1xNVMe129389.91212202125574.01152193
I1071N1xNVMe27202782691280246275.02414216
I1355N1xNVMe175628916.0 2.5134857.415.92.6
U185HL2xNVME104313610591781486112592142
U185HH2xNVME5936 131 4245188 6032 1204452 75-166 
DS220+2xHDD17962.716820.918261.51699.6

For disk performance comparison next system were use,

codeHardwareDiskspace used,%Power On, hoursLife Time Remain, %
N150RIntel N150, 12GB LPDDR5, Debian, TrueNAS 25.04 RC1, ZFSSamsung SSD 990 PRO with Heatsink 4TB (4 in RAID)156100
R100NIntel N100, 16GB DDR4, Ubuntu 24.04, CasaOS, ext4Movespeed YSSDHB-4TN7000 4TB NVMe84822100
R100SIntel N100, 16GB DDR4, Ubuntu 24.04, CasaOS, ext4Transcend SSD230S 4 TB SATA SSD491522100
N200NIntel N200, 32GB DDR5, Ubuntu 24.04, ext4Fanxiang S500PRO 2TB978343100
I1071NIntel Core i7-10710U, 64GB DDR4, Ubuntu 24.0Lexar SSD ARES 4TB741223899
I1355NIntel Core i7-1355U (15W), 96GB DDR5, Ubuntu 24.04XPG GAMMIX S70 BLADE 8TB856823100
U185HLIntel Core Ultra 9 (95W), 128GB DDR5, Ubuntu 24, btrfs RAID0WD_BLACK SN850X 8TB (2 in btrfs RAID0) - 3.40W48336100
U185HHIntel Core Ultra 9 (95W), 128GB DDR5, Ubuntu 24, btrfs RAID0WD_BLACK SN850X 8TB (2 in btrfs RAID0) - 8.25W48336100
DS220+Synology DS220+, J4025, 2GB, RAID1, btrfsWD Gold Enterprise Class 18 TB 7200 RPM(2 in RAID1)7737877N/A

1 Year is 8766 hours (37877 hours = 4.3 Years)


Commands used to test disk speed

Code Block
cd /mnt/pool/shared/user
# each command executed 3 times
# 1M
fio

...

Tech details

Code Block
root@truenas:/mnt/pool/shared/user# sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=randwrite --ramp_time=4
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=0): [f(1)][100.0%][w=147MiB/s][w=37.6k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=41307: Thu Apr 10 12:49:43 2025
  write: IOPS=21.1k, BW=82.5MiB/s (86.5MB/s)(3672MiB/44503msec); 0 zone resets
    slat (usec): min=4, max=80062, avg=45.37, stdev=167.42
    clat (nsec): min=614, max=4072.6k, avg=1133.86, stdev=6985.89
     lat (usec): min=5, max=80072, avg=46.50, stdev=167.70
    clat percentiles (nsec):
     |  1.00th=[  740],  5.00th=[  836], 10.00th=[  884], 20.00th=[  924],
     | 30.00th=[  956], 40.00th=[  972], 50.00th=[  996], 60.00th=[ 1020],
     | 70.00th=[ 1048], 80.00th=[ 1080], 90.00th=[ 1192], 95.00th=[ 1384],
     | 99.00th=[ 2008], 99.50th=[ 3312], 99.90th=[24704], 99.95th=[41728],
     | 99.99th=[72192]
   bw (  KiB/s): min=48766, max=137119, per=98.81%, avg=83487.14, stdev=11581.15, samples=87
   iops        : min=12191, max=34279, avg=20872.08, stdev=2895.09, samples=87
  lat (nsec)   : 750=1.20%, 1000=50.42%
  lat (usec)   : 2=47.36%, 4=0.59%, 10=0.14%, 20=0.16%, 50=0.10%
  lat (usec)   : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=3.69%, sys=85.07%, ctx=107206, majf=0, minf=37
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,940003,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=82.5MiB/s (86.5MB/s), 82.5MiB/s-82.5MiB/s (86.5MB/s-86.5MB/s), io=3672MiB (3850MB), run=44503-44503msec


root@truenas:/mnt/pool/shared/user# sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=testtest0 --bs=4k1M --size=4G --readwrite=randreadwrite --ramp_time=4
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [r(1)][91.3%][r=160MiB/s][r=41.0k IOPS][eta 00m:04s]
test: (groupid=0, jobs=1): err= 0: pid=41962: Thu Apr 10 12:50:58 2025
  read: IOPS=25.4k, BW=99.2MiB/s (104MB/s)(3708MiB/37391msec)
    slat (usec): min=2, max=467, avg=38.12, stdev=11.50
    clat (nsec): min=587, max=54938, avg=738.69, stdev=251.78
     lat (usec): min=3, max=470, avg=38.86, stdev=11.53
    clat percentiles (nsec):
     |  1.00th=[  636],  5.00th=[  668], 10.00th=[  684], 20.00th=[  692],
     | 30.00th=[  700], 40.00th=[  708], 50.00th=[  716], 60.00th=[  724],
     | 70.00th=[  740], 80.00th=[  756], 90.00th=[  804], 95.00th=[  876],
     | 99.00th=[ 1012], 99.50th=[ 1096], 99.90th=[ 2416], 99.95th=[ 3056],
     | 99.99th=[14144]
   bw (  KiB/s): min=91048, max=157648, per=98.65%, avg=100191.19, stdev=8691.21, samples=74
   iops        : min=22762, max=39412, avg=25047.76, stdev=2172.79, samples=74
  lat (nsec)   : 750=76.93%, 1000=21.93%
  lat (usec)   : 2=1.03%, 4=0.08%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.01%
  cpu          : usr=2.95%, sys=96.97%, ctx=420, majf=0, minf=37
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=949364,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=99.2MiB/s (104MB/s), 99.2MiB/s-99.2MiB/s (104MB/s-104MB/s), io=3708MiB (3889MB), run=37391-37391msec

 # Col 3
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=1M --size=4G --readwrite=read --ramp_time=4 # Col 1
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=1M --size=4G --readwrite=randwrite --ramp_time=4  # Col  7
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=1M --size=4G --readwrite=randread --ramp_time=4 # Col 5
# 4K
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=4K --size=4G --readwrite=write --ramp_time=4 # Col 4
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=4K --size=4G --readwrite=read --ramp_time=4 # Col 2
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=4k --size=4G --readwrite=randwrite --ramp_time=4 # Col 8
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=4k --size=4G --readwrite=randread --ramp_time=4 # Col 6


Table of Contents