...
OS: TrueNAS Scale 25.10 RC1,
Power Limits: 15W/15W (BIOS: Performance mode)
Disk pool record size: 128K
...
| 4xSamsung 990 Pro GMKtec NucBox G9 TrueNAS and Synology | Sequental | Random | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Read (MiB/s) | Write (MiB/s) | Read (MiB/s) | Write (MiB/s) | ||||||
| RAID | compression | bs = 1M | bs = 4K | bs = 1M | bs = 4K | bs = 1M | bs = 4K | bs = 1M | bs = 4K |
| Mirror | GZIP-1/4K | 1261 | 635 | 102 | 107 | 1269 | 489 | 102 | 96.9 |
| Mirror | GZIP-1/16K | 2876 | 360 | 147 | 224 | 2866 | 115 | 148 | 70.4 |
| Mirror | GZIP-1/128K | 2870 | 529 | 93.2 | 348 | 4390 | 28.0 | 91.1 | 19.8 |
| Mirror | GZIP-1/1M | 5082 | 560 | 93.0 | 375 | 4929 | 29.7 | 92.5 | 0.4 |
| Mirror | LZ4/4K | 1197 | 641 | 628 | 340 | 1272 | 484 | 567 | 241 |
| Mirror | LZ4/16K | 2790 | 882 | 1168 | 489 | 2821 | 377 | 1147 | 186 |
| Mirror | LZ4/128K | 4201 | 1047 | 1517 | 617 | 4096 | 66.7 | 1446 | 49.4 |
| Mirror | LZ4/1M | 4506 | 1064 | 1428 | 652 | 4768 | 14.8 | 1585 | 6.7 |
| Mirror | OFF/4K | 1219 | 665 | 646 | 335 | 1274 | 598 | 526 | 246 |
| Mirror | OFF/16K | 2872 | 890 | 1550 | 502 | 2796 | 444 | 1454 | 223 |
| Mirror | OFF/128K | 4491 | 1076 | 2015 | 595 | 4443 | 173 | 2012 | 60.2 |
| Mirror | OFF/1M | 5057 | 1100 | 2054 | 636 | 5051 | 29.9 | 2052 | 7.5 |
| Mirror | OFF/4M | 4227 | 1028 | 1925 | 477 | 1622 | 7.4 | 541 | 1.9 |
| Mirror | AES-256-GCM | 4462 | 1046 | 1931 | 461 | 1896 | 170 | 1876 | 34.5 |
| Mirror DS220+ | OFF 2xHDD | 179 | 62.7 | 168 | 20.9 | 182 | 61.5 | 169 | 9.6 |
Mirror DS220+ | AES-256-CBC (not direct) | 139 | 178 | 129 | 138 | 32.4 | 3.5 | 142 | 5.6 |
RAID0 on Ubuntu 24.04 using LVM
...
, BTRFS and MDADM
| 4xSamsung 990 Pro GMKtec NucBox G9 Ubuntu 24.04 LVM RAID0 | Sequental | Random | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Read (MiB/s) | Write (MiB/s) | Read (MiB/s) | Write (MiB/s) | ||||||
| RAID | compression | bs = 1M | bs = 4K | bs = 1M | bs = 4K | bs = 1M | bs = 4K | bs = 1M | bs = 4K |
| ZFS Mirror | OFF/4K | 1219 | 665 | 646 | 335 | 1274 | 598 | 526 | 246 |
| ZFS Stripe | OFF/128K | 4357 | 986 | 3405 | 508 | 4258 | 168 | 3007 | 80.3 |
| LVM Stripe (ext4) | OFF/4K | 2596 | 213 | 2434 | 165 | 2302 | 65.3 | 2437 | 162 |
| LVM Stripe (ext4) | OFF/16K | 2574 | 220 | 2357 | 167 | 2163 | 64.9 | 2362 | 157 |
| LVM Stripe (ext4) | OFF/128K | 2619 | 51 | 2369 | 148 | 2322 | 65.6 | 2372 | 143 |
| btrfs raid 0 | OFF/4K | 1947 | 126 | 2017 | 134 | 1943 | 42.5 | 1988 | 118 |
| btrfs raid 0 | ON/4K | 1960 | 44.7 | 2036 | 129 | 1930 | 40.0 | 1993 | 117 |
VG PE Size 4M, PV PE Size 4M ext fs default cluster size 4k
| mdraid (ext4) | OFF/4K | 2292 | 207 | 2132 | 157 | 2051 | 65.4 | 2037 | 148 |
| mdraid (ext4) | OFF/16K | 2271 | 211 | 2080 | 153 | 2048 | 64.5 | 2067 | 155 |
| Code Block |
|---|
mkfs.ext4 -C |
| Code Block |
mkfs.ext4 -C 16384 -O bigalloc -L EXT4-16K /dev/storage/fs
mkfs.btrfs --data raid0 --sectorsize 4k /dev/nvme0n1p1 /dev/nvme1n1p1 /dev/nvme2n1p1 /dev/nvme3n1p1 |
Commands used to test disk speed
| Code Block |
|---|
cd /mnt/pool/shared/user
# each command executed 3 times
# 1M
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=1M --size=4G --readwrite=write --ramp_time=4 # Col 3
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=1M --size=4G --readwrite=read --ramp_time=4 # Col 1
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=1M --size=4G --readwrite=randwrite --ramp_time=4 # Col 7
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=1M --size=4G --readwrite=randread --ramp_time=4 # Col 5
# 4K
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=4K --size=4G --readwrite=write --ramp_time=4 # Col 4
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=4K --size=4G --readwrite=read --ramp_time=4 # Col 2
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=4k --size=4G --readwrite=randwrite --ramp_time=4 # Col 8
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=4k --size=4G --readwrite=randread --ramp_time=4 # Col 6 |
Table for comparison with non-NAS devices
...
For disk performance comparison next system were use,
...
1 Year is 8766 hours (37877 hours = 4.3 Years)
Check Power Limits
| Code Block |
|---|
root@truenas:/mnt/pool/shared/user# turbostat 2>&1|grep Watt
RAPL: 43691 sec. Joule Counter Range, at 6 Watts
cpu0: MSR_RAPL_POWER_UNIT: 0x000a0e03 (0.125000 Watts, 0.000061 Joules, 0.000977 sec.)
cpu0: PKG Limit #1: ENabled (15.000 Watts, 8.000000 sec, clamp ENabled)
cpu0: PKG Limit #2: ENabled (15.000 Watts, 0.002441* sec, clamp DISabled)
cpu0: PKG Limit #4: 78.000000 Watts (UNlocked)
cpu0: DRAM Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled)
cpu0: Cores Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled)
cpu0: GFX Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled) |
RAIDZ1 (LZ4) commands output
| Code Block |
|---|
root@truenas:/mnt/pool/shared/user# zfs get all pool/shared |grep comp
pool/shared compressratio 1.41x -
pool/shared compression lz4 inherited from pool
pool/shared refcompressratio 1.41x -
|
| Code Block |
|---|
root@truenas:/mnt/pool/shared/user# sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=randwrite --ramp_time=4
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=0): [f(1)][100.0%][w=147MiB/s][w=37.6k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=41307: Thu Apr 10 12:49:43 2025
write: IOPS=21.1k, BW=82.5MiB/s (86.5MB/s)(3672MiB/44503msec); 0 zone resets
slat (usec): min=4, max=80062, avg=45.37, stdev=167.42
clat (nsec): min=614, max=4072.6k, avg=1133.86, stdev=6985.89
lat (usec): min=5, max=80072, avg=46.50, stdev=167.70
clat percentiles (nsec):
| 1.00th=[ 740], 5.00th=[ 836], 10.00th=[ 884], 20.00th=[ 924],
| 30.00th=[ 956], 40.00th=[ 972], 50.00th=[ 996], 60.00th=[ 1020],
| 70.00th=[ 1048], 80.00th=[ 1080], 90.00th=[ 1192], 95.00th=[ 1384],
| 99.00th=[ 2008], 99.50th=[ 3312], 99.90th=[24704], 99.95th=[41728],
| 99.99th=[72192]
bw ( KiB/s): min=48766, max=137119, per=98.81%, avg=83487.14, stdev=11581.15, samples=87
iops : min=12191, max=34279, avg=20872.08, stdev=2895.09, samples=87
lat (nsec) : 750=1.20%, 1000=50.42%
lat (usec) : 2=47.36%, 4=0.59%, 10=0.14%, 20=0.16%, 50=0.10%
lat (usec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=3.69%, sys=85.07%, ctx=107206, majf=0, minf=37
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,940003,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=82.5MiB/s (86.5MB/s), 82.5MiB/s-82.5MiB/s (86.5MB/s-86.5MB/s), io=3672MiB (3850MB), run=44503-44503msec
root@truenas:/mnt/pool/shared/user# sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=randread --ramp_time=4
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [r(1)][91.3%][r=160MiB/s][r=41.0k IOPS][eta 00m:04s]
test: (groupid=0, jobs=1): err= 0: pid=41962: Thu Apr 10 12:50:58 2025
read: IOPS=25.4k, BW=99.2MiB/s (104MB/s)(3708MiB/37391msec)
slat (usec): min=2, max=467, avg=38.12, stdev=11.50
clat (nsec): min=587, max=54938, avg=738.69, stdev=251.78
lat (usec): min=3, max=470, avg=38.86, stdev=11.53
clat percentiles (nsec):
| 1.00th=[ 636], 5.00th=[ 668], 10.00th=[ 684], 20.00th=[ 692],
| 30.00th=[ 700], 40.00th=[ 708], 50.00th=[ 716], 60.00th=[ 724],
| 70.00th=[ 740], 80.00th=[ 756], 90.00th=[ 804], 95.00th=[ 876],
| 99.00th=[ 1012], 99.50th=[ 1096], 99.90th=[ 2416], 99.95th=[ 3056],
| 99.99th=[14144]
bw ( KiB/s): min=91048, max=157648, per=98.65%, avg=100191.19, stdev=8691.21, samples=74
iops : min=22762, max=39412, avg=25047.76, stdev=2172.79, samples=74
lat (nsec) : 750=76.93%, 1000=21.93%
lat (usec) : 2=1.03%, 4=0.08%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=2.95%, sys=96.97%, ctx=420, majf=0, minf=37
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=949364,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=99.2MiB/s (104MB/s), 99.2MiB/s-99.2MiB/s (104MB/s-104MB/s), io=3708MiB (3889MB), run=37391-37391msec
root@truenas:/mnt/pool/shared/user# sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=write --ramp_time=4
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [W(1)][53.8%][w=555MiB/s][w=142k IOPS][eta 00m:06s]
test: (groupid=0, jobs=1): err= 0: pid=43581: Thu Apr 10 12:53:42 2025
write: IOPS=145k, BW=568MiB/s (595MB/s)(1808MiB/3184msec); 0 zone resets
slat (usec): min=3, max=7144, avg= 5.85, stdev=13.87
clat (nsec): min=581, max=149242, avg=722.02, stdev=731.07
lat (usec): min=4, max=7154, avg= 6.57, stdev=13.94
clat percentiles (nsec):
| 1.00th=[ 636], 5.00th=[ 644], 10.00th=[ 652], 20.00th=[ 660],
| 30.00th=[ 668], 40.00th=[ 676], 50.00th=[ 684], 60.00th=[ 692],
| 70.00th=[ 700], 80.00th=[ 732], 90.00th=[ 844], 95.00th=[ 892],
| 99.00th=[ 972], 99.50th=[ 1144], 99.90th=[ 1576], 99.95th=[ 4384],
| 99.99th=[47360]
bw ( KiB/s): min=565370, max=595054, per=99.72%, avg=579799.00, stdev=12657.07, samples=6
iops : min=141342, max=148763, avg=144949.50, stdev=3164.21, samples=6
lat (nsec) : 750=82.16%, 1000=17.03%
lat (usec) : 2=0.73%, 4=0.02%, 10=0.02%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%
cpu : usr=13.51%, sys=84.32%, ctx=1523, majf=0, minf=37
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,462792,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=568MiB/s (595MB/s), 568MiB/s-568MiB/s (595MB/s-595MB/s), io=1808MiB (1896MB), run=3184-3184msec
|
RAIDZ1 (OFF) commands output
| Code Block |
|---|
root@truenas:/mnt/pool/shared/user# zfs get all pool/shared |grep comp
pool/shared compressratio 1.41x -
pool/shared compression off local
pool/shared refcompressratio 1.41x -
|
...
Table for comparison with non-NAS devices
| Sequental | Random | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Read (MiB/s) | Write (MiB/s) | Read (MiB/s) | Write (MiB/s) | ||||||
| device | disk type | bs = 1M | bs = 4K | bs = 1M | bs = 4K | bs = 1M | bs = 4K | bs = 1M | bs = 4K |
| R100N | 1xNVMe | 1302 | 123 | 1047 | 104 | 1211 | 49.1 | 1056 | 103 |
| R100S | 1xSSD | 405 | 57.4 | 347 | 49.9 | 321 | 20.0 | 341 | 48.5 |
| N200N | 1xNVMe | 1293 | 89.9 | 1212 | 202 | 1255 | 74.0 | 1152 | 193 |
| I1071N | 1xNVMe | 2720 | 278 | 2691 | 280 | 2462 | 75.0 | 2414 | 216 |
| U185HL | 2xNVME | 1043 | 136 | 1059 | 178 | 1486 | 112 | 592 | 142 |
| U185HH | 2xNVME | 5936 | 131 | 4245 | 188 | 6032 | 120 | 4452 | 75-166 |
| DS220+ | 2xHDD | 179 | 62.7 | 168 | 20.9 | 182 | 61.5 | 169 | 9.6 |
For disk performance comparison next system were use,
| code | Hardware | Disk | space used,% | Power On, hours | Life Time Remain, % |
|---|---|---|---|---|---|
| N150R | Intel N150, 12GB LPDDR5, Debian, TrueNAS 25.04 RC1, ZFS | Samsung SSD 990 PRO with Heatsink 4TB (4 in RAID) | 1 | 56 | 100 |
| R100N | Intel N100, 16GB DDR4, Ubuntu 24.04, CasaOS, ext4 | Movespeed YSSDHB-4TN7000 4TB NVMe | 8 | 4822 | 100 |
| R100S | Intel N100, 16GB DDR4, Ubuntu 24.04, CasaOS, ext4 | Transcend SSD230S 4 TB SATA SSD | 49 | 1522 | 100 |
| N200N | Intel N200, 32GB DDR5, Ubuntu 24.04, ext4 | Fanxiang S500PRO 2TB | 97 | 8343 | 100 |
| I1071N | Intel Core i7-10710U, 64GB DDR4, Ubuntu 24.0 | Lexar SSD ARES 4TB | 74 | 12238 | 99 |
| U185HL | Intel Core Ultra 9 (95W), 128GB DDR5, Ubuntu 24, btrfs RAID0 | WD_BLACK SN850X 8TB (2 in btrfs RAID0) - 3.40W | 48 | 336 | 100 |
| U185HH | Intel Core Ultra 9 (95W), 128GB DDR5, Ubuntu 24, btrfs RAID0 | WD_BLACK SN850X 8TB (2 in btrfs RAID0) - 8.25W | 48 | 336 | 100 |
| DS220+ | Synology DS220+, J4025, 2GB, RAID1, btrfs | WD Gold Enterprise Class 18 TB 7200 RPM(2 in RAID1) | 77 | 37877 | N/A |
1 Year is 8766 hours (37877 hours = 4.3 Years)
Commands used to test disk speed
| Code Block |
|---|
cd /mnt/pool/shared/user
# each command executed 3 times
# 1M
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=1M --size=4G --readwrite=write --ramp_time=4 # Col 3
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=1M --size=4G --readwrite=read --ramp_time=4 # Col 1
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=1M --size=4G --readwrite=randwrite --ramp_time=4 # Col 7
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=1M --size=4G --readwrite=randread --ramp_time=4 # Col 5
# 4K
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=4K --size=4G --readwrite=write --ramp_time=4 # Col 4
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=4K --size=4G --readwrite=read --ramp_time=4 # Col 2
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=4k --size=4G --readwrite=randwrite --ramp_time=4 # Col 8
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=4k --size=4G --readwrite=randread --ramp_time=4 # Col 6 |
| Table of Contents |
|---|