...
Disk pool record size: 128K
TrueNAS 4x4TB NVMe RAID disk speed comparison
By RAID Mode RAIDZ1 (RAID5), Stripe (RAID0), Mirror (RAID10) and compression (OFF, LZ4, GZIP-1)
| 4xSamsung 990 Pro GMKtec NucBox G9 TrueNAS | Sequental | Random | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Read (MiB/s) | Write (MiB/s) | Read (MiB/s) | Write (MiB/s) | ||||||
| RAID | compression | bs = 1M | bs = 4K | bs = 1M | bs = 4K | bs = 1M | bs = 4K | bs = 1M | bs = 4K |
| RAIDZ1 | GZIP-1 | 3050 | 511 | 98.7 | 478 | 3146 | 76.3 | 96.3 | 21.9 |
| RAIDZ1 | LZ4 | 3996 | 894 | 1334 | 568 | 3173 | 99.2 | 1167 | 82.5 |
| RAIDZ1 | OFF | 4117 | 998 | 2513 | 589 | 2059 | 167 | 1577 | 62.4 |
| Stripe | GZIP-1 | 4055 | 517 | 89.0 | 333 | 4039 | 26.6 | 89.3 | 17.9 |
| Stripe | LZ4 | 4294 | 1046 | 1527 | 617 | 4180 | 97.7 | 1551 | 65.4 |
| Stripe | OFF | 4357 | 986 | 3405 | 508 | 4258 | 168 | 3007 | 80.3 |
| Mirror | GZIP-1 | 4496 | 529 | 93.2 | 348 | 4390 | 28.0 | 91.1 | 19.8 |
| Mirror | LZ4 | 4201 | 1047 | 1517 | 617 | 4096 | 66.7 | 1446 | 49.4 |
| Mirror | OFF | 4491 | 1076 | 2015 | 595 | 4443 | 173 | 2012 | 60.2 |
| Mirror DS220+ | OFF 2xHDD | 179 | 62.7 | 168 | 20.9 | 182 | 61.5 | 169 | 9.6 |
In RAID10 mode by dataset write size (4K, 16K, 128K) and compression (OFF, LZ4, GZIP-1)
With additional test for Crypto dataset OFF/128K/AES-256-GCM compared to AES-256-CBC of Synology DS220+Additional Crypto and low record size tests on Mirror
| 4xSamsung 990 Pro GMKtec NucBox G9 TrueNAS and Synology | Sequental | Random | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Read (MiB/s) | Write (MiB/s) | Read (MiB/s) | Write (MiB/s) | |||||||
| RAID | compression | bs = 1M | bs = 4K | bs = 1M | bs = 4K | bs = 1M | bs = 4K | bs = 1M | bs = 4K | |
| Mirror | GZIP-1/4K | 1261 | 635 | 102 | 107 | 1269 | 489 | 102 | 96.9 | |
| Mirror | GZIP-1/128K | 4496 | 529 | 93.2 | 348 | 4390 | 28.0 | 91.1 | 19.8 | |
| Mirror | LZ4/4K | 1197 | 641 | 628 | 340 | 1272 | 484 | 567 | 241 | |
| Mirror | LZ4/ | |||||||||
| 16K | 2790 | 882 | 1168 | 489 | 2821 | 377 | 1147 | 186 | ||
| Mirror | LZ4/128K | 4201 | 4201 | 1047 | 1517 | 617 | 4096 | 66.7 | 1446 | 49.4 |
| Mirror | LZ4/1M | 4506 | 1064 | 1428 | 652 | 4768 | 14.8 | 1585 | 6.7 | |
| Mirror | OFF/4K | 1219 | 665 | 646 | 335 | 1274 | 598 | 526 | 246 | |
| Mirror | OFF/16K | 2872 | 890 | 1550 | 502 | 2796 | 444 | 1454 | 223 | |
| Mirror | OFF/128K | 4491 | 1076 | 2015 | 595 | 4443 | 173 | 2012 | 60.2 | |
| Mirror | OFF/4M | 4227 | 1028 | 1925 | 477 | 1622 | 541 | |||
| Mirror | AES-256-GCM | 4462 | 1046 | 1931 | 461 | 1896 | 170 | 1876 | 34.5 | |
| Mirror DS220+ | OFF 2xHDD | 179 | 62.7 | 168 | 20.9 | 182 | 61.5 | 169 | 9.6 | |
Mirror DS220+ | AES-256-CBC (not direct) | 139 | 178 | 129 | 138 | 32.4 | 3.5 | 142 | 5.6 | |
Commands used
| Code Block |
|---|
cd /mnt/pool/shared/user
# each command executed 3 times
# 1M
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=1M --size=4G --readwrite=write --ramp_time=4 # Col 3
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=1M --size=4G --readwrite=read --ramp_time=4 # Col 1
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=1M --size=4G --readwrite=randwrite --ramp_time=4 # Col 7
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=1M --size=4G --readwrite=randread --ramp_time=4 # Col 5
# 4K
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=4K --size=4G --readwrite=write --ramp_time=4 # Col 4
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=4K --size=4G --readwrite=read --ramp_time=4 # Col 2
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=4k --size=4G --readwrite=randwrite --ramp_time=4 # Col 8
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=4k --size=4G --readwrite=randread --ramp_time=4 # Col 6 |
Table for comparision
Table for comparison with non-NAS devices
| Sequental | Random | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Read (MiB/s) | Write (MiB/s) | Read (MiB/s) | Write (MiB/s) | ||||||
| device | disk type | bs = 1M | bs = 4K | bs = 1M | bs = 4K | bs = 1M | bs = 4K | bs = 1M | bs = 4K |
| R100N | 1xNVMe | 1302 | 123 | 1047 | 104 | 1211 | 49.1 | Sequental | Random |
| Read (MiB/s) | Write (MiB/s) | Read (MiB/s) | Write (MiB/s) | ||||||
| device | disk type | bs = 1M | bs = 4K | bs = 1M | bs = 4K | bs = 1M | bs = 4K | bs = 1M | bs = 4K |
| R100N | 1xNVMe | 1302 | 123 | 1047 | 104 | 1211 | 49.1 | 1056 | 103 |
| R100S | 1xSSD | 405 | 57.4 | 347 | 49.9 | 321 | 20.0 | 341 | 48.5 |
| N200N | 1xNVMe | 1293 | 89.9 | 1212 | 202 | 1255 | 74.0 | 1152 | 193 |
| I1071N | 1xNVMe | 2720 | 278 | 2691 | 280 | 2462 | 75.0 | 2414 | 216 |
| I1355N | 1xNVMe | 1756 | 289 | 16.0 | 2.5 | 1348 | 57.4 | 15.9 | 2.6 |
| DS220+ | 2xHDD | 179 | 62.7 | 168 | 20.9 | 182 | 61.5 | 169 | 9.6 |
For disk performance comparison next system were use,
| code | Hardware | Disk | space used,% | Power On, hours | Life Time Remain, % |
|---|---|---|---|---|---|
| N150R | Intel N150, 12GB LPDDR5, Debian, TrueNAS 25.04 RC1, ZFS | Samsung SSD 990 PRO with Heatsink 4TB (4 in RAID) | 1 | 56 | 100 |
| R100N | Intel N100, 16GB DDR4, Ubuntu 24.04, CasaOS, ext4 | Movespeed YSSDHB-4TN7000 4TB NVMe | 8 | 4822 | 100 |
| R100S | Intel N100, 16GB DDR4, Ubuntu 24.04, CasaOS, ext4 | Transcend SSD230S 4 TB SATA SSD | 49 | 1522 | 100 |
| N200N | Intel N200, 32GB DDR5, Ubuntu 24.04, ext4 | Fanxiang S500PRO 2TB | 97 | 8343 | 100 |
| I1071N | Intel Core i7-10710U, 64GB DDR4, Ubuntu 24.0 | Lexar SSD ARES 4TB | 74 | 12238 | 99 |
| I1355N | Intel Core i7-1355U (15W), 96GB DDR5, Ubuntu 24.04 | XPG GAMMIX S70 BLADE 8TB | 85 | 6823 | 100 |
| DS220+ | Synology DS220+, J4025, 2GB, RAID1 | WD Gold Enterprise Class 18 TB 7200 RPM(2 in RAID1) | 77 | 37877 | N/A |
1 Year is 8766 hours (37877 hours = 4.3 Years)
Check Power Limits
...
| 85 | 6823 | 100 | |||
| DS220+ | Synology DS220+, J4025, 2GB, RAID1 | WD Gold Enterprise Class 18 TB 7200 RPM(2 in RAID1) | 77 | 37877 | N/A |
1 Year is 8766 hours (37877 hours = 4.3 Years)
Commands used to test disk speed
| Code Block |
|---|
cd /mnt/pool/shared/user
# each command executed 3 times
# 1M
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=1M --size=4G --readwrite=write --ramp_time=4 # Col 3
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=1M --size=4G --readwrite=read --ramp_time=4 # Col 1
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=1M --size=4G --readwrite=randwrite --ramp_time=4 # Col 7
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=1M --size=4G --readwrite=randread --ramp_time=4 # Col 5
# 4K
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=4K --size=4G --readwrite=write --ramp_time=4 # Col 4
fio --ioengine=libaio --direct=1 --name=test --filename=test0 --bs=4K --size=4G --readwrite=read --ramp_time=4 # Col 2
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=4k --size=4G --readwrite=randwrite --ramp_time=4 # Col 8
fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test1 --filename=test2 --bs=4k --size=4G --readwrite=randread --ramp_time=4 # Col 6 |
Check Power Limits
| Code Block |
|---|
root@truenas:/mnt/pool/shared/user# zfs get all pool turbostat 2>&1|grep compWatt poolRAPL: 43691 compressratiosec. Joule Counter Range, at 6 Watts cpu0: MSR_RAPL_POWER_UNIT: 0x000a0e03 1(0.61x125000 Watts, 0.000061 Joules, 0.000977 sec.) cpu0: PKG Limit #1: ENabled (15.000 Watts, 8.000000 sec, - pool compression lz4 local pool refcompressratio 1.00x -clamp ENabled) cpu0: PKG Limit #2: ENabled (15.000 Watts, 0.002441* sec, clamp DISabled) cpu0: PKG Limit #4: 78.000000 Watts (UNlocked) cpu0: DRAM Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled) cpu0: Cores Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled) cpu0: GFX Limit: DISabled (0.000 Watts, 0.000977 sec, clamp DISabled) |
RAIDZ1 (LZ4) commands output
...