Orange Pi 5 Plus - Slow NVME speeds

[OP5_PLUS]

Hi,

newbie here again....

 

I recently switched to Armbian from Orange Pi Ubuntu.

I am running Armbian 23.8.1 Jammy with Linux 5.10.160-legacy-rk35xx

 

My issue is a noticeable slowdown in NVME performance. I now get 522.35 MB/sec (as per hdparm test), but before on orange pi ubuntu it was around 1900 MB/s . 

 

FYI...Armbian is installed on and booting from the NVME. everything else seems to work perfect and I don't get smartctl errors.

 

Please advise

 

- Crucial P3 4TB PCIe M.2 2280 SSD NVMe (PCIe Gen 3 x4)

- Have all the latest updates

 

/dev/nvme0n1:

 Timing cached reads:   7988 MB in  2.00 seconds = 3997.60 MB/sec

 Timing buffered disk reads: 1568 MB in  3.00 seconds = 522.35 MB/sec

 

Thank you in advance for any help you can offer.

 

[Efe Çetin]

I am not able to reproduce it

[OP5_PLUS]

Thanks for the reply @Efe Çetin. 

 

I also forgot to mention that write speeds are at 1200. but read speeds are usually at ~500, but on a few occasions went down to ~90. NVME also works OK on intel system. 

 

The only difference I could think of is installing hardware acceleration and also using the eMMC storage module as a backup drive.

 

I just might start from scratch again and see if that makes a difference but I put a lot of time setting up all my apps (on Docker).

 

If there a driver or setting I can try that can fix the performance?

[OttawaHacker]

12 hours ago, OP5_PLUS said:

I also forgot to mention that write speeds are at 1200. but read speeds are usually at ~500, but on a few occasions went down to ~90. NVME also works OK on intel system. 

Have you plugged the hard drive on the USB3 port? From the docs, only one of the blue ports is USB3.

[OP5_PLUS]

@OttawaHacker Thanks.

 

But nothing plugged into the USB ports

[royk]

• Did you dd write the images?
• Check the temperature 
• Try adding this line to armbianEnv.txt

extraargs="pcie_aspm.policy=performance"

[OP5_PLUS]

@royk ...thanks for your post:

 

1. Yes, I did dd write the image (used the orange pi users manual instructions to install the Armbian image)

2. Temps are at 30C-40C (have heat spreader installed)

3. No performance change after I added extraargs="pcie_aspm.policy=performance" to armbianEnv.txt (tested after reboot)

 

I have a 4TB drive and armbian has the Host Memory Buffer Size set at (HSIZE): 8192. The Manufacturer, Crucial sets the MS windows driver to 64MB, could this be the issue?

 

Thanks again for your help

Edited September 10, 2023 by OP5_PLUS
clarification

[OP5_PLUS]

Lastest benchmarks:

 

Crucial P3 4TB PCIe M.2 2280 SSD NVMe (PCIe Gen 3 x4)

 

hdparm -t /dev/nvme0n1p2

   /dev/nvme0n1p2:

   Timing buffered disk reads: 1488 MB in  3.00 seconds = 495.97 MB/sec

 

 

Other:

HDParm                  Disk Read                       1070.58 MB/s             

HDParm                  Cached Disk Read         798.72 MB/s              

DD                            Disk Write                      546 MB/s                 

FIO                           4k random read            103434 IOPS (413737 KB/s)

FIO                           4k random write           35992 IOPS (143971 KB/s) 

IOZone                    4k read                           175062 KB/s              

IOZone                    4k write                          173754 KB/s              

IOZone                    4k random read            42104 KB/s               

IOZone                    4k random write           94175 KB/s 

[royk]

@OP5_PLUS Are you using the upper m.2 slot for WiFi/BT or something else? If so you could try without.

What does "lsblk -t" return?

 

What improved the speed a bit with my nvme is (but first check the current settings with lsblk -t):

echo 4096 | sudo tee /sys/block/nvme0n1/queue/read_ahead_kb

echo 1024 | sudo tee /sys/block/nvme0n1/queue/nr_requests

 

This should be better for NVME drives, but I don't see a difference

echo none | sudo tee /sys/block/sda/queue/scheduler

 

If it returns something like "Invalid argument" then try lower values like 2048, 1024, 512....

 

Another thing you could try is (especially when you wrote the image with dd):

sudo fstrim -v /

 

Setting the sector size to 4k seems to improve the performance by some people. Last time I tried that I wasn't able to boot from it and didn't try again (likely I did something wrong).

After doing this don't use dd to write the image, just create one ext4 partition and install with armbian-install. 

What worked for me to restore a backup was 'sudo cp -ra /location/backup/* /location/nvme'

https://www.bjonnh.net/article/20210721_nvme4k/

 

Edited September 10, 2023 by royk

[OP5_PLUS]

@royk

1. I am NOT using the upper m.2 slot for WiFi/BT.

2. Current sector size is 4k

3. Already tried fstrim -v /, and is configured on a weekly schedule.

 

4.lsblk -t result:

NAME         ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE  RA WSAME

nvme0n1              0    512      0     512     512    0 none     1023 128    0B

├─nvme0n1p1          0    512      0     512     512    0 none     1023 128    0B

└─nvme0n1p2          0    512      0     512     512    0 none     1023 128    0B

 

armbian-install did not work for me originally, did not boot. I will try again.

 

Thanks for the guidance. You're awsome !!!

[royk]

@OP5_PLUS If you've got the same original bootloader as the one I've got (displays an orange at power on when there is no bootable storage connected), it should boot. On my nvme there is one ext4 partition with the kernel/dtb in the folder /boot (no mount). If it doesn't boot then you could check if the UUID is set right in /boot/armbianEnv.txt and in /etc/fstab (you should not have a boot mount).

The UUID you'll see with the command "blkid" and might change after resizing the partition.

If it still doesn't boot than you probably need to install the bootloader to MTD with armbian-install, this takes a few minutes.

 

When you run "sudo nvme id-ns -H /dev/nvme0n1" you'll most likely see this message at the end (which indicates that the sector size is 512):

LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)                      

LBA Format  1 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0x1 Better 

 

It's not likely that it will improve that much after changing it to 4k but if you do you might want to reboot after running "sudo nvme format --lbaf=1 /dev/nvme0n1" to make sure the OS recognizes the 4k sectors correctly.

 

Another thing I could think of is that the PCIe linking isn't working  correctly. What do you see with: "dmesg | grep PCIe"

fe150000 should be the M.2 PCIe 3.0 port. I guess the first digit (2) indicates the link speed...

 

[    5.440728] rk-pcie fe150000.pcie: PCIe Linking... LTSSM is 0x3
[    5.489614] rk-pcie fe190000.pcie: PCIe Link up, LTSSM is 0x130011
[    5.489640] rk-pcie fe180000.pcie: PCIe Link up, LTSSM is 0x130011
[    5.497438] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.522945] rk-pcie fe150000.pcie: PCIe Link up, LTSSM is 0x230011
[    5.522972] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.549640] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.576306] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.602972] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.629640] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.656305] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.682975] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    5.709637] rk-pcie fe170000.pcie: PCIe Linking... LTSSM is 0x3
[    8.186316] rk-pcie fe170000.pcie: PCIe Link Fail

 

Running this command gives probably more usefull information (output to pcie.txt as the terminal buffer might be too small):

sudo lspci -vv | tee pcie.txt

0000:01:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD (rev 01) (prog-if 02 [NVM Express])
        Subsystem: KIOXIA Corporation Toshiba RC500 NVMe SSD 500GB

LnkSta: Speed 8GT/s (ok), Width x4 (ok)

 

As last, although I understand you want the max performance out of it, it might be possible that it's just a weak spot of your nvme but still works fast (enough) on regular daily tasks. I didn't notice much of a difference with my OPi5 with PCIe 2.0 (about the same results as you've got)

 

Edited September 12, 2023 by royk