NVMe SSD boot with the Raspberry Pi 5

From Geekworm Wiki
Jump to navigation Jump to search

Most of the content of this tutorial comes from Jeff's blog: NVMe SSD boot with the Raspberry Pi 5, Thanks fo Jeff

Important Notes

NVMe SSD Incompatibility List:

We recommend avoiding the following NVMe SSD drives which is equipped with a Phison controller due to their proven incompatibility:

These specific models have demonstrated compatibility issues, and it is advisable to avoid them when considering NVMe SSD options for the X10xx series NVMe shield. You can run "lspci" command to check the controller brand of the SSD.

We hope the Raspberry Pi Foundation will resolve this issue soon.

Also note:

  • Compatible with M.2 NVMe SSDs
  • Not compatible with M.2 SATA SSDs, M.2 PCIe AHCI SSDs, or other M.2 non-NVMe devices
  • Older NVMe drives with less efficient flash media may not perform as well as newer drives
  • New NVMe SSDs are not partitioned and will need to be both partitioned and formatted when first connected to the Raspberry Pi before they will be accessed in the Explorer.
  • We get feedback from customers that Polaris Controller will also have compatibility problems. Please replace the other SSD test if it not work, whether it is compatible with the Raspberry Pi 5 does not depend on the X100X series boards


Enable PCIe

By default the PCIe connector is not enabled. To enable it you should add the following option into /boot/firmware/config.txt and reboot:

sudo nano /boot/firmware/config.txt

Then add the following comment;

# Enable the PCIe External connector.
dtparam=pciex1
# This line is an alias for above (you can use either/or to enable the port).
dtparam=nvme

Press Ctrl-O, then enter, to write the change to the file.

Press Ctrl-X to exit nano (the editor).


And the connection is certified for Gen 2.0 speed (5 GT/sec), but you can force it to Gen 3.0 (10 GT/sec) if you add the following line after:

dtparam=pciex1_gen=3

WARNING

The Raspberry Pi 5 is not certified for Gen 3.0 speeds, and connections to PCIe devices at these speeds may be unstable.

Then DON'T forgot to reboot

sudo reboot

You also can refer to official documentatio:Enabling PCIe

Use the lspci command to display your PCIe devices

 sudo lspci

The output is as follows, note that the content of the third line depends on the NVME control you are using.

pi@raspberrypi: ~ $ sudo lspci
0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0000:01:00.0 Non-Volatile memory controller: Silicon Motion, Inc. SM2263EN/SM2263XT SSD Controller (rev 03)
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0001:01:00.0 Ethernet controller: Device 1de4:0001

If your NVMe SSD is not recognized, updating the bootloader firmware is essential!You can either use a terminal, and simply use the following commands to update your firmware: (Firstly the Raspberry Pi 5 needs to be connected to the internet)

sudo apt update
sudo apt upgrade
rpi-eeprom-update -a

If a firmware update is available you will be prompted to install it and your Raspberry Pi will reboot. More information about updating the firmware on your Raspberry Pi is available in the official documentation.

After updating the firmware and your SSD is still recognized, Please consider switching to a different brand or model and try again.

Then run the following command to check the BOOTLOADER version:

vcgencmd bootloader_version

Maybe the result is following:

pi@raspberrypi:~ $ vcgencmd bootloader_version
2024/02/16 15:28:41
version 4c845bd37c8a7c7ff79173cdc50dd3facf63996f (release)
timestamp 1708097321
update-time 1709085188
capabilities 0x0000007f

'2024/02/16' is the firmware version;

Flash OS onto NVME SSD

To get the NVMe SSD to boot your Pi, it needs to have an OS, so the Raspberry Pi OS needs to be flashed onto NVME SSDs, this is very important!


Only support Raspberry Pi OS (Bookworm) version, Raspberry Pi OS bullseye or Ubuntu or Home Assistant OS is NOT supported, refer to https://www.raspberrypi.com/software/operating-systems/

Here are a few ways to flash the OS to an NVME SSD:


1. Use SD Card Copier tool to flash OS onto the NVME SSD On Raspberry Pi OS(recommended)

This method works if you have an SD card and have booted the device successfully

Cick Applications =>Accessories =>SD Card Copier on the main screen, run the SD Card Copier program, and copy the OS to the NVME ssd as shown in the figure below.

PI5-sd-copier.png

Click Start to run. Then shut down, unplug the SD card, and restart the device.

This is a visual operation and we highly recommend it!


2. Flash the SSD with Raspberry Pi Imager

You can also directly use the Raspberry Pi Imager tool on a MAC computer or windows computer to flash a fresh Pi OS to NVME SSD, but you will need an additional USB to nvme adapter.

  1. Install Pi Imager and open it
  2. Plug your NVMe SSD into your computer using a USB to NVMe adapter
  3. Choose an OS to install
  4. Choose the drive (connected through your adapter) to flash
  5. Click write (and set any options you'd like)
  6. Once you have finished flashing the OS, DON'T remove the nvme SSD. You need to change the config.txt in the root directory of the NVME SSD drive to enable pcie (edit config.txt and add 'dtparam=pciex1' on the end of file), this is very important and too many people forget this.

Then pull the NVMe drive, attach it to your Pi 5, and it should boot off it (with or without a microSD card inserted)—assuming you have the bootloader up to date and set the BOOT_ORDER appropriately!

NOTE:

If you are flashing a fresh Pi OS to NVME ssd, you must #Enable PCIe; if you are COPY or CLONE an old Pi OS from SD card to NVME ssd, and you have already enabled pcie in the old Pi OS in advance, then you don't need to do enable pcie again!

Set NVMe early in the boot order

The PCIe connection should work after a reboot, but your Pi won't try booting off an NVMe SSD yet. For that, you need to change the BOOT_ORDER in the Raspberry Pi's bootloader configuration:

Run the following command to edit the EEPROM on the Raspberry Pi 5.

sudo rpi-eeprom-config --edit

Then change the BOOT_ORDER line to the following:

BOOT_ORDER=0xf416

- Press Ctrl-O, then enter, to write the change to the file.

- Press Ctrl-X to exit nano (the editor).

Read Raspberry Pi's documentation on BOOT_ORDER for all the details. For now, the pertinent bit is the 6 at the end: that is what tells the Pi to attempt NVMe boot first!

Reboot your Raspberry Pi 5 to make the change take effect.

Decrease wattage when turned off

By default, the Raspberry Pi 5 consumes around 1W to 1.4W of power when turned off. This can be decreased by manually editing the EEPROM configuration with sudo rpi-eeprom-config -e. Change the settings to the following:

BOOT_UART=1
POWER_OFF_ON_HALT=1
BOOT_ORDER=0xf416

Then reboot the device with sudo reboot, this should drop the power consumption when powered down to around 0.01W.

FAQ

Q1: Can't boot/recognize from NVMe SSD?

A: Here are a few steps to help you troubleshoot some issues:

  1. Firstly, you can read this post to get some help;
  2. Make sure the SSD type you use is correct, only support M.2 NVME (Sata) Key-M SSD NOT support M.2 NGFF Key-B SSD. Refer to the following diagram (click it to zoom out): Supported-SSD-1.jpg
  3. Check if the FFC is fully inserted. Is the HAT tightly connected? We recommend you to send us the FFC connection schematic as well.
  4. Only Raspberry Pi OS (Bookworm) supports PCIe, Raspberry Pi OS bullseye is NOT supported, refer to https://www.raspberrypi.com/software/operating-systems/
  5. Although you can also boot from a SD card, Pi OS is required for the NVMe SSD to boot, DON'T forget Flash OS onto NVMe SSD.
  6. If you're flashing a fresh Pi OS , DON'T forget to ENABLE PCIe.
  7. Run the lspci andlsblk command to check the NVMe SSD is recognized.
  8. Add PCIE_PROBE=1 to the eeprom, (Run sudo rpi-eeprom-config --edit command, then add this line, then press CTRL+O to write change to file, press CTRL+X to exit, then reboot)
  9. Update the bootloader firmware to the last verson, refer to How to update eeprom firmware, this is very important!!!And finally be sure to run the command vcgencmd bootloader_version to check if your upgrade was successful!
  10. For X1003, if an error occurs, replace the FFC cable to test.
  11. If your SSD is old and has a DOS partition, delete the DOS partition and repartition the SSD (feedback from buyer)
  12. Contact us at email: support@geekworm.com


Q2: Don't recognize / boot from NVME SSD?

A: If you have enabled pcie and find that you still cannot boot from NVME SSD, please perform the following actions

1. Follow the picture below to check the version of X1001 or X1000

X1001-V1.1-IMG-6871-old-new.jpg

2. If it is an Old Version, refer to How to update eeprom firmware to update the firmware with pieeprom-2023-10-30.bin file.

3. If it is an New Version, refer to How to update eeprom firmware to update the firmware to lasted version.

4. If you still have not solved the problem, please refer to the next FAQ:Q1


Return to X1000 / X1001 / X1002 / X1003

Add your comment
Geekworm Wiki welcomes all comments. If you do not want to be anonymous, register or log in. It is free.


Anonymous user #11

10 days ago
Score 0++

Bought X1002 & WD Green SN350 2TB

Works just fine without any configuration. The only configuration change to enable PCI_gen3:

  1. PCIe Gen3

dtparam=pciex1_gen=3

within:

sudo nano /boot/firmware/config.txt

  1. PCIe Gen3

dtparam=pciex1_gen=3


$ sudo lspci 0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21) 0000:01:00.0 Non-Volatile memory controller: Sandisk Corp WD PC SN540 / Green SN350 NVMe SSD 1 TB (DRAM-less) 0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21) 0001:01:00.0 Ethernet controller: Device 1de4:0001


$ duf ╭───────────────────────────────────────────────────────────────────────────────────────────────────╮ │ 2 local devices │ ├────────────────┬────────┬────────┬────────┬───────────────────────────────┬──────┬────────────────┤ │ MOUNTED ON │ SIZE │ USED │ AVAIL │ USE% │ TYPE │ FILESYSTEM │ ├────────────────┼────────┼────────┼────────┼───────────────────────────────┼──────┼────────────────┤ │ / │ 1.8T │ 789.3G │ 949.7G │ [########............] 43.1% │ ext4 │ /dev/nvme0n1p2 │ │ /boot/firmware │ 511.0M │ 75.3M │ 435.6M │ [##..................] 14.7% │ vfat │ /dev/nvme0n1p1 │

╰────────────────┴────────┴────────┴────────┴───────────────────────────────┴──────┴────────────────╯

Anonymous user #8

one month ago
Score 0++
Just bought an x1001 and paired it with a 1T Samsung 980 Pro. Installed with no problems other than having to reposition the PCIe cable once. Love the performance!

Anonymous user #6

one month ago
Score 0++

[SOLVED] Errors booting from NVME with X1002 base and Crucial T500 Gen4 NVMe.

Symptoms: Either: 1. *Extremely long* boot times (~240 seconds) with pcieport and nvme errors in dmesg including:

[ 4.230345] pcieport 0000:00:00.0: AER: Corrected error received: 0000:00:00.0 [ 4.230359] pcieport 0000:00:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID) [ 4.230362] pcieport 0000:00:00.0: device [14e4:2712] error status/mask=00001000/00002000 [ 4.230366] pcieport 0000:00:00.0: [12] Timeout

...and...

[ 41.971850] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 41.971859] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 41.971862] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 42.003987] nvme0n1: I/O Cmd(0x2) @ LBA 6559080, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 42.003993] I/O error, dev nvme0n1, sector 6559080 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 [ 42.019855] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 42.050334] nvme nvme0: 4/0/0 default/read/poll queues [ 72.331604] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 72.331612] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 72.331615] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 72.364112] nvme0n1: I/O Cmd(0x2) @ LBA 101390768, 16 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364119] I/O error, dev nvme0n1, sector 101390768 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 72.364131] nvme0n1: I/O Cmd(0x2) @ LBA 1328360, 8 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364135] I/O error, dev nvme0n1, sector 1328360 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 72.364143] nvme0n1: I/O Cmd(0x2) @ LBA 1327704, 8 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364146] I/O error, dev nvme0n1, sector 1327704 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 72.364156] nvme0n1: I/O Cmd(0x2) @ LBA 5690072, 8 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364159] I/O error, dev nvme0n1, sector 5690072 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 72.364172] nvme0n1: I/O Cmd(0x2) @ LBA 7574968, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364175] I/O error, dev nvme0n1, sector 7574968 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 72.364183] nvme0n1: I/O Cmd(0x2) @ LBA 7629088, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364186] I/O error, dev nvme0n1, sector 7629088 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 [ 72.364280] nvme0n1: I/O Cmd(0x2) @ LBA 17834496, 88 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364283] I/O error, dev nvme0n1, sector 17834496 op 0x0:(READ) flags 0x80700 phys_seg 11 prio class 2 [ 72.364297] nvme0n1: I/O Cmd(0x2) @ LBA 17834592, 168 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364300] I/O error, dev nvme0n1, sector 17834592 op 0x0:(READ) flags 0x80700 phys_seg 21 prio class 2 [ 72.380221] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 72.410751] nvme nvme0: 4/0/0 default/read/poll queues [ 103.652649] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 103.652658] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 103.652661] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 103.684774] nvme0n1: I/O Cmd(0x2) @ LBA 7566288, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 103.684782] I/O error, dev nvme0n1, sector 7566288 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 [ 103.684802] nvme0n1: I/O Cmd(0x2) @ LBA 8615992, 56 blocks, I/O Error (sct 0x3 / sc 0x71) [ 103.684805] I/O error, dev nvme0n1, sector 8615992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 103.684813] nvme0n1: I/O Cmd(0x2) @ LBA 4027408, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 103.684816] I/O error, dev nvme0n1, sector 4027408 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 103.684827] nvme0n1: I/O Cmd(0x2) @ LBA 1333928, 8 blocks, I/O Error (sct 0x3 / sc 0x71) [ 103.684830] I/O error, dev nvme0n1, sector 1333928 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 103.700653] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 103.729101] nvme nvme0: 4/0/0 default/read/poll queues [ 136.100697] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 136.100706] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 136.100709] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 136.152704] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 136.183196] nvme nvme0: 4/0/0 default/read/poll queues [ 170.724717] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 170.724726] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 170.724729] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 170.760849] nvme0n1: I/O Cmd(0x2) @ LBA 1694616, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 170.760856] I/O error, dev nvme0n1, sector 1694616 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 [ 170.780730] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 170.811175] nvme nvme0: 4/0/0 default/read/poll queues [ 206.051720] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 206.051728] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 206.051731] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 206.099734] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 206.130254] nvme nvme0: 4/0/0 default/read/poll queues [ 228.538013] ieee80211 phy0: brcmf_p2p_send_action_frame: Unknown Frame: category 0xa, action 0x8 [ 242.403171] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 242.403180] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 242.403183] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 242.435299] nvme0n1: I/O Cmd(0x2) @ LBA 1993800, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 242.435306] I/O error, dev nvme0n1, sector 1993800 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 [ 242.451184] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 242.481653] nvme nvme0: 4/0/0 default/read/poll queues

2. Long, but better boot times (~35 sec), still with pcieport and nvme errors as above, but fewer of them.

Hardware: Raspberry Pi 5 with X1002 bottom hat SSD: Crucial T500 PCIe Gen 4 NVMe M.2 SSD 1TB

Firmware: eeprom version: updated to Feb 5 2024 (1707143914) while troubleshooting. Unsure if earlier version would have worked.

Updates made to enable boot from NVME: Changes in eeprom via rpi-eeprom-config -e: BOOT_ORDER=0xf416 (added the 6) and added PCIE_PROBE=1. Also changed POWER_OFF_ON_HALT=1, but that was just to reduce power consumption and had no effect on speedy or error-free booting.

Changes to /boot/firmware/config.txt Added line "dtparam=pciex1" to the bottom of the file. Note: it works equally well using the alias "dtparam=nvme" as instead. Added line "dtparam=pciex1_gen3" to the bottom of the file. <-- ***THIS IS WHAT FINALLY SOLVED THE DMESG ERRORS AND SLOW BOOT TIMES*** in spite of this Geekworm guide stating the setting was not recommended and may lead to instability. In my case, it was the only setting to result in stable operation.

Other things that didn't work / had no effect: Added "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" to /boot/firmware/cmdline.txt

Added PSU_MAX_CURRENT=5000 to eeprom via rpi-eeprom-config -e

Anonymous user #9

one month ago
Score 0++

spend whole day on the same issue. having Crucial P3 Plus 500GB ssd : Model Number: CT500P3PSSD8 Firmware Version: P9CR411

thank you man for sharing, you saved my live :)

Anonymous user #10

24 days ago
Score 0++
What is a good boot time booting from NVMe?

Anonymous user #5

2 months ago
Score 0++
How do you change the nut from the 2242 position to the 2230 position?

Anonymous user #7

one month ago
Score 0++
I used a hot air station to unsolder and solder it into place

Anonymous user #4

2 months ago
Score 0++

hi from vienna,

i have no luck till now withj the x1002:

  • first i had a WD black 770 and read that this is on the incompatible list
  • returned it and ordered a crucial t500 1TB nvme ssd but still there is NO reaction (no blue light, nothing reported in lspci,..)

looks like the x1002 is not found / connected / or ist just plain dead

  • did the edits as described above in boot/config.txt
  • no success

any advice please

cheerz

Anonymous user #4

2 months ago
Score 0++
oha ... flushed a new eeprom and VOILA now it works :-)

Anonymous user #3

2 months ago
Score 0++

While following the instructions I noticed the following messages in dmesg: pcieport 0000:00:00.0: AER: Corrected error received: 0000:00:00.0 pcieport 0000:00:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID) pcieport 0000:00:00.0: device [14e4:2712] error status/mask=00001000/00002000 pcieport 0000:00:00.0: [12] Timeout

I am running the latest eeprom CURRENT: Fri 5 Jan 15:57:40 UTC 2024 (1704470260) LATEST: Fri 5 Jan 15:57:40 UTC 2024 (1704470260)

The device itself boots just fine.

Anonymous user #2

2 months ago
Score 0++
If the nvme is not recognised after adding dtparam=pciex1 to config.txt, add PCIE_PROBE=1 to the eeprom. That made my pi see the nvme.

Harry

2 months ago
Score 0++

Thanks.

Run command:

sudo rpi-eeprom-config --edit
Add PCIE_PROBE=1 line, then press CTRL+O to write change to file, press CTRL+X to exit, then reboot

Anonymous user #1

2 months ago
Score 0++

Not listed here but should be - you should also add the following line under the BOOT_ORDER line when configuring the eeprom:

PCIE_PROBE=1

My Raspberry Pi 5 would see the NVMe SSD, but would not boot from it without this line. Using an SK Hynix PC601 256GB NVMe SSD

Walker

2 months ago
Score 0++
Thanks a lot for the advice.