NVMe SSD boot with the Raspberry Pi 5

From Geekworm Wiki
Revision as of 11:51, 26 December 2023 by Lisa (talk | contribs) (→‎FAQ)
Jump to navigation Jump to search

Most of the content of this tutorial comes from Jeff's blog: NVMe SSD boot with the Raspberry Pi 5, Thanks fo Jeff

Enable PCIe

By default the PCIe connector is not enabled. To enable it you should add the following option into /boot/firmware/config.txt and reboot:

sudo nano /boot/config.txt

Then add the following comment;

# Enable the PCIe External connector.
dtparam=pciex1
# This line is an alias for above (you can use either/or to enable the port).
dtparam=nvme

Press Ctrl-O, then enter, to write the change to the file.

Press Ctrl-X to exit nano (the editor).


And the connection is certified for Gen 2.0 speed (5 GT/sec), but you can force it to Gen 3.0 (10 GT/sec) if you add the following line after:

dtparam=pciex1_gen=3

However, we DON'T recommend you to use Gen 3.0 as it is not officially supported and may not work stably.

Then DON'T forgot to reboot

sudo reboot

You also can refer to official documentatio:Enabling PCIe

Flash OS onto NVME SSD

To get the NVMe SSD to boot your Pi, it needs to have an OS, so the Raspberry Pi OS needs to be flashed onto NVME SSDs, this is very important!


Only support Raspberry Pi OS (Bookworm) version, Raspberry Pi OS bullseye is NOT supported, rfer to https://www.raspberrypi.com/software/operating-systems/

Here are a few ways to flash the OS to an NVME SSD:


1. Use SD Card Copier tool to flash OS onto the NVME SSD On Raspberry Pi OS(recommended)

This method works if you have an SD card and have booted the device successfully

Cick Applications =>Accessories =>SD Card Copier on the main screen, run the SD Card Copier program, and copy the OS to the NVME ssd as shown in the figure below.

PI5-sd-copier.png

Click Start to run. Then shut down, unplug the SD card, and restart the device.

This is a visual operation and we highly recommend it!


2. Flash the SSD with Raspberry Pi Imager

You can also directly use the Raspberry Pi Imager tool on a MAC computer or windows computer to flash a fresh Pi OS to NVME SSD, but you will need an additional USB to nvme adapter.

  1. Install Pi Imager and open it
  2. Plug your NVMe SSD into your computer using a USB to NVMe adapter
  3. Choose an OS to install
  4. Choose the drive (connected through your adapter) to flash
  5. Click write (and set any options you'd like)

Once flashing is complete, pull the NVMe drive, attach it to your Pi 5, and it should boot off it (with or without a microSD card inserted)—assuming you have the bootloader up to date and set the BOOT_ORDER appropriately!


NOTE:

If you are flashing a fresh Pi OS to NVME ssd, you must #Enable PCIe; if you are COPY or CLONE an old Pi OS from SD card to NVME ssd, and you have already enabled pcie in the old Pi OS in advance, then you don't need to do enable pcie again!

Set NVMe early in the boot order

The PCIe connection should work after a reboot, but your Pi won't try booting off an NVMe SSD yet. For that, you need to change the BOOT_ORDER in the Raspberry Pi's bootloader configuration:

Run the following command to edit the EEPROM on the Raspberry Pi 5.

sudo rpi-eeprom-config --edit

Then change the BOOT_ORDER line to the following:

BOOT_ORDER=0xf416

- Press Ctrl-O, then enter, to write the change to the file.

- Press Ctrl-X to exit nano (the editor).

Read Raspberry Pi's documentation on BOOT_ORDER for all the details. For now, the pertinent bit is the 6 at the end: that is what tells the Pi to attempt NVMe boot first!

Reboot your Raspberry Pi 5 to make the change take effect.

How to update eeprom firmware

NOTE: If you find that there is no WIFI or ssd is not recognized after restarting, you can try this part of the update, otherwise please ignore it.

After modifying the BOOT_ORDER for NVME SSD, reboot the PI5 and you maybe get the NO WIFI or other error. Try to upgrade the eeprom to fix it (Raspberry Pi Foundation official may upgrade the OS to solve this kind of problem, please pay attention to the update of Raspberry Pi OS, if the OS upgrade can solve this problem, you can ignore this step).

Run the following command to

wget https://github.com/timg236/rpi-eeprom/raw/pieeprom-2023-10-30-2712/firmware-2712/default/pieeprom-2023-10-30.bin 
sudo rpi-eeprom-config --edit pieeprom-2023-10-30.bin

Update-eeprom.png


It is recommended to use the latest version of the pieeprom-xxxx-xx-xx.bin file, please visit this link to get the latest version:

PS: After our testing, we found that only the firmware piepprom-2023-10-30.bin has solved the NO WIFI problem for the time being.

FAQ

Q: Can't boot/recognize from NVMe SSD?

A: Here are a few steps to help you troubleshoot some issues:

  1. Make sure all hardware connections are properly connected and coupled.
  2. Only Raspberry Pi OS (Bookworm) supports PCIe, Raspberry Pi OS bullseye is NOT supported, refer to https://www.raspberrypi.com/software/operating-systems/
  3. Although you can also boot from a SD card, Pi OS is required for the NVMe SSD to boot, DON'T forget Flash OS onto NVMe SSD.
  4. If you're flashing a fresh Pi OS , DON'T forget to ENABLE PCIe.
  5. Run the lspci or lsblk command to check the NVMe SSD is recognized.
  6. Update the firmware, refer to #How to update eeprom firmware
  7. Contact us at email: support@geekworm.com
  8. Refer to the issue solved by customer: https://geekworm.com/community/forum/topic/120764/geekworm-x1001-for-raspberry-pi-5

Return to X1001 / X1000

Add your comment
Geekworm Wiki welcomes all comments. If you do not want to be anonymous, register or log in. It is free.


Anonymous user #11

25 days ago
Score 0++

Bought X1002 & WD Green SN350 2TB

Works just fine without any configuration. The only configuration change to enable PCI_gen3:

  1. PCIe Gen3

dtparam=pciex1_gen=3

within:

sudo nano /boot/firmware/config.txt

  1. PCIe Gen3

dtparam=pciex1_gen=3


$ sudo lspci 0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21) 0000:01:00.0 Non-Volatile memory controller: Sandisk Corp WD PC SN540 / Green SN350 NVMe SSD 1 TB (DRAM-less) 0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21) 0001:01:00.0 Ethernet controller: Device 1de4:0001


$ duf ╭───────────────────────────────────────────────────────────────────────────────────────────────────╮ │ 2 local devices │ ├────────────────┬────────┬────────┬────────┬───────────────────────────────┬──────┬────────────────┤ │ MOUNTED ON │ SIZE │ USED │ AVAIL │ USE% │ TYPE │ FILESYSTEM │ ├────────────────┼────────┼────────┼────────┼───────────────────────────────┼──────┼────────────────┤ │ / │ 1.8T │ 789.3G │ 949.7G │ [########............] 43.1% │ ext4 │ /dev/nvme0n1p2 │ │ /boot/firmware │ 511.0M │ 75.3M │ 435.6M │ [##..................] 14.7% │ vfat │ /dev/nvme0n1p1 │

╰────────────────┴────────┴────────┴────────┴───────────────────────────────┴──────┴────────────────╯

Anonymous user #8

one month ago
Score 0++
Just bought an x1001 and paired it with a 1T Samsung 980 Pro. Installed with no problems other than having to reposition the PCIe cable once. Love the performance!

Anonymous user #6

2 months ago
Score 0++

[SOLVED] Errors booting from NVME with X1002 base and Crucial T500 Gen4 NVMe.

Symptoms: Either: 1. *Extremely long* boot times (~240 seconds) with pcieport and nvme errors in dmesg including:

[ 4.230345] pcieport 0000:00:00.0: AER: Corrected error received: 0000:00:00.0 [ 4.230359] pcieport 0000:00:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID) [ 4.230362] pcieport 0000:00:00.0: device [14e4:2712] error status/mask=00001000/00002000 [ 4.230366] pcieport 0000:00:00.0: [12] Timeout

...and...

[ 41.971850] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 41.971859] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 41.971862] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 42.003987] nvme0n1: I/O Cmd(0x2) @ LBA 6559080, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 42.003993] I/O error, dev nvme0n1, sector 6559080 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 [ 42.019855] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 42.050334] nvme nvme0: 4/0/0 default/read/poll queues [ 72.331604] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 72.331612] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 72.331615] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 72.364112] nvme0n1: I/O Cmd(0x2) @ LBA 101390768, 16 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364119] I/O error, dev nvme0n1, sector 101390768 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 72.364131] nvme0n1: I/O Cmd(0x2) @ LBA 1328360, 8 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364135] I/O error, dev nvme0n1, sector 1328360 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 72.364143] nvme0n1: I/O Cmd(0x2) @ LBA 1327704, 8 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364146] I/O error, dev nvme0n1, sector 1327704 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 72.364156] nvme0n1: I/O Cmd(0x2) @ LBA 5690072, 8 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364159] I/O error, dev nvme0n1, sector 5690072 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 72.364172] nvme0n1: I/O Cmd(0x2) @ LBA 7574968, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364175] I/O error, dev nvme0n1, sector 7574968 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 72.364183] nvme0n1: I/O Cmd(0x2) @ LBA 7629088, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364186] I/O error, dev nvme0n1, sector 7629088 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 [ 72.364280] nvme0n1: I/O Cmd(0x2) @ LBA 17834496, 88 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364283] I/O error, dev nvme0n1, sector 17834496 op 0x0:(READ) flags 0x80700 phys_seg 11 prio class 2 [ 72.364297] nvme0n1: I/O Cmd(0x2) @ LBA 17834592, 168 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364300] I/O error, dev nvme0n1, sector 17834592 op 0x0:(READ) flags 0x80700 phys_seg 21 prio class 2 [ 72.380221] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 72.410751] nvme nvme0: 4/0/0 default/read/poll queues [ 103.652649] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 103.652658] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 103.652661] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 103.684774] nvme0n1: I/O Cmd(0x2) @ LBA 7566288, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 103.684782] I/O error, dev nvme0n1, sector 7566288 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 [ 103.684802] nvme0n1: I/O Cmd(0x2) @ LBA 8615992, 56 blocks, I/O Error (sct 0x3 / sc 0x71) [ 103.684805] I/O error, dev nvme0n1, sector 8615992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 103.684813] nvme0n1: I/O Cmd(0x2) @ LBA 4027408, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 103.684816] I/O error, dev nvme0n1, sector 4027408 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 103.684827] nvme0n1: I/O Cmd(0x2) @ LBA 1333928, 8 blocks, I/O Error (sct 0x3 / sc 0x71) [ 103.684830] I/O error, dev nvme0n1, sector 1333928 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 103.700653] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 103.729101] nvme nvme0: 4/0/0 default/read/poll queues [ 136.100697] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 136.100706] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 136.100709] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 136.152704] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 136.183196] nvme nvme0: 4/0/0 default/read/poll queues [ 170.724717] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 170.724726] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 170.724729] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 170.760849] nvme0n1: I/O Cmd(0x2) @ LBA 1694616, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 170.760856] I/O error, dev nvme0n1, sector 1694616 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 [ 170.780730] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 170.811175] nvme nvme0: 4/0/0 default/read/poll queues [ 206.051720] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 206.051728] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 206.051731] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 206.099734] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 206.130254] nvme nvme0: 4/0/0 default/read/poll queues [ 228.538013] ieee80211 phy0: brcmf_p2p_send_action_frame: Unknown Frame: category 0xa, action 0x8 [ 242.403171] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 242.403180] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 242.403183] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 242.435299] nvme0n1: I/O Cmd(0x2) @ LBA 1993800, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 242.435306] I/O error, dev nvme0n1, sector 1993800 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 [ 242.451184] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 242.481653] nvme nvme0: 4/0/0 default/read/poll queues

2. Long, but better boot times (~35 sec), still with pcieport and nvme errors as above, but fewer of them.

Hardware: Raspberry Pi 5 with X1002 bottom hat SSD: Crucial T500 PCIe Gen 4 NVMe M.2 SSD 1TB

Firmware: eeprom version: updated to Feb 5 2024 (1707143914) while troubleshooting. Unsure if earlier version would have worked.

Updates made to enable boot from NVME: Changes in eeprom via rpi-eeprom-config -e: BOOT_ORDER=0xf416 (added the 6) and added PCIE_PROBE=1. Also changed POWER_OFF_ON_HALT=1, but that was just to reduce power consumption and had no effect on speedy or error-free booting.

Changes to /boot/firmware/config.txt Added line "dtparam=pciex1" to the bottom of the file. Note: it works equally well using the alias "dtparam=nvme" as instead. Added line "dtparam=pciex1_gen3" to the bottom of the file. <-- ***THIS IS WHAT FINALLY SOLVED THE DMESG ERRORS AND SLOW BOOT TIMES*** in spite of this Geekworm guide stating the setting was not recommended and may lead to instability. In my case, it was the only setting to result in stable operation.

Other things that didn't work / had no effect: Added "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" to /boot/firmware/cmdline.txt

Added PSU_MAX_CURRENT=5000 to eeprom via rpi-eeprom-config -e

Anonymous user #9

one month ago
Score 0++

spend whole day on the same issue. having Crucial P3 Plus 500GB ssd : Model Number: CT500P3PSSD8 Firmware Version: P9CR411

thank you man for sharing, you saved my live :)

Anonymous user #10

one month ago
Score 0++
What is a good boot time booting from NVMe?

Anonymous user #5

2 months ago
Score 0++
How do you change the nut from the 2242 position to the 2230 position?

Anonymous user #7

2 months ago
Score 0++
I used a hot air station to unsolder and solder it into place

Anonymous user #4

2 months ago
Score 0++

hi from vienna,

i have no luck till now withj the x1002:

  • first i had a WD black 770 and read that this is on the incompatible list
  • returned it and ordered a crucial t500 1TB nvme ssd but still there is NO reaction (no blue light, nothing reported in lspci,..)

looks like the x1002 is not found / connected / or ist just plain dead

  • did the edits as described above in boot/config.txt
  • no success

any advice please

cheerz

Anonymous user #4

2 months ago
Score 0++
oha ... flushed a new eeprom and VOILA now it works :-)

Anonymous user #3

2 months ago
Score 0++

While following the instructions I noticed the following messages in dmesg: pcieport 0000:00:00.0: AER: Corrected error received: 0000:00:00.0 pcieport 0000:00:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID) pcieport 0000:00:00.0: device [14e4:2712] error status/mask=00001000/00002000 pcieport 0000:00:00.0: [12] Timeout

I am running the latest eeprom CURRENT: Fri 5 Jan 15:57:40 UTC 2024 (1704470260) LATEST: Fri 5 Jan 15:57:40 UTC 2024 (1704470260)

The device itself boots just fine.

Anonymous user #2

3 months ago
Score 0++
If the nvme is not recognised after adding dtparam=pciex1 to config.txt, add PCIE_PROBE=1 to the eeprom. That made my pi see the nvme.

Harry

2 months ago
Score 0++

Thanks.

Run command:

sudo rpi-eeprom-config --edit
Add PCIE_PROBE=1 line, then press CTRL+O to write change to file, press CTRL+X to exit, then reboot

Anonymous user #1

3 months ago
Score 0++

Not listed here but should be - you should also add the following line under the BOOT_ORDER line when configuring the eeprom:

PCIE_PROBE=1

My Raspberry Pi 5 would see the NVMe SSD, but would not boot from it without this line. Using an SK Hynix PC601 256GB NVMe SSD

Walker

3 months ago
Score 0++
Thanks a lot for the advice.