Open main menu

Most of the content of this tutorial comes from Jeff's blog: NVMe SSD boot with the Raspberry Pi 5, Thanks fo Jeff

Enabling PCIe

By default the PCIe connector is not enabled. To enable it you should add the following option into /boot/firmware/config.txt and reboot:

sudo nano /boot/config.txt

Then add the following comment;

# Enable the PCIe External connector.
dtparam=pciex1
# This line is an alias for above (you can use either/or to enable the port).
dtparam=nvme

Press Ctrl-O, then enter, to write the change to the file.

Press Ctrl-X to exit nano (the editor).


And the connection is certified for Gen 2.0 speed (5 GT/sec), but you can force it to Gen 3.0 (10 GT/sec) if you add the following line after:

dtparam=pciex1_gen=3

You also can refer to official documentatio:Enabling PCIe

Then DON'T forgot to reboot

sudo reboot

Set NVMe early in the boot order

The PCIe connection should work after a reboot, but your Pi won't try booting off an NVMe SSD yet. For that, you need to change the BOOT_ORDER in the Raspberry Pi's bootloader configuration:

Run the following command to edit the EEPROM on the Raspberry Pi 5.

sudo rpi-eeprom-config --edit

Then change the BOOT_ORDER line to the following:

BOOT_ORDER=0xf416

- Press Ctrl-O, then enter, to write the change to the file.

- Press Ctrl-X to exit nano (the editor).

Read Raspberry Pi's documentation on BOOT_ORDER for all the details. For now, the pertinent bit is the 6 at the end: that is what tells the Pi to attempt NVMe boot first!

Reboot your Raspberry Pi 5 to make the change take effect.


NVMe boot won't work unless you have the external PCI Express port enabled, and there's a working NVMe drive with a valid boot partition! If you don't have that (e.g. you used Raspberry Pi Imager with an external USB NVMe adapter to flash Pi OS to an NVMe drive from another computer), then follow the steps in the next section to clone your existing Pi OS install to an NVMe SSD.


Here are a few ways to flash the OS to an NVME SSD:

1. Use SD Card Copier tool to flash OS onto the NVME SSD On Raspberry Pi OS(recommended)

If you're using a Raspberry Pi OS, you can just run the SD card copier tool to flash OS onto the NVME SSD.

Cick Applications =>Accessories =>SD Card Copier on the main screen, run the SD Card Copier program, and copy the OS to the NVME ssd as shown in the figure below.

 

Click Start to run. Then shut down, unplug the SD card, and restart the device.


2. Flash the SSD with Raspberry Pi Imager

To get the NVMe SSD to boot your Pi, it needs to have an OS. One option would be to clone an existing installation to it using rpi-clone or some other tool (see below), but my preferred option is to flash a fresh Pi OS install using Raspberry Pi Imager.

  1. Install Pi Imager and open it
  2. Plug your NVMe SSD into your computer using a USB to NVMe adapter
  3. Choose an OS to install
  4. Choose the drive (connected through your adapter) to flash
  5. Click write (and set any options you'd like)

Once flashing is complete, pull the NVMe drive, attach it to your Pi 5, and it should boot off it (with or without a microSD card inserted)—assuming you have the bootloader up to date and set the BOOT_ORDER appropriately!

3. Clone your microSD boot volume to an NVMe SSD

Assuming you already have Raspberry Pi OS on a microSD card that is booting your Raspberry Pi 5 internally, and the NVMe SSD is connected and visible (check if you see a device /dev/nvme0n1 after running lsblk), you can use rpi-clone to clone the internal microSD boot volumes to your NVMe SSD:

# Install rpi-clone.
git clone -b 123-nvme https://github.com/geerlingguy/rpi-clone.git
cd rpi-clone
sudo cp rpi-clone rpi-clone-setup /usr/local/sbin

# Clone to the NVMe drive (usually nvme0n1, but check with `lsblk`).
sudo rpi-clone nvme0n1

Note: You may want to wipe all disk partitions before cloning:

sudo umount /dev/nvme0n1p?
sudo wipefs --all --force /dev/nvme0n1p?
sudo wipefs --all --force /dev/nvme0n1
sudo dd if=/dev/zero of=/dev/nvme0n1 bs=1024 count=1

Upgrade eeprom, solve NO wifi issue

NOTE: If you find that there is no WIFI after restarting, you can try this part of the update, otherwise please ignore it.

After modifying the BOOT_ORDER for NVME SSD, reboot the PI5 and you will get the NO WIFI error. We need to upgrade the eeprom to fix it (Raspberry Pi Foundation official may upgrade the OS to solve this kind of problem, please pay attention to the update of Raspberry Pi OS, if the OS upgrade can solve this problem, you can ignore this step).

Run the following command to

wget https://github.com/timg236/rpi-eeprom/raw/pieeprom-2023-10-30-2712/firmware-2712/default/pieeprom-2023-10-30.bin 
sudo rpi-eeprom-config --edit pieeprom-2023-10-30.bin

 


It is recommended to use the latest version of the pieeprom-xxxx-xx-xx.bin file, please visit this link to get the latest version:

PS: After our testing, we found that only the firmware piepprom-2023-10-30.bin has solved the NO WIFI problem for the time being.


Return to X1001 / X1000

Add your comment
Geekworm Wiki welcomes all comments. If you do not want to be anonymous, register or log in. It is free.


 

Anonymous user #12

13 days ago
Score 0  

The WD Green SN580 works like a charm with the latest rpi-eeprom-update.

Used the rpi-clone fork from Jeff Geerling to copy a fresh Bookworm install onto the SSD. 740 MB/s write and 780 MB/s read for a 1GB file in 10 blocks of 100MB each. With PCIe3 enabled tho
 

Anonymous user #11

one month ago
Score 0  

Bought X1002 & WD Green SN350 2TB

Works just fine without any configuration. The only configuration change to enable PCI_gen3:

  1. PCIe Gen3

dtparam=pciex1_gen=3

within:

sudo nano /boot/firmware/config.txt

  1. PCIe Gen3

dtparam=pciex1_gen=3


$ sudo lspci 0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21) 0000:01:00.0 Non-Volatile memory controller: Sandisk Corp WD PC SN540 / Green SN350 NVMe SSD 1 TB (DRAM-less) 0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21) 0001:01:00.0 Ethernet controller: Device 1de4:0001


$ duf ╭───────────────────────────────────────────────────────────────────────────────────────────────────╮ │ 2 local devices │ ├────────────────┬────────┬────────┬────────┬───────────────────────────────┬──────┬────────────────┤ │ MOUNTED ON │ SIZE │ USED │ AVAIL │ USE% │ TYPE │ FILESYSTEM │ ├────────────────┼────────┼────────┼────────┼───────────────────────────────┼──────┼────────────────┤ │ / │ 1.8T │ 789.3G │ 949.7G │ [########............] 43.1% │ ext4 │ /dev/nvme0n1p2 │ │ /boot/firmware │ 511.0M │ 75.3M │ 435.6M │ [##..................] 14.7% │ vfat │ /dev/nvme0n1p1 │

╰────────────────┴────────┴────────┴────────┴───────────────────────────────┴──────┴────────────────╯
 

Anonymous user #8

2 months ago
Score 0  
Just bought an x1001 and paired it with a 1T Samsung 980 Pro. Installed with no problems other than having to reposition the PCIe cable once. Love the performance!
 

Anonymous user #6

3 months ago
Score 0  

[SOLVED] Errors booting from NVME with X1002 base and Crucial T500 Gen4 NVMe.

Symptoms: Either: 1. *Extremely long* boot times (~240 seconds) with pcieport and nvme errors in dmesg including:

[ 4.230345] pcieport 0000:00:00.0: AER: Corrected error received: 0000:00:00.0 [ 4.230359] pcieport 0000:00:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID) [ 4.230362] pcieport 0000:00:00.0: device [14e4:2712] error status/mask=00001000/00002000 [ 4.230366] pcieport 0000:00:00.0: [12] Timeout

...and...

[ 41.971850] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 41.971859] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 41.971862] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 42.003987] nvme0n1: I/O Cmd(0x2) @ LBA 6559080, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 42.003993] I/O error, dev nvme0n1, sector 6559080 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 [ 42.019855] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 42.050334] nvme nvme0: 4/0/0 default/read/poll queues [ 72.331604] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 72.331612] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 72.331615] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 72.364112] nvme0n1: I/O Cmd(0x2) @ LBA 101390768, 16 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364119] I/O error, dev nvme0n1, sector 101390768 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 72.364131] nvme0n1: I/O Cmd(0x2) @ LBA 1328360, 8 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364135] I/O error, dev nvme0n1, sector 1328360 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 72.364143] nvme0n1: I/O Cmd(0x2) @ LBA 1327704, 8 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364146] I/O error, dev nvme0n1, sector 1327704 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 72.364156] nvme0n1: I/O Cmd(0x2) @ LBA 5690072, 8 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364159] I/O error, dev nvme0n1, sector 5690072 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 72.364172] nvme0n1: I/O Cmd(0x2) @ LBA 7574968, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364175] I/O error, dev nvme0n1, sector 7574968 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 72.364183] nvme0n1: I/O Cmd(0x2) @ LBA 7629088, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364186] I/O error, dev nvme0n1, sector 7629088 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 [ 72.364280] nvme0n1: I/O Cmd(0x2) @ LBA 17834496, 88 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364283] I/O error, dev nvme0n1, sector 17834496 op 0x0:(READ) flags 0x80700 phys_seg 11 prio class 2 [ 72.364297] nvme0n1: I/O Cmd(0x2) @ LBA 17834592, 168 blocks, I/O Error (sct 0x3 / sc 0x71) [ 72.364300] I/O error, dev nvme0n1, sector 17834592 op 0x0:(READ) flags 0x80700 phys_seg 21 prio class 2 [ 72.380221] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 72.410751] nvme nvme0: 4/0/0 default/read/poll queues [ 103.652649] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 103.652658] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 103.652661] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 103.684774] nvme0n1: I/O Cmd(0x2) @ LBA 7566288, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 103.684782] I/O error, dev nvme0n1, sector 7566288 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 [ 103.684802] nvme0n1: I/O Cmd(0x2) @ LBA 8615992, 56 blocks, I/O Error (sct 0x3 / sc 0x71) [ 103.684805] I/O error, dev nvme0n1, sector 8615992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 103.684813] nvme0n1: I/O Cmd(0x2) @ LBA 4027408, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 103.684816] I/O error, dev nvme0n1, sector 4027408 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 103.684827] nvme0n1: I/O Cmd(0x2) @ LBA 1333928, 8 blocks, I/O Error (sct 0x3 / sc 0x71) [ 103.684830] I/O error, dev nvme0n1, sector 1333928 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 103.700653] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 103.729101] nvme nvme0: 4/0/0 default/read/poll queues [ 136.100697] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 136.100706] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 136.100709] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 136.152704] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 136.183196] nvme nvme0: 4/0/0 default/read/poll queues [ 170.724717] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 170.724726] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 170.724729] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 170.760849] nvme0n1: I/O Cmd(0x2) @ LBA 1694616, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 170.760856] I/O error, dev nvme0n1, sector 1694616 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 [ 170.780730] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 170.811175] nvme nvme0: 4/0/0 default/read/poll queues [ 206.051720] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 206.051728] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 206.051731] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 206.099734] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 206.130254] nvme nvme0: 4/0/0 default/read/poll queues [ 228.538013] ieee80211 phy0: brcmf_p2p_send_action_frame: Unknown Frame: category 0xa, action 0x8 [ 242.403171] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 [ 242.403180] nvme nvme0: Does your device have a faulty power saving mode enabled? [ 242.403183] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug [ 242.435299] nvme0n1: I/O Cmd(0x2) @ LBA 1993800, 64 blocks, I/O Error (sct 0x3 / sc 0x71) [ 242.435306] I/O error, dev nvme0n1, sector 1993800 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2 [ 242.451184] nvme 0000:01:00.0: enabling device (0000 -> 0002) [ 242.481653] nvme nvme0: 4/0/0 default/read/poll queues

2. Long, but better boot times (~35 sec), still with pcieport and nvme errors as above, but fewer of them.

Hardware: Raspberry Pi 5 with X1002 bottom hat SSD: Crucial T500 PCIe Gen 4 NVMe M.2 SSD 1TB

Firmware: eeprom version: updated to Feb 5 2024 (1707143914) while troubleshooting. Unsure if earlier version would have worked.

Updates made to enable boot from NVME: Changes in eeprom via rpi-eeprom-config -e: BOOT_ORDER=0xf416 (added the 6) and added PCIE_PROBE=1. Also changed POWER_OFF_ON_HALT=1, but that was just to reduce power consumption and had no effect on speedy or error-free booting.

Changes to /boot/firmware/config.txt Added line "dtparam=pciex1" to the bottom of the file. Note: it works equally well using the alias "dtparam=nvme" as instead. Added line "dtparam=pciex1_gen3" to the bottom of the file. <-- ***THIS IS WHAT FINALLY SOLVED THE DMESG ERRORS AND SLOW BOOT TIMES*** in spite of this Geekworm guide stating the setting was not recommended and may lead to instability. In my case, it was the only setting to result in stable operation.

Other things that didn't work / had no effect: Added "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" to /boot/firmware/cmdline.txt

Added PSU_MAX_CURRENT=5000 to eeprom via rpi-eeprom-config -e
 

Anonymous user #9

2 months ago
Score 0  

spend whole day on the same issue. having Crucial P3 Plus 500GB ssd : Model Number: CT500P3PSSD8 Firmware Version: P9CR411

thank you man for sharing, you saved my live :)
 

Anonymous user #10

one month ago
Score 0  
What is a good boot time booting from NVMe?
 

Anonymous user #5

3 months ago
Score 0  
How do you change the nut from the 2242 position to the 2230 position?
 

Anonymous user #7

3 months ago
Score 0  
I used a hot air station to unsolder and solder it into place
 

Anonymous user #4

3 months ago
Score 0  

hi from vienna,

i have no luck till now withj the x1002:

  • first i had a WD black 770 and read that this is on the incompatible list
  • returned it and ordered a crucial t500 1TB nvme ssd but still there is NO reaction (no blue light, nothing reported in lspci,..)

looks like the x1002 is not found / connected / or ist just plain dead

  • did the edits as described above in boot/config.txt
  • no success

any advice please

cheerz
 

Anonymous user #4

3 months ago
Score 0  
oha ... flushed a new eeprom and VOILA now it works :-)
 

Anonymous user #3

3 months ago
Score 0  

While following the instructions I noticed the following messages in dmesg: pcieport 0000:00:00.0: AER: Corrected error received: 0000:00:00.0 pcieport 0000:00:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID) pcieport 0000:00:00.0: device [14e4:2712] error status/mask=00001000/00002000 pcieport 0000:00:00.0: [12] Timeout

I am running the latest eeprom CURRENT: Fri 5 Jan 15:57:40 UTC 2024 (1704470260) LATEST: Fri 5 Jan 15:57:40 UTC 2024 (1704470260)

The device itself boots just fine.
 

Anonymous user #2

3 months ago
Score 0  
If the nvme is not recognised after adding dtparam=pciex1 to config.txt, add PCIE_PROBE=1 to the eeprom. That made my pi see the nvme.
 

Harry

3 months ago
Score 0  

Thanks.

Run command:

sudo rpi-eeprom-config --edit
Add PCIE_PROBE=1 line, then press CTRL+O to write change to file, press CTRL+X to exit, then reboot
 

Anonymous user #1

3 months ago
Score 0  

Not listed here but should be - you should also add the following line under the BOOT_ORDER line when configuring the eeprom:

PCIE_PROBE=1

My Raspberry Pi 5 would see the NVMe SSD, but would not boot from it without this line. Using an SK Hynix PC601 256GB NVMe SSD
 

Walker

3 months ago
Score 0  
Thanks a lot for the advice.