207.173.135.189: Migrated comment #4433

2024-02-13T19:56:51Z

Migrated comment #4433

New page

[SOLVED] Errors booting from NVME with X1002 base and Crucial T500 Gen4 NVMe.

Symptoms: Either:
1. *Extremely long* boot times (~240 seconds) with pcieport and nvme errors in dmesg including:

[ 4.230345] pcieport 0000:00:00.0: AER: Corrected error received: 0000:00:00.0
[ 4.230359] pcieport 0000:00:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)
[ 4.230362] pcieport 0000:00:00.0: device [14e4:2712] error status/mask=00001000/00002000
[ 4.230366] pcieport 0000:00:00.0: [12] Timeout

...and...

[ 41.971850] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
[ 41.971859] nvme nvme0: Does your device have a faulty power saving mode enabled?
[ 41.971862] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug
[ 42.003987] nvme0n1: I/O Cmd(0x2) @ LBA 6559080, 64 blocks, I/O Error (sct 0x3 / sc 0x71)
[ 42.003993] I/O error, dev nvme0n1, sector 6559080 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2
[ 42.019855] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[ 42.050334] nvme nvme0: 4/0/0 default/read/poll queues
[ 72.331604] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
[ 72.331612] nvme nvme0: Does your device have a faulty power saving mode enabled?
[ 72.331615] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug
[ 72.364112] nvme0n1: I/O Cmd(0x2) @ LBA 101390768, 16 blocks, I/O Error (sct 0x3 / sc 0x71)
[ 72.364119] I/O error, dev nvme0n1, sector 101390768 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[ 72.364131] nvme0n1: I/O Cmd(0x2) @ LBA 1328360, 8 blocks, I/O Error (sct 0x3 / sc 0x71)
[ 72.364135] I/O error, dev nvme0n1, sector 1328360 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[ 72.364143] nvme0n1: I/O Cmd(0x2) @ LBA 1327704, 8 blocks, I/O Error (sct 0x3 / sc 0x71)
[ 72.364146] I/O error, dev nvme0n1, sector 1327704 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[ 72.364156] nvme0n1: I/O Cmd(0x2) @ LBA 5690072, 8 blocks, I/O Error (sct 0x3 / sc 0x71)
[ 72.364159] I/O error, dev nvme0n1, sector 5690072 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[ 72.364172] nvme0n1: I/O Cmd(0x2) @ LBA 7574968, 64 blocks, I/O Error (sct 0x3 / sc 0x71)
[ 72.364175] I/O error, dev nvme0n1, sector 7574968 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[ 72.364183] nvme0n1: I/O Cmd(0x2) @ LBA 7629088, 64 blocks, I/O Error (sct 0x3 / sc 0x71)
[ 72.364186] I/O error, dev nvme0n1, sector 7629088 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2
[ 72.364280] nvme0n1: I/O Cmd(0x2) @ LBA 17834496, 88 blocks, I/O Error (sct 0x3 / sc 0x71)
[ 72.364283] I/O error, dev nvme0n1, sector 17834496 op 0x0:(READ) flags 0x80700 phys_seg 11 prio class 2
[ 72.364297] nvme0n1: I/O Cmd(0x2) @ LBA 17834592, 168 blocks, I/O Error (sct 0x3 / sc 0x71)
[ 72.364300] I/O error, dev nvme0n1, sector 17834592 op 0x0:(READ) flags 0x80700 phys_seg 21 prio class 2
[ 72.380221] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[ 72.410751] nvme nvme0: 4/0/0 default/read/poll queues
[ 103.652649] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
[ 103.652658] nvme nvme0: Does your device have a faulty power saving mode enabled?
[ 103.652661] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug
[ 103.684774] nvme0n1: I/O Cmd(0x2) @ LBA 7566288, 64 blocks, I/O Error (sct 0x3 / sc 0x71)
[ 103.684782] I/O error, dev nvme0n1, sector 7566288 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2
[ 103.684802] nvme0n1: I/O Cmd(0x2) @ LBA 8615992, 56 blocks, I/O Error (sct 0x3 / sc 0x71)
[ 103.684805] I/O error, dev nvme0n1, sector 8615992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[ 103.684813] nvme0n1: I/O Cmd(0x2) @ LBA 4027408, 64 blocks, I/O Error (sct 0x3 / sc 0x71)
[ 103.684816] I/O error, dev nvme0n1, sector 4027408 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[ 103.684827] nvme0n1: I/O Cmd(0x2) @ LBA 1333928, 8 blocks, I/O Error (sct 0x3 / sc 0x71)
[ 103.684830] I/O error, dev nvme0n1, sector 1333928 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[ 103.700653] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[ 103.729101] nvme nvme0: 4/0/0 default/read/poll queues
[ 136.100697] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
[ 136.100706] nvme nvme0: Does your device have a faulty power saving mode enabled?
[ 136.100709] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug
[ 136.152704] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[ 136.183196] nvme nvme0: 4/0/0 default/read/poll queues
[ 170.724717] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
[ 170.724726] nvme nvme0: Does your device have a faulty power saving mode enabled?
[ 170.724729] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug
[ 170.760849] nvme0n1: I/O Cmd(0x2) @ LBA 1694616, 64 blocks, I/O Error (sct 0x3 / sc 0x71)
[ 170.760856] I/O error, dev nvme0n1, sector 1694616 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2
[ 170.780730] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[ 170.811175] nvme nvme0: 4/0/0 default/read/poll queues
[ 206.051720] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
[ 206.051728] nvme nvme0: Does your device have a faulty power saving mode enabled?
[ 206.051731] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug
[ 206.099734] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[ 206.130254] nvme nvme0: 4/0/0 default/read/poll queues
[ 228.538013] ieee80211 phy0: brcmf_p2p_send_action_frame: Unknown Frame: category 0xa, action 0x8
[ 242.403171] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
[ 242.403180] nvme nvme0: Does your device have a faulty power saving mode enabled?
[ 242.403183] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug
[ 242.435299] nvme0n1: I/O Cmd(0x2) @ LBA 1993800, 64 blocks, I/O Error (sct 0x3 / sc 0x71)
[ 242.435306] I/O error, dev nvme0n1, sector 1993800 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2
[ 242.451184] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[ 242.481653] nvme nvme0: 4/0/0 default/read/poll queues

2. Long, but better boot times (~35 sec), still with pcieport and nvme errors as above, but fewer of them.

Hardware:
Raspberry Pi 5 with X1002 bottom hat
SSD: Crucial T500 PCIe Gen 4 NVMe M.2 SSD 1TB

Firmware:
eeprom version: updated to Feb 5 2024 (1707143914) while troubleshooting. Unsure if earlier version would have worked.

Updates made to enable boot from NVME:
Changes in eeprom via rpi-eeprom-config -e: BOOT_ORDER=0xf416 (added the 6) and added PCIE_PROBE=1. Also changed POWER_OFF_ON_HALT=1, but that was just to reduce power consumption and had no effect on speedy or error-free booting.

Changes to /boot/firmware/config.txt
Added line "dtparam=pciex1" to the bottom of the file. Note: it works equally well using the alias "dtparam=nvme" as instead.
Added line "dtparam=pciex1_gen3" to the bottom of the file. <-- ***THIS IS WHAT FINALLY SOLVED THE DMESG ERRORS AND SLOW BOOT TIMES*** in spite of this Geekworm guide stating the setting was not recommended and may lead to instability. In my case, it was the only setting to result in stable operation.

Other things that didn't work / had no effect:
Added "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" to /boot/firmware/cmdline.txt
Added PSU_MAX_CURRENT=5000 to eeprom via rpi-eeprom-config -e

CommentStreams:Aadc43263e05b8bfdba42b23d6853bd7 - Revision history

207.173.135.189: Migrated comment #4433