AxelFoley

Update: well its was not the NVMe drive or the USB Peripherals. I disconnected them both and the Armbian image still refuses to boot the kernel There is still and unknown device Device 0: unknown device Card did not respond to voltage select! : -110 switch to partitions #0, OK and the image still fails to boot Applying kernel provided DT fixup script (rockchip-fixup.scr) ## Executing script at 09000000 Moving Image from 0x2080000 to 0x2200000, end=3f90000 ## Loading init Ramdisk from Legacy Image at 06000000 ... Image Name: uInitrd Image Type: AArch64 Linux RAMDisk Image (gzip compressed) Data Size: 14238821 Bytes = 13.6 MiB Load Address: 00000000 Entry Point: 00000000 Verifying Checksum ... OK ## Flattened Device Tree blob at 01f00000 Booting using the fdt blob at 0x1f00000

Nah I think the above hypothesis is wrong, in some respects. My (poor) understanding of the boot process is; 1). First/Primary Boot Stage loader is BROM in the SOC's Chip initiated to enable enough HW to find the main boot loader (e.g. U-boot's SPL Secondary Boot Loader which is loaded on SPIFlash/eMMC/SDMMC Card), 2). The U-Boots SPL then initiates more HW (such as DDR Memory) to enable the full U-Boot to be loaded into larger memory space. >> I suspect that this is a legacy constraint ... legacy systems had limited memory so any boot loader needs to fit into those smaller memory environments for backwards compatibility. >> however if it can probe and detect more memory (NAND,NOR Flash Chips, eMMC, SDMMC & DDR Memory). This is Where the TPL Comes in 3). Tertiary Program Loader (to look for larger memory HW in which the SPI can load a bigger version of U-Boot out side of legacy constrained HW). >> So I think that both eMMC's U-Boot SPL Part of the boot image is being initialised on the eMMC (nee MMC lines in the serial output as eMMC is treated as SDMMC). It immediately loads the TPL to initialise the DDR memory and this explains why you see the TPL before the SPL in the output. The reason why the working Auyfan's has more verbose DDR output is because he is using a debug version of U-Boot; Auyfan working u-boot ( v1.3(debug):370ab80) SPL finds kernel image on eMMC Found /boot/extlinux/extlinux.conf ...Retrieving file: /boot/vmlinuz-4.4.190-1233-rockchip-ayufan-gd3f1be0ed310 Armbian non working u-boot v1.3(release):845ee93 SPL finds kernel image on eMMC Found U-Boot script /boot/boot.scr so TPL hands back to SPL again after initialising the DDR Memory so SPL can load the big U-Boot image into DDR Memory .... but this is where I think the problem is .... Both images seem to try to load a Ram Disk to load the kernel into Aufan; ## Flattened Device Tree blob at 01f00000 Booting using the fdt blob at 0x1f00000 Loading Ramdisk to f5902000, end f5f00ca8 ... OK Loading Device Tree to 00000000f58e7000, end 00000000f5901810 ... OK Armbian; Applying kernel provided DT fixup script (rockchip-fixup.scr) ## Executing script at 09000000 Moving Image from 0x2080000 to 0x2200000, end=3f90000 ## Loading init Ramdisk from Legacy Image at 06000000 ... Image Name: uInitrd Image Type: AArch64 Linux RAMDisk Image (gzip compressed) Data Size: 14238821 Bytes = 13.6 MiB Load Address: 00000000 Entry Point: 00000000 Verifying Checksum ... OK ## Flattened Device Tree blob at 01f00000 Booting using the fdt blob at 0x1f00000 So I need to look at what this "rockchip-fixup.scr" is actually doing that Auyan's image is not. because it looks to me like the Armbian image is failing to load the kernel into DDR memory. Now this may be down to this line in the Armbian Boot process; Device 0: unknown device Card did not respond to voltage select! : -110 switch to partitions #0, OK >> I wonder if its the NVMe storage that's confusing the Armbian image boot process and preventing it from loading the kernel into DDR. Time for bed but tomorrow I'll disconnect the PCIe NVMe device and see if Armbian boots !

Well after a bit of tracing the serial output of the boot load process and looking at a working ayufan image against a Armbian Image (5 kernel) it looks like they have a different way of implementing U-Boot for the RockPro64 RK3399. Note my test boards come with PCIe NVMe Drives and the board provides 128Mb SPI boot Flash and 64GbE eMMC Storage from which I am trying to boot the kernel. Not booting eMMC: Armbian TPL -> SPL-> U-Boot -> Kernel Booting eMMC: Ayufan SPL -> U-Boot -> kernel >> What I notice is that Ayufan image has a serial boot output prefixed by what looks like a LDDR Memory test/probe. The boot sequence should be consistent at first then deviate, not deviate right from the start ! RTFM Here https://wiki.pine64.org/wiki/RK3399_boot_sequence "It employs 5 strategies, in order, to load a bootloader from off-chip: loading from NOR flash on SPI1 loading from NAND flash on SPI1 loading from eMMC loading from SD on the SDMMC controller bringing up the OTG0 USB controller in device mode and accepting control transfers to load programs Loading a bootloader from storage is done in 4 steps: Probing the boot device Finding the ID block Loading the first stage of the bootloader to main SRAM and running it (if the first stage returned to BROM) Loading the second stage to low DRAM and running it" >> I have a suspicion that Ayufan's images have loaded a U-Boot Boot loader into SPI Flash and, perhaps that may be the issue as to why his images work but others do not. His kernel may be located at address below in eMMC but other boot images are not. "Booting using the fdt blob at 0x1f00000" I'll see if I can wipe the SPI Flash and see if Armbian starts to boot. FailedBoot.log SuccessfulBootlog.log

@Igor Understood .... Subscription taken out and Serial cable orders so I can track the boot sequence tomorrow .....

>> Thx Serial console cable on order lets see what I can see tomorrow.

Tried setting extraargs="mmc_cmdqueue=off" Still same error. When I googled around a bit, one suggestion was that it was caused by the 1st n byte's of the eMMC chip getting nulled out deleting the normal boot partition location and that the solution was to move the boot sector to a new address range outside that first n bytes... but I'm no boot expert ... I'll do some digging;

Hi @Igor, I am happy to contribute to the project if it helps. Do the maintainers need a Pine RockPro64 with 64Gb eMMC ? I can order a couple more at the end of the month and send one to the maintainers. just let me know with a DM. I'm also happy to test releases if people want me to, but I spend most of my free time on DE10 Nano and the Pine RocPro64 Pico Clusters I run are mainly to test orchestration toolsets like Ansible and SaltStack and Distributed DB's like Cassandra. I have a VMWare Server to run RHEL Container platforms like Openshift to test things like Service Discovery Consul and Monitoring with Prometheus/Grafana for proprietary apps. That's my background I'm no bootstrap expert (after UEFI I gave up interest in the subject), but I can pick stuff up I don't check this forum often so somebody would have to reach out to me to be part of a QA Process for that board. I seem to spend weeks when I am doing a cluster rebuild just testing and trying to find a stable image that is; 1). Debian based distro 2). 64bit (arm64) 3). That boots to eMMC. 4). That has a Desktop build for the control plane node that supports attached Nvme Storage 5). That has a CLI build for the worker nodes. Its seems to be a constant problem with these Pine RockPro64 boards. They are hardly an AliExpress Clone outfit ... Pine are quite well established 🙂 www.pine64.org Let me know how you feel I can contribute

This was my goto distro and Im going to have to find another one because of the lack of support for eMMC going forward. Why has the 5 kernels led to the drop in support for eMMC?

https://wiki.pine64.org/wiki/ROCKPro64_Software_Release "If you are booting from a Micro SD card, then both Linux kernel versions will work. If you are trying to boot from an eMMC module then the 4.4.y will work, but the newer 5.10.y will not." Doh ! So you will need to flash to 4.4.y or boot from SD Card if you want Armbian

Happens on both the CLI and GUI Desktop Images tried flashing in Rufus and Etcher It simply dose not work on eMMC .... so I don't know how this could get released without the basics being checked. I'm not going to buy SD Cards for my 20 node clusters !

No you are not the only person. The Image is screwed and wont boot when flashed to eMMC. It boots from SD. I'm looking into the problem now.

I have been monitoring the voltage and current draw of an individual RockPro64 on boot with the PCIe NVME Connected going to console. I then launched the desktop as both root (Armbian desktop) and pico user (looks like XFCE) and looked for any power spikes. There was none! Both tests caused a lockup and HW Freeze. On boot the peak current draw was 0.7A with 12v steady, and current draw dropping to 0.3A steady. Then I launched Xorg as root (in the past tended to be more stable) and user pico (always locked up immediately) I did notice a difference when I ran startx as pico user and it launched XFCE .. its current draw peaked at 0.7a and voltage dropped to 11.96V Desktop immediately locked the board on launch. When I ran startx as root and it launched armbian desktop ... it only peaked at 0.6A and voltage never dipped below 12v. The desktop was responsive until when I loaded the Armbian forum in chrome and triggered the board lockup ..... there was no current spike or voltage drop! The board locked up while only drawing 0.29A. I checked the Pine forum and other people are reporting exactly the same issue as myself with the PCIe/NVMe setup. One person in 2018 reported he fixed the issue by launching a different kernel. I have logged the problem on the pine forum and got this response; "Currently working on the issue. It seems - as odd is its sounds - that the problem is somehow linked to pulseaudio. If you uninstall pulseaudio, and use alsa instead, the issue will just vanish. We have tried blacklisting PCIe for pulse in udev, and it prevents the issue from happening, but it also returns a segmentation error (SATA card / other adapter not accessible). Its very very strange". Should I stop posting here and move the discussion to Pine ?

@TonyMac32 Thankfully I have the RockPro64 v2.1 board.

@pfry Looking at the power Specifications for PCIe x 4 => Suggest that it can be powered from 3.3v (9.9W) & 12v (25W), however from the RockPro64 Power schema it looks like they negate to feed the 12v rail from the Supply voltage to the PCIe. Instead Pine feeds the PCIe Interface on the board only by the 3.3v (3A) rail (9.9W). From the Power schema It also looks like the board designers feed PCIe from the 5.1v rail converted by the RK808 PMU to 1.8v on vcc1v8_pcie (not sure how this is intended to be used on PCIe) I am using the Pine PCIe ver 3.0 NVMe Card with a Samsung 970 EVO 500GB and a RockPro64 v2.1 Board. PSU is a 102W 12v LRS-100-12 The Max power draw of the EVO 970 NVMe is 5.8W (1.76A) which should be within spec for that 3.3v rail. But this bit worry's me "vcc_sys: could not add device link regulator.6 err -2" vcc_sys is the 3.3v rail that feeds the vcc3v3_pcie feed into the PCIe Socket. Although dmeag later says it can enable a regulator for the 3.3v vcc3v3_pcie rail hanging off vcc3v3_sys. I also see this warning; "Apr 8 19:09:24 localhost kernel: [ 2.010352] pci_bus 0000:01: busn_res: can not insert [bus 01-ff] under [bus 00-1f] (conflicts with (null) [bus 00-1f])" I may have to get out a JTAG debugger to work this one out :-( Not sure if this is a kernel driver issue or HW Power design. ill see if I can get any Gerber files to scope out the voltage and current spikes.

I think I have found the issue !!!!!!! Its the PCIe Express NVMe Card. I remove it and the desktop seems not to hang .... I add it back in .... and the desktop hangs. I wonder if this has something to do with power spikes when there is graphics activity the errors in the dmesg indicate that vpcie1v8 = 5.1v rail Shared with GPU vpcie0v9 = 3v Rail Both of these rails hang off the same core buck converter SY8113B, the other SY8113B manages the USB peripherals seperatly. @AndrewDB .... looks like you may be correct it was power all along but it looks like its a kernel issue with the PCIe Power Management ?