another "no boot" :(

[EvIl_DeViL]

Hi all,

Igor: I wanted to say many thank you and congratulate for the awesome job you made with armbian!

 

I have a little big problem:

I'm using the BPI-R1 (a.k.a. Lamobo R1 Linux version 4.2.0-sunxi (root@kitchen) (gcc version 4.8.2 (Ubuntu/Linaro 4.8.2-16ubuntu4)) but out of the blue I got the well known problem of the RCP reset to the 1970.

I followed the instructions reported here https://github.com/igorpecovnik/lib/issues/111#issuecomment-145634651 but I got some strange issues

1) If I plug my USB keyboard I cannot interrupt the boot process to get a shell before the kernel loads.

2) If I change my boot.cmd and update the boot.scr settings with verbosity=9 and init=/bin/fixme.sh it has no effect; it seems to load a "default/standard" boot.scr located somewhere (even if pre kernel boot says /boot/boot.scr loaded)

3) If I plug my uart/USB adapter I can get a shell, but after I follow the procedure described here 

 

https://github.com/igorpecovnik/lib/issues/111#issuecomment-147452433 RTC resets again

 

Strange thing: I don't get the log of the kernel loading on the dmesg; it seems like it logs to the RAM and after it mounts the SD the system dump 

 

the log into the dmeseg. I think this doens't happen because it never mounts the SD card.

 

the dump I'm attaching herein has been captured with my serial terminal which by now is the only way I can access my system.

 

Relevant logs are:

setenv - set environment variables



Usage:

setenv [-f] name value ...

    - [forcibly] set environment variable 'name' to 'value ...'

setenv [-f] name

    - [forcibly] delete environment variable 'name'

[    0.000000] Kernel command line: [empty!!! I swear that here in the past I saw the envs var defined in the /boot/boot.cmd!!! maybe setenv usage is a hint about a failed setenv command?]

[    3.573034] sunxi-rtc 1c20d00.rtc: setting system clock to 1970-01-01 00:00:13 UTC (13)

[    3.607353] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6

 

sorry to bother but I'm struggling after this issue since about two weeks and I don't know where I can begin to fix that...

 

Tell me if I need to open an issue on github (I saw Igor closed the previous one so I didn't open a new one)

 

 

many thanks!

captureNoBootUART_wrongboot.txt

boot.cmd.txt

[zador.blood.stained]

Hi. Decided to register here to comment on my instructions.

1) Seems like USB support for Lamobo R1 is not fully enabled in U-Boot config for some reason, maybe it's not supported yet, but it doesn't matter if you have serial console access.

2) Setenv fails because string gets too long if you add extra parameters like init or loglevel, can be solved by using double quotes like in this script: https://github.com/igorpecovnik/lib/blob/second/config/boot.cmd

3) Last post with fixtime.sh script was an idea for automatic debricking of new installations without user interaction, but since you have serial console, these https://github.com/igorpecovnik/lib/issues/111#issuecomment-145634651instructions shoud be more useful, try setting RTC clock manually and check if changes are still there on next boot.

[EvIl_DeViL]

1) that 4 bucks cheap-o ttl serial converter turns out to be one of the best investments I made in the last few years XD

2) woo good hint about the double quotes! I didn't know it and it looks that now my boot args are working... just to clarify: why someone shouldn't double quote boot args by default?

3) I already tried those steps but it looks like my corrupted uboot conf file was bothering the fix. Anyway I repeated the procedure after the boot fix and startup now hangs on

[    4.419814] random: init urandom read with 7 bits of entropy available

and doesn't give me the bash shell.

4) is it possible to specify both ttyS0,115200 and tty1 as console (like before kernel starts)? Just to have a little more flexibility...

 

I attached a new boot log got for completeness.

 

Thank you zador. Once again you pointed me in the right direction

bootFix_Lamobo_R1.txt

[zador.blood.stained]

2) Good old "Don't touch it if it works" method of dealing with stuff.

3) Those commands simply boot kernel with custom init parameter without executing boot.cmd (boot.scr), so it doesn't matter if your boot.scr was broken. Try adding loglevel=5 (or maybe 6) to bootargs and post boot log somewhere. It should work unless you have non-default setup or if I'm missing something about your board compared to mine (cubietruck).

4) Last time I checked only one console parameter was recognized by the kernel for some reason, although more than one should work in theory.

[Igor]

1) Lamobo U-boot support is made by me with a quick patch. I didn't bother with fixing USB port(s) but I'll take a look. Let's say it's a known bug.

2) It will be from now on by default

 

There were some problems with 4.2.x kernel but fixed. It's possible that it hangs because of that so I recommend you to update kernel to latest - check Documentation, section "how to unbrick my system", because you will need to do it manually ... since the system doesn't boot.

 

Thanks!

[EvIl_DeViL]

@zador

3) I taught maybe the "broken boot" could undo the changes made with the custom booting... who knows I posted a log with loglevel=7 so you can see all the messages. I didn't altered my boot in any way AFAIK. maybe some strange install? (samba, apache, mysql, gparted) But I installed everything from default repository via apt-get or synaptic

4) I found something interesting: https://www.kernel.org/doc/Documentation/serial-console.txt (<- seems pretty official to me )

You can specify multiple console= options on the kernel command line.
Output will appear on all of them. The last device will be used when
you open /dev/console. So, for example:

	console=ttyS1,9600 console=tty0

I tried and it works! now I have output echoed on both interfaces so I can plug it via HDMI or TTLSerialUSB without altering the uboot script.

 

@Igor

1) in fact uboot log says "Unknown command 'usb' - try 'help'"... I panicked a little when I read "hit any key to stop autoboot" and nothing happend after I did so XD luckily I had the said adapter.

(Remembered a little the old MS "no keyboard found: press f1 to continue...")

 

what kind of datas/configs could I loose if I follow said procedure? could the startup hang be caused by something not related to those /mnt files? (udev, init.d, systemd, video drivers)

 

as always thank you guys!

loglevel7.txt

[zador.blood.stained]

Strange... Can you install (extract) latest kernel & dtb to this card from another system? Also just in case execute force check file system (fsck.ext4 -f).

If it still doesn't boot and if you have spare sd card, you can try booting fresh image, if you don't - you can backup existing with dd and install new image on this card.

 

Edit: First try adding init=/bin/bash to kernel command line, maybe there are issues starting init process.

 

Edit 2: Also try flashing latest u-boot (next)

 

Edit 3: Compared you log to my dmesg. Definitely init problem, most likely systemd and RTC related one. For me systemd starts right after usb devices init. You still need to boot to bash to correct RTC time.

[EvIl_DeViL]

No success

 

I unpacked the four *.deb in the root of my sd card following the "How to unbrick the system?" guide.

I had the 4.2.0 version and after I unpacked the *.deb and replaced everything I saw the new version is 4.2.2

I also relinked the zImage as stated and boot log confirm that now I'm booting the 4.2.2

 

I post herein a log with 3 boot:

the first is a boot with init=/bin/bash, the second is your fix, the third is the boot after the fix hanging now even before the *random* stuff.

newkernel+initbash.txt

[zador.blood.stained]

Boot #3 hangs exactly in the same place, I don't have "random" line on my 4.2.3 kernel

Boot #2 is interesting. It starts bash process. Did you try typing something to serial console or pressing Enter?

As for #1 - this is strange.

 

Edit: Also for #2 - try removing /etc/bash.bashrc.custom

 

Edit 2: Sorry, missed that you did change time, looks like boot #3 has correct time. Remove init=/bin/bash from you boot script and try booting again.

[zador.blood.stained]

Posting again just so you get new message notification - if you did change time, you don't need init=/bin/bash in your boot script. Remove it and try normal boot.

[EvIl_DeViL]

ok I removed it but no luck. Only thing new is that it came back hanging to *random*

It seems strange that there's no log or a way to debug this o.o

maybe there's a way to block all the non-vital systemd stuff and adding them back one by one?

 

p.s. I saw you removed the stuff about /etc/bash.bashrc.custom but what's this for? and why that could have helped?

newKernel2.txt

[zador.blood.stained]

Try adding some extra command line parameters from this page: https://wiki.freedesktop.org/www/Software/systemd/Debugging/

Start with systemd.log_level=debug systemd.log_target=console systemd.journald.forward_to_console=1

 

Edit: Okay, for some reason you again have wrong time. Maybe your RTC battery is dead? Or do you actually have RTC battery? Checked: you don't.

I was right earlier:

...or if I'm missing something about your board compared to mine (cubietruck).

Now that I re-think all this, you don't have issue with time in future, that affects systemd. I should have noticed it earlier, only old kernel resets time to 1970-01-01, mainline would print 2085-something-something.

[zador.blood.stained]

Okay, now, assuming that you have some kind of filesystem corruption, that affects your init related files, there are two options:

• Easy - extract needed config files from your image and start from scratch with new image,
• Hard - try to repair this image.

[EvIl_DeViL]

Thing is this happened a couple of times yet with this board and I went for the easy way twice; but I don't like when computers win so this time I tried the hard way but without luck. I don't like that out of the blue this board would betray me once more.

This time I'll keep my eyes wide open and I'll reboot every install I do to see if the system works and I will find the unholy package.

[zador.blood.stained]

Ok. My suggestion is - make backups. No. Make backups. Still no.  Make incremental backups before updating any critical packages.

Now that I think... if it fails to boot after upgrade - check if you have /sbin/init, check if it is a symlink and it points to /lib/systemd/systemd and check size and md5sum of /lib/systemd/systemd. Maybe you can still win easily 

I didn't ask what is your OS version, but I assume it is Debian Jessie and init system is systemd.

[EvIl_DeViL]

yup but everything on *nix could be addressed as critical... even changing the desktop's background XD

 

Anyway I have Ubuntu Trusty on the board and although I have the /sbin/init file it says it's a "shared library (application/x-sharedlib)" and not a symlink to /lib/systemd/systemd which actually doesn't even exists.

maybe trusty's using upstart?

$/lib/systemd$ ls -l
total 536
drwxr-xr-x 8 root root   4096 set 28 13:41 system
-rwxr-xr-x 1 root root  50868 set 11 13:35 systemd-hostnamed
-rwxr-xr-x 1 root root  54872 set 11 13:35 systemd-localed
-rwxr-xr-x 1 root root 177736 set 11 13:35 systemd-logind
-rwxr-xr-x 1 root root  21992 set 11 13:35 systemd-multi-seat-x
-rwxr-xr-x 1 root root  59032 set 11 13:35 systemd-timedated
-rwxr-xr-x 1 root root 165536 set 11 13:35 systemd-udevd

did we found the culprit? *_*

[zador.blood.stained]

Wiki says upstart is default before 15.04, so then it's probably upstart problem

http://upstart.ubuntu.com/wiki/Debugging

[Igor]

@Evil_Devil

 

At http://mirror.igorpecovnik.com/test/ you will find (wait for update - any minute) a kernel 4.3 (Armbian 4.6), DTB and a u-boot for Lamobo R1. Uboot USB issues are gone, the rest need to be tested. It's a first test build. 

 

Give it a try.

[EvIl_DeViL]

Hi I have updated to kernel 4.3 armbian 4.6 manually but I still can't boot.

I checked a "virgin" flashed sd with armbian, and /sbin/init is a simple file and not a symlink to /lib/systemd/systemd

 

I feel all my hopes flied away once again...

kernel_43.txt

[zador.blood.stained]

Ok. Last more or less sane idea is - install debsums package with its dependencies and running "debsums -cs" from bash shell. Debsums should detect any corrupted files (apart from config) if there are any (and it will probably whine about manually overwritten kernel related files).

Since I didn't find debsums on ports.ubuntu.com for trusty It is in Universe repository. Alternatively you can run it from another Ubuntu or Debian installation and just point to your rootfs and admindir.

If it doesn't output anything useful, then it means probably broken config files or missing packages, and unless you want to poke random config files looking for corruption signs, it would be easier to start from scratch with fresh image.

I'm assuming you tried adding --verbose to kernel command line as suggested on upstart debugging page and it didn't output anything new. Also that you ran fsck.ext4 -f for your sd card partition and it didn't output any errors.

 

Edit: earlier I extracted upstart package for trusty

➜  sbin  % file init
init: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=0bc26f5010188b174e1170c23ec32354b6e9e21c, stripped

Edit 2: You can run "debsums -acs" to output also changed (compared to default) config files, theoretically it might help, if you remember what you did or did not modify.

Edit 3: You may need to mount /proc and remount rootfs to rw in bash shell before running debsums.

mount proc /proc -t proc
mount -o remount,rw /