Technicavolous Posted July 4, 2016 Posted July 4, 2016 Since I had a power supply problem with my XU4 I decided to download the image and start from scratch. First time I recall everything went smooth until my PS failed. Now, starting from a fresh image, when the first boot gets to SSH key regeneration, it hangs forever. Tried several times, tested the sd card, used a different one for test. Still hangs at SSH key regeneration. I had never tried moving the OS to the HD. All the contents there are good and it has passed all the long smart tests. If I reboot after a while hanging at SSH key regeneration it boots and does all the password stuff as expected, and everything else seems fine. Should I worry about it?
wildcat_paris Posted July 4, 2016 Posted July 4, 2016 @Technicavolous ok, I have the reply from the other thread so your hdd is ok btw, you are using the legacy version (kernel 3.x) because vanilla (kernel 4.x) is highly experimental. - can you move your SDcard and do a sudo fsck -s /dev/something1 and something2 can you unplug the HDD for a moment? - try to boot from another "fresh" SDcard. the SSH key generation stalled can come from a broken SD card (broken with the other power supply) or more simply by a lack of random data from /dev/random (if empty it can stall the generation, so try to move the mouse, press some keys from keyboard) also the Wifi plug (as host) can draw random data
wildcat_paris Posted July 4, 2016 Posted July 4, 2016 (edited) and plugging the ethernet cable can bring interruptions and random data to the pool from another thread... http://forum.armbian.com/index.php/topic/1436-boot-hangs-at-random-nonblocking-pool-is-initialized/?p=12071 Edited July 4, 2016 by wildcat_paris updated link
Technicavolous Posted July 5, 2016 Author Posted July 5, 2016 Tried two different brand new 8GB cards (I bought a bunch of 'em ) one that I used as is and one that I did SDFormatter with size readjust. Same issue. Disconnected hard drive, same issue. I did notice that I've been trying this with 5.14 and the first run I used 5.10. I have a very slow connection so I'm redownloading 5.14 and pulling out the 1TB drive that I have the 5.10 image archived on. I'll see if I have a bad download or track down whatever I'm doing wrong. In the meantime I burned a couple of other images and they've worked fine.Usually when something doesn't work I format and start over with good results. This is persistent. When 5.14 completes download (hours ...) I'll report back. BTW I'm using Rufus on win7 to burn the images to SD, IIR this is the preferred way, no? Thanks again.
wildcat_paris Posted July 5, 2016 Posted July 5, 2016 @Technicavolous yes rufus is recommanded by the Armbian team (I only use dd / dc3dd on linux based systems) 7zip would have detected an error during decompression ok, tell us when you have pieces of news please
Technicavolous Posted July 8, 2016 Author Posted July 8, 2016 Sorry for the delay, that work stuff ;] I am unable to get past this issue. Redownloaded, tried just the board and no cloudshell, tried different SD cards, tried with no wifi dongle, put an OScope on the power lol it's clean ... But it always hangs at SSH ke regeneration. Everything else works fine after I force a reboot, I cannot find any malfunction or error. Since it hangs stating it's making SSH keys, I imagine I should go learn a little SSH linux and find out if it is indeed making the key and hanging on the next step? I am able to ssh into the odroid after reboot so I'm not so sure what's actually happening, or if it even matters. You know how it is, something that isn't 'right' just bugs ya. Regardless, I'm moving forward as if it were not a problem. This forum is awesome - I've learned more reading posts here than just about anywhere else. I'll definitely be a regular.
wildcat_paris Posted July 8, 2016 Posted July 8, 2016 @Technicavolous I was thinking of "haveged" package which provides some entropy from CPU internal registers a bit like recent linux kernels with jitter_entropy it seems haveged package is included in Armbian images for a while. What I don't know: is haveged started before ssh key generation? So, yes, if you can access using SSH during key generation (or a UART console) to check how the entropy pool is low or not cat /proc/sys/kernel/random/entropy_avail so we have a clue if it is the low entropy pool that is the source of the problem OR something different in the booting process.
zador.blood.stained Posted July 8, 2016 Posted July 8, 2016 Since it hangs stating it's making SSH keys, I imagine I should go learn a little SSH linux and find out if it is indeed making the key and hanging on the next step? I am able to ssh into the odroid after reboot so I'm not so sure what's actually happening, or if it even matters. Actually several tasks are executed at this point: Creating SSH keys Creating swap file Adjusting swap related parameters Tweaking ramlog if it is installed There should be some messages in /var/log/armhwinfo.log regarding some steps. Are you sure that it completely hangs? Can you switch to other TTYs (i.e. to tty2 by pressing Ctrl-Alt-F2) and login from there if a prompt is available? 1
Technicavolous Posted July 9, 2016 Author Posted July 9, 2016 So it gives [OK] resizing filesystem, [OK] creating 128 mb swap, [OK] starting ssh keys regeneration and then "hangs" but @wildcat - no I can't ssh in @zador - yes indeed can ctrl-alt-f2 and log in on another tty! I should have thought of that thanks! It does what is expected and has me change the root password and add the user, then it starts the desktop. After the desktop starts if I ctrl-alt-f1 it still shows the 'hung' start screen on tty1. @wildcat I switched to tty3 and the output of the cat you asked for is simply 1814 When I went back to tty2 where the desktop was, it had the root prompt. So I typed reboot and the desktop appeared for a brief flash and the unit rebooted nicely back to the desktop. I think that flash of the desktop was video memory. When it rebooted I did the entropy cat command again and it returned 4089. I've redone this thing so many times it's funny and every time exactly the same. My gut says everything is fine but if you guys want any other data just let me know I'll be happy to post it. I've been using this last install long enough I'm going to go ahead and move it to the hard drive and see how that goes. This one is going to become a regular tool on the desk. I'm going to have a little fun with Armbian on my Orange Pi next, :] A million thanks, there will most likely be more questions! Tracy
Technicavolous Posted July 9, 2016 Author Posted July 9, 2016 BTW apt-get update and apt-get upgrade to 5.16 went fine ;]
zador.blood.stained Posted July 9, 2016 Posted July 9, 2016 I've redone this thing so many times it's funny and every time exactly the same. My gut says everything is fine but if you guys want any other data just let me know I'll be happy to post it. I've been using this last install long enough I'm going to go ahead and move it to the hard drive and see how that goes. Please try this: Flash a new image Boot it and wait until it hangs Log in from tty2 When it asks to create a new user, press Ctrl-C Execute "top -n 1 >> /root/log.txt" and "pstree >> /root/log.txt" Provide here contents of created log.txt This should help in understanding what doesn't work
Technicavolous Posted July 9, 2016 Author Posted July 9, 2016 OK here is the contents of log.txt #[?25l#[?1c#[H#[J#[0;10mtop - 19:24:59 up 6 min, 1 user, load average: 0.01, 0.08, 0.05#[0;10m#[39;49m#[0;10m#[39;49m#[K Tasks:#[0;10m#[39;49m#[1m 128 #[0;10m#[39;49mtotal,#[0;10m#[39;49m#[1m 1 #[0;10m#[39;49mrunning,#[0;10m#[39;49m#[1m 127 #[0;10m#[39;49msleeping,#[0;10m#[39;49m#[1m 0 #[0;10m#[39;49mstopped,#[0;10m#[39;49m#[1m 0 #[0;10m#[39;49mzombie#[0;10m#[39;49m#[0;10m#[39;49m#[K %Cpu(s):#[0;10m#[39;49m#[1m 0.6 #[0;10m#[39;49mus,#[0;10m#[39;49m#[1m 0.9 #[0;10m#[39;49msy,#[0;10m#[39;49m#[1m 0.0 #[0;10m#[39;49mni,#[0;10m#[39;49m#[1m 97.8 #[0;10m#[39;49mid,#[0;10m#[39;49m#[1m 0.7 #[0;10m#[39;49mwa,#[0;10m#[39;49m#[1m 0.0 #[0;10m#[39;49mhi,#[0;10m#[39;49m#[1m 0.0 #[0;10m#[39;49msi,#[0;10m#[39;49m#[1m 0.0 #[0;10m#[39;49mst#[0;10m#[39;49m#[0;10m#[39;49m#[K KiB Mem: #[0;10m#[39;49m#[1m 2042792 #[0;10m#[39;49mtotal,#[0;10m#[39;49m#[1m 363056 #[0;10m#[39;49mused,#[0;10m#[39;49m#[1m 1679736 #[0;10m#[39;49mfree,#[0;10m#[39;49m#[1m 7616 #[0;10m#[39;49mbuffers#[0;10m#[39;49m#[0;10m#[39;49m#[K KiB Swap:#[0;10m#[39;49m#[1m 131068 #[0;10m#[39;49mtotal,#[0;10m#[39;49m#[1m 0 #[0;10m#[39;49mused,#[0;10m#[39;49m#[1m 131068 #[0;10m#[39;49mfree.#[0;10m#[39;49m#[1m 108312 #[0;10m#[39;49mcached Mem#[0;10m#[39;49m#[0;10m#[39;49m#[K #[K #[7m PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND #[0;10m#[39;49m#[K #[0;10m#[1m 1337 root 20 0 4684 1240 800 R 15.2 0.1 0:00.09 top #[0;10m#[39;49m#[K #[0;10m 1 root 20 0 5188 3112 1480 S 0.0 0.2 0:07.28 systemd #[0;10m#[39;49m#[K #[0;10m 2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd #[0;10m#[39;49m#[K #[0;10m 3 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/0 #[0;10m#[39;49m#[K #[0;10m 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H #[0;10m#[39;49m#[K #[0;10m 6 root 20 0 0 0 0 S 0.0 0.0 0:00.10 kworker/u16:0 #[0;10m#[39;49m#[K #[0;10m 7 root rt 0 0 0 0 S 0.0 0.0 0:00.01 migration/0 #[0;10m#[39;49m#[K #[0;10m 8 root 20 0 0 0 0 S 0.0 0.0 0:00.16 rcu_preempt #[0;10m#[39;49m#[K #[0;10m 9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh #[0;10m#[39;49m#[K #[0;10m 10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_sched #[0;10m#[39;49m#[K #[0;10m 11 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 #[0;10m#[39;49m#[K #[0;10m 12 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 #[0;10m#[39;49m#[K #[0;10m 13 root rt 0 0 0 0 S 0.0 0.0 0:00.01 migration/1 #[0;10m#[39;49m#[K #[0;10m 14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 #[0;10m#[39;49m#[K #[0;10m 15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0 #[0;10m#[39;49m#[K #[0;10m 16 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H #[0;10m#[39;49m#[K #[0;10m 17 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 #[0;10m#[39;49m#[K #[0;10m 18 root rt 0 0 0 0 S 0.0 0.0 0:00.01 migration/2 #[0;10m#[39;49m#[K #[0;10m 19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 #[0;10m#[39;49m#[K #[0;10m 21 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0H #[0;10m#[39;49m#[K #[0;10m 22 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 #[0;10m#[39;49m#[K #[0;10m 23 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/3 #[0;10m#[39;49m#[K #[0;10m 24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 #[0;10m#[39;49m#[K #[0;10m 26 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0H #[0;10m#[39;49m#[K #[0;10m 27 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 #[0;10m#[39;49m#[K #[0;10m 28 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/4 #[0;10m#[39;49m#[K #[0;10m 29 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/4 #[0;10m#[39;49m#[K #[0;10m 31 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:0H #[0;10m#[39;49m#[K #[0;10m 32 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 #[0;10m#[39;49m#[K #[0;10m 33 root rt 0 0 0 0 S 0.0 0.0 0:00.01 migration/5 #[0;10m#[39;49m#[K #[0;10m 34 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/5 #[0;10m#[39;49m#[K #[0;10m 36 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/5:0H #[0;10m#[39;49m#[K #[0;10m 37 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/6 #[0;10m#[39;49m#[K #[0;10m 38 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/6 #[0;10m#[39;49m#[K #[0;10m 39 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/6 #[0;10m#[39;49m#[K #[0;10m 41 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/6:0H #[0;10m#[39;49m#[K #[0;10m 42 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/7 #[0;10m#[39;49m#[K #[0;10m 43 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/7 #[0;10m#[39;49m#[K #[0;10m 44 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/7 #[0;10m#[39;49m#[K #[0;10m 45 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/7:0 #[0;10m#[39;49m#[K #[0;10m 46 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/7:0H #[0;10m#[39;49m#[K #[0;10m 47 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper #[0;10m#[39;49m#[K #[0;10m 48 root 20 0 0 0 0 S 0.0 0.0 0:00.06 kdevtmpfs #[0;10m#[39;49m#[K #[0;10m 49 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns #[0;10m#[39;49m#[K #[0;10m 50 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback #[0;10m#[39;49m#[K #[0;10m 51 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset #[0;10m#[39;49m#[K #[0;10m 52 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto #[0;10m#[39;49m#[K #[0;10m 53 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd #[0;10m#[39;49m#[K #[0;10m 54 root 20 0 0 0 0 S 0.0 0.0 0:00.04 khubd #[0;10m#[39;49m#[K #[0;10m 58 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/512-s2mps11 #[0;10m#[39;49m#[K #[0;10m 59 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/4:1 #[0;10m#[39;49m#[K #[0;10m 72 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ion_noncontig_h #[0;10m#[39;49m#[K #[0;10m 73 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 devfreq_wq #[0;10m#[39;49m#[K #[0;10m 74 root 20 0 0 0 0 S 0.0 0.0 0:00.26 kworker/5:1 #[0;10m#[39;49m#[K #[0;10m 75 root 20 0 0 0 0 S 0.0 0.0 0:00.26 kworker/0:1 #[0;10m#[39;49m#[K #[0;10m 76 root 20 0 0 0 0 S 0.0 0.0 0:00.12 kworker/2:1 #[0;10m#[39;49m#[K #[0;10m 77 root 20 0 0 0 0 S 0.0 0.0 0:00.10 kworker/3:1 #[0;10m#[39;49m#[K #[0;10m 78 root 20 0 0 0 0 S 0.0 0.0 0:00.18 kworker/1:1 #[0;10m#[39;49m#[K #[0;10m 79 root 20 0 0 0 0 S 0.0 0.0 0:00.09 kworker/6:1 #[0;10m#[39;49m#[K #[0;10m 80 root 20 0 0 0 0 S 0.0 0.0 0:00.10 kworker/7:1 #[0;10m#[39;49m#[K#[68;1H #[?25h#[?0c#[Ksystemd-+-2*[agetty] |-cron |-dbus-daemon |-dhclient |-haveged |-login---bash---pstree |-ntpd---ntpd |-rsyslogd-+-{in:imklog} | |-{in:imuxsock} | `-{rs:main Q:Reg} |-sshd |-systemd---(sd-pam) |-systemd-journal |-systemd-logind |-systemd-udevd `-wicd---wicd-monitor What docs should I be looking at to understand this output? At this point I CAN ssh in. When I do, putty pops up a message about the host key being different than the one cached, which seems like a good message. putty or any other ssh works fine. It prompted me to add the user and everything else went great. I am posting this message from armbian before upgrading to 5.16. Which, BTW, last time appeared to go well. I am saving this particular SD for consistency but I have another one I'm willing to start over with as many times as you fellows wish. If you need photos I've finally figured that out also. Cameras ... log.txt
zador.blood.stained Posted July 9, 2016 Posted July 9, 2016 All processees in firstrun script were definitely finished. Did you try pressing Enter after SSH keys message? These messages may appear AFTER prompt or overwrite it on screen.
Technicavolous Posted July 9, 2016 Author Posted July 9, 2016 Well, with a giant capitalized message stating to be patient and not interrupt the first boot, no, I didn't touch anything for quite a while ;] With a fresh image booted I waited just a few moments and hit enter, and there, yes a prompt. Waited for a moment, then logged in as root and it took me through the user data and started the desktop. Connected to my wifi, ran an update and upgrade, and all went well. Lather, rinse, repeat, did this several times consistently. It's hypercritical to hit the enter key at what appears to be a hang ... embarrassing but I really thought I had to wait! Wile I'm no guru, I'm not new to linux, I really should have tried something so simple as hitting the enter key. BUT, on the ride I learned a heck of alot of what's happening with armbian boot, and you guys made figuring out what I thought was a problem, fun. And at least I know all my hardware is good lol. I'm certainly on my way to learning armbian, and you've given me the confidence to stick with this one and 'get good at it.' Thanks!
wildcat_paris Posted July 10, 2016 Posted July 10, 2016 It's hypercritical to hit the enter key at what appears to be a hang ... embarrassing but I really thought I had to wait! Wile I'm no guru, I'm not new to linux, I really should have tried something so simple as hitting the enter key. Have you tried cat /proc/sys/kernel/random/entropy_avail because either the script waits for "enter" or pushing enter created an interrupt that feeds random pool we cannot say which solution is correct.
Technicavolous Posted July 10, 2016 Author Posted July 10, 2016 Each time I've run that it gives me a 4 digit number. Even when I broke out of adding a user Just now it gave me 4096.
wildcat_paris Posted July 10, 2016 Posted July 10, 2016 @Technicavolous ok, so the random pool is full (not depleted). we can close this kind of problem thanks for the feedback note: usually /proc/sys/kernel/random/poolsize=4096 so max entropy_avail=4095 but values > 4000 are barely seen as each new process in the system use data from the entropy pool in more recent linux kernels
Technicavolous Posted July 11, 2016 Author Posted July 11, 2016 Every time I check entropy it is a 4 digit number, but sometimes is less than 4096, one time was 1200 something. I never even knew 'entropy' was part of the kernel. I thought it was space and time but that's another forum ;] 1
wildcat_paris Posted July 11, 2016 Posted July 11, 2016 @Technicavolous we could ask Igor to add a "Space and Time" section in forum.armbian.com and then we call the Men-In-Black to clean the thread. the random pool is so important as it is so needed in the symmetric ciphers (to reduce replay attacks), to generate private keys for asymmetric stuff and to randomize memory space / process to counter stuff like buffer overflow attacks empty random pool = unsafe system (embedded systems often randomness/entropy is lacking) I feel like a grandpa writing this
Recommended Posts