[ODROID C2] Debian Buster SSH connection refused for long time after boot

[zero_derivative]

Greetings.

 

I'm running a headless cluster of Odroid C2's (eMMC) using the latest Armbian buster minimal image with kernel 4.19.69-meson64. I'm encountering an issue where it takes a long time for SSH connection to be accepted (initially up to 30 min consistently). I get a connection refused.

 

The issue seems to be due to the entropy pool becoming depleted during the early boot process which blocks SSH from starting while it refills the pool.

The problem seems to be known and documented well here: 

https://daniel-lange.com/archives/152-Openssh-taking-minutes-to-become-available,-booting-takes-half-an-hour-...-because-your-server-waits-for-a-few-bytes-of-randomness.html

 

As suggested I have installed installed haveged which brings the SSH startup from 30 mins to 10 mins after boot, which is an improvement, but still not good.

 

Has anyone else experienced this issue? Any ideas?

 

[guidol]

Yes - I can confirm that the C2 take some time before I can ssh into it... but with normal armbian

 

Debian Buster with Armbian Linux 5.3.0-meson64
package bsp-kernel[5.97.190917] u-boot[5.96] dtb[5.97.190917] firmware[5.96] config[5.96]

 

it took me around 2-3 minutes.

 

After your info about the entropy pool I did found firstly the following page which also suggest havegd:

https://daniel-lange.com/archives/152-Openssh-taking-minutes-to-become-available,-booting-takes-half-an-hour-...-because-your-server-waits-for-a-few-bytes-of-randomness.html

 

But this on my mind,  I compared my NanoPi Neo2 with the C2

 

NanoPi Neo2:

dmesg | grep -E "(rng|random)"

[    0.000000] random: get_random_bytes called from start_kernel+0x2e4/0x478 with crng_init=0
[    5.888987] random: fast init done
[    7.924082] random: systemd: uninitialized urandom read (16 bytes read)
[    7.933060] random: systemd: uninitialized urandom read (16 bytes read)
[    7.945521] random: systemd: uninitialized urandom read (16 bytes read)
[   11.610613] random: crng init done
[   11.610625] random: 7 urandom warning(s) missed due to ratelimiting

Odroid C2:

dmesg | grep -E "(rng|random)"

[    0.000000] random: get_random_bytes called from start_kernel+0x2f4/0x488 with crng_init=0
[    4.708300] random: fast init done
[    6.124106] random: systemd: uninitialized urandom read (16 bytes read)
[    6.131317] random: systemd: uninitialized urandom read (16 bytes read)
[    6.132394] random: systemd: uninitialized urandom read (16 bytes read)
[   84.643984] random: crng init done
[   84.643999] random: 7 urandom warning(s) missed due to ratelimiting

So a hugh difference between 11 and 84 in counting,

 

As I only got this problem on the Odroid C2 (Amlogic S905) and my Sunvell T95K Pro (Amlogic S912) I searched for a

Amlogic-CPU-Solution and did found the following for the Ordoid C1 (Amlogic S805):

 

[FIXED] Random Number Generator on odroid-c1 ==> Hardware Random Number Generator Accelerator

https://forum.odroid.com/viewtopic.php?f=115&t=8874

https://odroid.com/dokuwiki/doku.php?id=en:c1_hardware_number_generator

 

You have to install rng-tools:

apt-cache search rng-tools
rng-tools - Daemon to use a Hardware TRNG
rng-tools-debian - daemon to use a Hardware TRNG (classic version)
rng-tools5 - Daemon to use a Hardware TRNG

sudo apt-get install rng-tools

After I did install the rng-tools I could immediately ssh into my C2 then the /etc/rc.local was processed

(do get a voice info on my system).

 

The time to fill the entropy pool is now with 12 as short as on the NanoPi Neo2

dmesg | grep -E "(rng|random)"
[    0.000000] random: get_random_bytes called from start_kernel+0x2f4/0x488 with crng_init=0
[    4.684192] random: fast init done
[    6.221509] random: systemd: uninitialized urandom read (16 bytes read)
[    6.229215] random: systemd: uninitialized urandom read (16 bytes read)
[    6.230358] random: systemd: uninitialized urandom read (16 bytes read)
[   12.413199] random: crng init done
[   12.413207] random: 7 urandom warning(s) missed due to ratelimiting

@zero_derivative Thanks for the info about the entropy pool!
@Igor maybe rng-tools should be an default installed packet on amlogic-devices?

@balbes150 maybe also some users of your Amlogic-images would like this

[Igor]

2 hours ago, guidol said:

maybe rng-tools should be an default installed packet on amlogic-devices?

 

Good catch!

We can afford 130Kb of extra size. I would propose to add it to the base set https://github.com/armbian/build/blob/master/lib/configuration.sh#L150 Send a PR, but we need few tests on other devices to see if this package does no harm?

Also make this number +1

https://github.com/armbian/build/blob/master/lib/configuration.sh#L23

[guidol]

7 hours ago, Igor said:

 

Good catch!

Send a PR, but we need few tests on other devices to see if this package does no harm?

Also make this number +1

https://github.com/armbian/build/blob/master/lib/configuration.sh#L23

Sorry - I dont understand how to make a PR

I did read the page https://guides.github.com/activities/hello-world/#pr

but it seems I need a Repository/Branch/Commit to create a PR and all the Screenshots sens me in a circle/bootloop
First when I selected new PR then I had to select 2 branches to compare, but I only want to add a solution....

Should I add a Issue and write the solution inside? (that worked for me in the past)

I also doesnt know the meaning of making this number +1

I only know +1 from Yahoo or the gone GooglePlus

[Igor]

12 minutes ago, guidol said:

Sorry - I dont understand how to make a PR

This is a very good simple example to start with. Perhaps this is wrote better: https://www.armbian.com/get-involved/#submit > Opening a pull request ...

 

Rather go this way.

1. Open an (NEW) issue "SSH takes very long time", label it "Improvement" and make a quick symptom description, link to forum topic https://github.com/armbian/build/issues

2. Create a pull request with title "Fixing ssh login time delay" and inside comments put: "Closes LINK_TO_THAT_ISSUE"

 

• Pull request procedure: 
• 1. Fork the project
• 2. Edit things:
•  - add package rng-tools here https://github.com/armbian/build/blob/master/lib/configuration.sh#L150
•  - change number from 11 to 12 here https://github.com/armbian/build/blob/master/lib/configuration.sh#L23
• 3. Create a pull request

 

Try! 

[zero_derivative]

FWIW My boot delays were much more drastic and consistent across 6 Odroid C2's initially using the minimal image ...

dmesg | grep -E "(rng|random)"

[    0.000000] random: get_random_bytes called from start_kernel+0xa8/0x400 with crng_init=0
[    3.074316] random: fast init done
[    4.216928] random: systemd: uninitialized urandom read (16 bytes read)
[    4.220823] random: systemd: uninitialized urandom read (16 bytes read)
[    4.221479] random: systemd: uninitialized urandom read (16 bytes read)
[ 1846.057161] random: crng init done
[ 1846.057175] random: 7 urandom warning(s) missed due to ratelimiting

My Odroid C2's also show a ridiculously low amount (single to low double digits) of available entropy without haveged and the rate at which available entropy increases is very very very slow ...

cat /proc/sys/kernel/random/entropy_avail
5

 

With haveged it's marginally better, and the rate at which available entropy increases is better than without out it, but still too slow  ...

cat /proc/sys/kernel/random/entropy_avail
77

 

With rng-tools installed things are looking much better ...

cat /proc/sys/kernel/random/entropy_avail
2002

and now SSH is available much quicker ...

dmesg | grep -E "(rng|random)"
[    0.000000] random: get_random_bytes called from start_kernel+0xa8/0x400 with crng_init=0
[    2.954319] random: fast init done
[    4.298379] random: systemd: uninitialized urandom read (16 bytes read)
[    4.308032] random: systemd: uninitialized urandom read (16 bytes read)
[    4.309349] random: systemd: uninitialized urandom read (16 bytes read)
[   10.320571] random: crng init done
[   10.320579] random: 7 urandom warning(s) missed due to ratelimiting

Awesome!

 

Thanks @guidol for the rng-tools tip

[guidol]

19 minutes ago, Igor said:

 

3. Create a pull request

 

Try! 

OK - I did try my very best
https://github.com/armbian/build/pull/1566

[Igor]

4 minutes ago, guidol said:

OK - I did try my very best

Great! See, it was not that hard. 

[guidol]

15 minutes ago, Igor said:

Great! See, it was not that hard. 

for me as first time user it was "hard" - I did try it and it worked - BUT I didnt understand all steps completly