Jump to content

Hanging on kernel load after Shutdown/Reboot/Kernel panic

Lore

By Lore
in Rockchip

 Share

Recommended Posts

Lore

Lore

Posted
Posted

I have run into this issue multiple times and the process flow is this:

 

Flash image "Armbian_20.11.6_Helios64_buster_current_5.9.14.img" to a micro SD card.

Run through the config (installing OMV with SAMBA)

Kick over to the OMV to configure the share / media so its available on my network

Runs for some time (this last install was running for maybe a few weeks before it triggered a Kernel panic)

Power off.

Power on.

Board boots, hangs on "Starting Kernel..."

 

Is there some trick I am missing to get this thing to actually boot / reboot correctly?

 

Troubleshooting info:

No jumpers are shorted.

I had a similar issue with a previous img as well.

1
aprayoga

aprayoga

Posted
Posted

@Lore, do you have other Linux system ?
please modify /boot/armbianEnv.txt on the micro SD card and add/modify following lines: 

verbosity=7
console=serial
extraargs=earlyprintk ignore_loglevel

 

It should make the serial console more verbose

Lore

Lore

Posted
  • Author
Posted

Yeah, I have Ubuntu running as a Console connection to it.  Ill give that a shot and post the logs when I get a chance. Thanks.

Lore

Lore

Posted
  • Author
Posted

I was able to get it booting.  The following is the error that was causing it to hang.

  

Begin: Running /scripts/local-premount ... Scanning for Btrfs filesystems
done.
Begin: Will now check root file system ... fsck from util-linux 2.33.1
[/sbin/fsck.ext4 (1) -- /dev/mmcblk0p1] fsck.ext4 -a -C0 /dev/mmcblk0p1 
/dev/mmcblk0p1: recovering journal
/dev/mmcblk0p1 contains a file system with errors, check forced.
/dev/mmcblk0p1: Inode 753 seems to contain garbage.  

/dev/mmcblk0p1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
	(i.e., without -a or -p options)
fsck exited with status code 4
done.
Failure: File system check of the root filesystem failed
The root filesystem on /dev/mmcblk0p1 requires a manual fsck

 

I then just ran 

fsck.ext4 /dev/mmcblk0p1

 

 

After that it ran and booted to OMV on its own.

1
Lore

Lore

Posted
  • Author
Posted

So, I came down tonight to the system in the same state, halted on the FSCK.

 

Seems to have booted fine after I manually ran it, but if it reboots every few days and I have to go do the FSCK every time, I might actually cry.

aprayoga

aprayoga

Posted
Posted

@Lore you could add fsck.repair=yes to extraargs line on /boot/armbianEnv.txt

  

extraargs=earlyprintk ignore_loglevel fsck.repair=yes

it will run

fsck.ext4 -y /dev/mmcblk0p1

automatically.

 

---

You have another issue (I'd say bigger issue), why you often got files system corruption.

Does it always happen after a reboot?

Could you monitor/log serial console during reboot ? maybe there are some processes that killed forcefully by systemd because missing the timeout to shutdown

 

 

Lore

Lore

Posted
  • Author
Posted

The corruption seems to happen over time, if I leave it run for a few days it will end up rebooting itself and Ill have to FSCK again before it will boot normally. 

 

I just did a FSCK, so I can try to reboot manually but I dont think that will show me any corruption.  If I leave it to do its thing with the console up and monitoring I can hopefully capture the output and Ill update.

Lore

Lore

Posted
  • Author
Posted

So, it rebooted itself.  Here is the reboot output.

 

  Reveal hidden contents
aprayoga

aprayoga

Posted
Posted
  On 2/4/2021 at 3:26 PM, Lore said:

Here is a pastebin of me just pressing the reset button.  Full unedited output.

 

https://pastebin.com/jB1xJjLE

 

It booted normally but I did notice that there was a FSCK that happened.

Expand  

yes, it is expected since pressing the reset button will force the system reset and skip graceful shutdown procedure, which one of the step flaggging the filesystem as clean. therefore next boot will read the filesystem dirty and do fsck to checck if there is any corruption.
 

  On 2/5/2021 at 2:00 PM, Lore said:

So, it rebooted itself.  Here is the reboot output.

 

  Reveal hidden contents
Expand  

Have you tried to limit the cpu speed and governor?

Change CPU governor to Performance, and optionally reduce the CPU speed to keep the system run cool.

 

armbian-config > System > CPU

 

Minimum CPU speed = 1200000

Maximum CPU speed = 1200000

CPU governor = performance

Lore

Lore

Posted
  • Author
Posted

I have the full enclosure version so I have fans and such and I have not noticed any excess heat.

 

I will give it a shot though.

Lore

Lore

Posted
  • Author
Posted

I cant get into armbian-config, my account is locked somehow and my bad password attempts don't seem to be timing out.

SIGSEGV

SIGSEGV

Posted
Posted

@aprayoga, @gprovost,

One thing that has worked for me since I got the board around on Nov of last year is to install package 'tuned' and set the profile to 'server-powersave'.

The two BIG cores can go up to 1.8GHz and the little cores go up to 1.4GHz under heavy usage without any lockups - most of the time in idle they stay at 408MHz.

  

apt -y install tuned
tuned-adm profile server-powersave
systemctl restart tuned

 

SIGSEGV

SIGSEGV

Posted
Posted
  On 2/9/2021 at 7:08 PM, snakekick said:

hello @SIGSEGV tuned is only a another CPU Scaling Governor or?

do you have problems before you switch to tuned and this solve your problems ?

 

Expand  

I did have a few lockups when running my setup scripts that actually install a few of the packages I needed/wanted to run on the Helios64. It would lock sometimes while idle.

'tuned' was one of the packages that is installed on our cloud environments and some options are available through the cockpit management interface. I wouldn't describe 'tuned' as a CPU Scaling Governor.

The one thing I noticed was that if configured tuned in the early steps of installation - my scripts would finish running their task and I didn't have lockups (if tuned was not configured early in the process it would freeze at random points).

 

tuned project - their website description is as follows 

Tuned is a system tuning service for Linux. It:

monitors connected devices using the udev device manager
tunes system settings according to a selected profile
supports various types of configuration like sysctl, sysfs, or kernel boot command line parameters, which are integrated in a plug-in architecture
supports hot plugging of devices and can be controlled from the command line or through D-Bus, so it can be easily integrated into existing administering solutions: for example, with Cockpit
can be run in no-daemon mode with limited functionality (for example, no support for D-Bus, udev, tuning of newly created processes, and so on) for systems with reduced resources
stores all its configuration cleanly in one place – in the Tuned profile – instead of having configuration on multiple places and in custom scripts

 

Take my word with a grain of salt - this is only my experience with the Helios64.

 

 

Lore

Lore

Posted
  • Author
Posted
  On 2/9/2021 at 3:17 AM, gprovost said:

Are you trying to ssh with root user ?

Expand  

I have tried with both root and the account I created at install.  Both give the same reject response.  Using both SSH and Console.

 

  On 2/9/2021 at 3:33 PM, SIGSEGV said:

@aprayoga, @gprovost,

One thing that has worked for me since I got the board around on Nov of last year is to install package 'tuned' and set the profile to 'server-powersave'.

The two BIG cores can go up to 1.8GHz and the little cores go up to 1.4GHz under heavy usage without any lockups - most of the time in idle they stay at 408MHz.

  


apt -y install tuned
tuned-adm profile server-powersave
systemctl restart tuned

 

Expand  

Ill give Tuned a shot... If I can ever get logged back in.

gprovost

gprovost

Posted
Posted
  On 2/10/2021 at 2:03 AM, Lore said:

I have tried with both root and the account I created at install.  Both give the same reject response.  Using both SSH and Console.

Expand  

 If you can still connect to OMV web portal, then go check your created user and be user they right to connect to SSH.

Lore

Lore

Posted
  • Author
Posted
  On 2/10/2021 at 3:31 AM, gprovost said:

 If you can still connect to OMV web portal, then go check your created user and be user they right to connect to SSH.

Expand  

The web portal doesn't work anymore, if that is because of the corruption issues or the bad passwords I am not sure.

Lore

Lore

Posted
  • Author
Posted

I finally corrupted to the point of non use so I have installed to the eMMC for now.  Hopefully I wont have this corruption issue now.

Guest
This topic is now closed to further replies.
 Share
Go to topic listing
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines

  I accept