Polarisgeek Posted June 23, 2021 Share Posted June 23, 2021 I've been having a recent issue over the last couple months where my ArmbianEvn.txt file is being overwritten thus causing Helios to hang in the boot process. Using some samples i've found on these forums, I was able to re-create my file and get the Helios back up and running and have since stored a copy so that when it happens, I can just copy and paste and back up running in no time. To date, i've had this happen to me 4 times. I'm not worried about having go through this at this point in time, but I was curious if anyone else has seen this issue or not? I've saved a couple samples of what was written into the ArmbianEnv.txt file: /var/log/armbian-hardware-monitor.log { rotate 12 weekly compress missi /var/log/alternatives.log { monthly rotate 12 compress delaycompress missingok notifempty create 644 root root } I'm running Buster 5.10.21. I have 5-14TB EXOS drives in Raid 6 and have OMV as the only thing installed. 2 Quote Link to comment Share on other sites More sharing options...
barnumbirr Posted June 23, 2021 Share Posted June 23, 2021 Had the issue happen semi-regularly to my Helios as well. Same "gibberish" saved to ArmbianEnv.txt as in your example. Haven't been able to figure out where the issue comes from unfortunately. 1 Quote Link to comment Share on other sites More sharing options...
magostinelli Posted June 24, 2021 Share Posted June 24, 2021 I had also the same problem, take a look on /etc/logrotate.d folder, here there are other files corrupted. 1 Quote Link to comment Share on other sites More sharing options...
djurny Posted June 24, 2021 Share Posted June 24, 2021 Hi, You can try the following to get an idea of which process is modifying the file: sudo apt-get install auditd sudo auditctl -w /boot/armbianEnv.txt -p wa sudo tail -F /var/log/audit/audit.log The output should show actions performed on the file and (IIRC) the process ID performing those actions, perhaps that might help? e.g.: type=CWD msg=audit(1624526184.097:207): cwd="/home/djurny" type=PATH msg=audit(1624526184.097:207): item=0 name="/boot/" inode=7601 dev=b3:01 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(1624526184.097:207): item=1 name="/boot/armbianEnv.txt" inode=43198 dev=b3:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0 type=PROCTITLE msg=audit(1624526184.097:207): proctitle=xx type=SYSCALL msg=audit(1624526184.109:208): arch=40000028 syscall=94 per=800000 success=yes exit=0 a0=3 a1=81a4 a2=c80eeb00 a3=81a4 items=1 ppid=9641 pid=9642 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts5 ses=2 comm="vi" exe="/usr/bin/vim.basic" subj==unconfined key=(null) type=PATH msg=audit(1624526184.109:208): item=0 name=(null) inode=43198 dev=b3:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0 type=PROCTITLE msg=audit(1624526184.109:208): proctitle=xx type=SYSCALL msg=audit(1624526184.109:209): arch=40000028 syscall=226 per=800000 success=yes exit=0 a0=1787820 a1=b6e3e1a0 a2=1907fc0 a3=1c items=1 ppid=9641 pid=9642 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts5 ses=2 comm="vi" exe="/usr/bin/vim.basic" subj==unconfined key=(null) type=CWD msg=audit(1624526184.109:209): cwd="/home/djurny" type=PATH msg=audit(1624526184.109:209): item=0 name="/boot/armbianEnv.txt" inode=43198 dev=b3:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0 type=PROCTITLE msg=audit(1624526184.109:209): proctitle=xx Source: Find which process is modifying a file [duplicate] It honestly sounds a bit more like a filesystem that has lost track or a inode mixup somewhere? As someone mentioned, perhaps something else is writing to this file, thinking it is something else? Perhaps try to locate a symbolic (or hard)link to armbanEnv.txt: ## find any symbolic links to armbianEnv.txt sudo find / -xdev -type l -ls | egrep -i -- armbianEnv.txt ## find hardlinks to armbianEnv.txt - need to be on the same filesystem! sudo find / -xdev -samefile /boot/armbianEnv.txt Groetjes, 4 Quote Link to comment Share on other sites More sharing options...
barish Posted June 28, 2022 Share Posted June 28, 2022 Same has happened on EspressoBin running Buster 5.10. I don‘t believe it to be a filesystem problem. It looks like something happening at boot time perhaps while running /usr/lib/armbian-hardware-optimization.sh ? I will try to go further into it. 2 Quote Link to comment Share on other sites More sharing options...
barish Posted June 28, 2022 Share Posted June 28, 2022 (edited) On 6/24/2021 at 11:50 AM, djurny said: Hi, You can try the following to get an idea of which process is modifying the file: sudo apt-get install auditd sudo auditctl -w /boot/armbianEnv.txt -p wa sudo tail -F /var/log/audit/audit.log The output should show actions performed on the file and (IIRC) the process ID performing those actions, perhaps that might help? Unfortunately, auditd is not working at all. It just doesn't show any file access whatsoever, even though the demon is running and the rule is set. So this is of no help on my system. edit: Maybe the Armbian kernel is compiled without audit kernel module!? Edited June 28, 2022 by barish 1 Quote Link to comment Share on other sites More sharing options...
schwar3kat Posted June 28, 2022 Share Posted June 28, 2022 Ditto on a Pine64 running Buster or Bullseye (can't remember which). I thought it must have got corrupted somehow, but it's just a spare that I have so It didn't bother me. I noticed that it looked like /var/log/ stuff and thought that it was peculiar. I took the opportunity to try the new Jammy instead and forgot about it until I saw this post. 1 Quote Link to comment Share on other sites More sharing options...
Bernie_O Posted June 28, 2022 Share Posted June 28, 2022 (edited) I have the same problem. armbianEnv.txt gets overwritten with parts of logrotate config files. And some config files under /etc/logrotate.d/ are corrupted with stuff that looks like it originally belongs to armbianEnv.txt (e.g. boot parameters and MAC adresses). Is there anything I can do? As someone suspects that the script `armbian-hardware-optimization` might be responsible: would disabling the according service harm the system (or can the system run faultlessly without that service)? EDIT: this is on an espressobin running Debian Buster Edited June 28, 2022 by Bernie_O add used board and OS 0 Quote Link to comment Share on other sites More sharing options...
barish Posted June 28, 2022 Share Posted June 28, 2022 This is about three Helios, one Pine64 and two EspressoBin boards. @Werner could we move this thread to a place visible to others than Rockchip please? 1 Quote Link to comment Share on other sites More sharing options...
Werner Posted June 28, 2022 Share Posted June 28, 2022 2 minutes ago, barish said: could we move this thread to a place visible to others than Rockchip please? sure thing. 2 Quote Link to comment Share on other sites More sharing options...
schwar3kat Posted June 29, 2022 Share Posted June 29, 2022 I forgot that I had made a copy of the fs of the corrupted Pine64 SD. This is what's in armbianEnv.txt: /var/log.hdd/alternatives.log { monthly rotate 12 compress delaycompress missingok notifempt: create 644 root root } \00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00 usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u ------------------------------------------------------------------------------------------------- The first part of this content matches the content in the file /etc/logrotate.d/alternatives Most of the files in /etc/logrotate.d are also corrupted with only 'alternatives' seemingly intact. Many have streams of \00\00\00\00\00\ ... appended and many are truncated. There is an empty file named 'sedd3SIEh'. 'apport' has been overwritten with the missing part of armbianEnv.txt as follows: verbosity=1 bootlogo=false console=both disp_mode=1920x1080p60 overlay_prefix=sun50i-a64 rootdev=UUID=3e9afe94-8407-4434-9be6-9528 It's as if the sed command went haywire. /etc/logrotate.conf is also corupted, truncated and a string of \00\00\00.. appended. 1 Quote Link to comment Share on other sites More sharing options...
schwar3kat Posted June 29, 2022 Share Posted June 29, 2022 /usr/lib/armbian/armbian-hardware-optimization has two sed statements that work on these files: They look fairly innocuous,. I wonder if there is a new version of sed that is messing this up? Or perhaps the contents of one of the files is confusing sed? Alphabetically the first file corrupted (after alternatives, which is not corrupt) is apache2. I was running apache2 web server. How many of you are also running apache2? 0 Quote Link to comment Share on other sites More sharing options...
barish Posted June 29, 2022 Share Posted June 29, 2022 (edited) In my case, /boot/armbianEnv.txt looks like this (but I have had variations, of this, too): /var/log/alternatives.log { monthly rotate 12 compress delaycompress missingok notifempty create 644 root root } usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u,0x0bc2:0x231a:u Then, in /etc/logrotate.d, two files seem to be corrupt: 1. alternatives, 2. armbian-hardware-monitor, which had the complete contents of armbianEnv.txt. I run nginx, sambad, forked-daapd, nextcloud, mini-dlna and dnsmasq. No apache though. Edit: /etc/logrotate.conf is also truncated, thus corrupted. Hadn't noticed before, thanks @schwar3kat. Edited June 29, 2022 by barish 0 Quote Link to comment Share on other sites More sharing options...
barish Posted July 1, 2022 Share Posted July 1, 2022 I haven't found the cause, but I deactivated armbian-hardware-optimize.service, and did about 70 reboots afterwards, without any file corruption taking place. So I hope that armbian-hardware-optimization.sh (which is called by the above service) is involved in the error and its deactivation prevents any further file corruption. 0 Quote Link to comment Share on other sites More sharing options...
schwar3kat Posted July 4, 2022 Share Posted July 4, 2022 Is this perhaps restricted to Debian or is it affecting Ubuntu builds as well? 0 Quote Link to comment Share on other sites More sharing options...
barish Posted July 7, 2022 Share Posted July 7, 2022 I find it hard to imagine that maybe bash scripts interfere with each other and bash variables get confused across shell borders!?! It’s up to now the only guess I can come up with. This is maybe related to the early boot process where not everything is aligned yet? This is all wild guessing, I haven’t found a way to further analyze, because even auditd is failing on the armbian image I am using. 0 Quote Link to comment Share on other sites More sharing options...
Werner Posted July 8, 2022 Share Posted July 8, 2022 Just for the heck of it have you tried to set chattr +i on the file? 0 Quote Link to comment Share on other sites More sharing options...
schwar3kat Posted July 9, 2022 Share Posted July 9, 2022 On 7/8/2022 at 4:19 PM, Werner said: Just for the heck of it have you tried to set chattr +i on the file? That might be interesting. Perhaps the calling process will leave some errors in the logs. A sort of audit mechanism. I'm not running this image anymore, so I can't check, but others may like to try. 0 Quote Link to comment Share on other sites More sharing options...
jock Posted July 9, 2022 Share Posted July 9, 2022 I experienced this issue tons of times in the past while working on tvbox support, mostly it was happening after freeze/reboot during kernel boot. As far as I can remember I took a look and there was an armbian service which was rewriting armbianEnv.txt at startup (maybe to update usb quirks?), in fact the order of keys in armbianEnv.txt sometimes changes. I suspect that when, for a reason or another, the cache is not correctly wrote to flash, armbianEnv.txt gets corrupted. Recently it didn't happen anymore to me, to be honest. 2 Quote Link to comment Share on other sites More sharing options...
schwar3kat Posted July 10, 2022 Share Posted July 10, 2022 13 hours ago, jock said: there was an armbian service which was rewriting armbianEnv.txt at startup Now that you have mentioned it, I have also noticed the order change. Do you recall which service made the changes? 0 Quote Link to comment Share on other sites More sharing options...
jock Posted July 10, 2022 Share Posted July 10, 2022 9 hours ago, schwar3kat said: Now that you have mentioned it, I have also noticed the order change. Do you recall which service made the changes? I see this in armbian-hardware-optimization: armbian-hardware-optimization:369: echo "" >> /boot/armbianEnv.txt; sed -i '/^$/d;$G' /boot/armbianEnv.txt; sed -i '/^$/d;$G' /boot/armbianEnv.txt armbian-hardware-optimization:372: sed -i '/^$/d' /boot/armbianEnv.txt armbian-hardware-optimization:393: sed -i '/^usbstoragequirks/d' /boot/armbianEnv.txt armbian-hardware-optimization:394: echo "usbstoragequirks=${USBQUIRKS}" >>/boot/armbianEnv.txt And these others in armbian-firstrun: armbian-firstrun:72: # randomize mac in armbianEnv.txt armbian-firstrun:73: if [[ -f /boot/armbianEnv.txt ]]; then armbian-firstrun:75: sed -i "s/^ethaddr=.*/ethaddr=$MACADDR/" /boot/armbianEnv.txt armbian-firstrun:77: sed -i "s/^eth1addr=.*/eth1addr=$MACADDR/" /boot/armbianEnv.txt armbian-hardware-optimization has some other issues IMHO; there is a restoring/setting sound state which takes from 20 to 40 seconds on some of my systems slowing down the whole bootstrap and I'm not a big fan of the "big if switch" where all the optimizations for all the boards are grouped together, but those are other issues not to be discussed here. Anyway, touching armbianEnv.txt on every boot is very risky to me. That file is very important and some boards may not boot at all if it is misconfigured, even worse if corrupted. 0 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.