infinity Posted August 11, 2016 Share Posted August 11, 2016 I've done a apt-get upgrade on my banana PI running ARMBian. Since then I get every morning this error message via email: /etc/cron.daily/ramlog: Error: ramlog is not running... failed! run-parts: /etc/cron.daily/ramlog exited with return code 110 In release history I can't find any recent changes regarding ramlog. Whats wrong? If I invoke the cron-command manually in SSH, i get this: root@bananapi:~# /etc/init.d/ramlog restart [FAIL] Error: ramlog is not running... failed! If I do simply a start instead of restart, I get this: root@bananapi:~# /etc/init.d/ramlog start [FAIL] Starting ramlog-tmpfs 2.0Error: /var/log is in use... failed! The list of open files: (You need to close below daemons if you want to start/stop ramlog manually) COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 1661 root 2w REG 179,1 41119 2362 /var/log/syslog rsyslogd 1661 root 4w REG 179,1 174185 2296 /var/log/kern.log rsyslogd 1661 root 5w REG 179,1 894 3414 /var/log/debug rsyslogd 1661 root 6w REG 179,1 238312 2420 /var/log/messages rsyslogd 1661 root 7w REG 179,1 15976 2161 /var/log/daemon.log rsyslogd 1661 root 8w REG 179,1 30531 2193 /var/log/auth.log rsyslogd 1661 root 9w REG 179,1 6745 2414 /var/log/mail.log rsyslogd 1661 root 10w REG 179,1 6745 2292 /var/log/mail.info nmbd 1905 root 2w REG 179,1 407 128011 /var/log/samba/log.nmbd nmbd 1905 root 8w REG 179,1 407 128011 /var/log/samba/log.nmbd memcached 1963 memcache 1w REG 179,1 0 2331 /var/log/memcached.log memcached 1963 memcache 2w REG 179,1 0 2331 /var/log/memcached.log nginx 2056 root 2w REG 179,1 1170 256440 /var/log/nginx/error.log nginx 2056 root 4w REG 179,1 1170 256440 /var/log/nginx/error.log nginx 2056 root 5w REG 179,1 56626 256020 /var/log/nginx/seahub.access.log nginx 2056 root 6w REG 179,1 590 256058 /var/log/nginx/seahub.error.log nginx 2058 www-data 2w REG 179,1 1170 256440 /var/log/nginx/error.log nginx 2058 www-data 4w REG 179,1 1170 256440 /var/log/nginx/error.log nginx 2058 www-data 5w REG 179,1 56626 256020 /var/log/nginx/seahub.access.log nginx 2058 www-data 6w REG 179,1 590 256058 /var/log/nginx/seahub.error.log nginx 2060 www-data 2w REG 179,1 1170 256440 /var/log/nginx/error.log nginx 2060 www-data 4w REG 179,1 1170 256440 /var/log/nginx/error.log nginx 2060 www-data 5w REG 179,1 56626 256020 /var/log/nginx/seahub.access.log nginx 2060 www-data 6w REG 179,1 590 256058 /var/log/nginx/seahub.error.log fail2ban- 2338 root 5w REG 179,1 7506 2290 /var/log/fail2ban.log smbd 2343 root 2w REG 179,1 63978 128030 /var/log/samba/log.smbd smbd 2343 root 8w REG 179,1 63978 128030 /var/log/samba/log.smbd smbd 2370 root 2w REG 179,1 63978 128030 /var/log/samba/log.smbd smbd 2370 root 13w REG 179,1 63978 128030 /var/log/samba/log.smbd Test result: ramlog cannot be started or stopped at the moment. Why did this work 3 days ago before the update? Also: While checking out the forum for this issue I've seen this post from azlux http://forum.armbian.com/index.php/topic/753-where-is-ramlog/?p=13172 Could it be possible to integrate it as per default instead of ramlog into ARMbian in future releases? Another Question: Is SystemD now activated in ARMBian? Release history from v4.81 / 28.12.2015 implies it is activated. I'm confused, as Ramlog was the reason for keeping SystemD deactivated in ARMBian, right? I updated ARMBian some months ago, thus Systemd has been working on my system for some months without my notice, but ramlog worked also, so I'm a bit confused how that could work with activated systemd until last monday Link to comment Share on other sites More sharing options...
Igor Posted August 12, 2016 Share Posted August 12, 2016 Basic problem here is that ramlog is not compatible with systemd. You must have run Debian Wheezy and when a normal upgrade to Jessie is issued a systemd is installed. I assume it was this way? If you want to run without systemd and ramlog, check this: http://without-systemd.org/wiki/index.php/How_to_remove_systemd_from_a_Debian_jessie/sid_installation Link to comment Share on other sites More sharing options...
zador.blood.stained Posted August 12, 2016 Share Posted August 12, 2016 I've done a apt-get upgrade on my banana PI running ARMBian. Armbian based on which Debian/Ubuntu release exactly? In release history I can't find any recent changes regarding ramlog. Whats wrong? Ramlog on Wheezy and Trusty (also on old Armbian Jessie release without systemd) relies on "hack" to start before rsyslog (it modifies rsyslog init script), so rsyslog update might mess this up. Another Question: Is SystemD now activated in ARMBian? Release history from v4.81 / 28.12.2015 implies it is activated. I'm confused, as Ramlog was the reason for keeping SystemD deactivated in ARMBian, right? I updated ARMBian some months ago, thus Systemd has been working on my system for some months without my notice, but ramlog worked also, so I'm a bit confused how that could work with activated systemd until last monday Armbian currently uses init system from upstream distribution without any changes. Systemd was disabled in Jessie due to another reasons, not related to ramlog. You must have run Debian Wheezy and when a normal upgrade to Jessie is issued a systemd is installed. I assume it was this way? Only release upgrade (wheezy -> jessie or trusty -> xenial) could install systemd if it was not installed previously, but this doesn't happen via apt-get upgrade or even apt-get dist-upgrade. Link to comment Share on other sites More sharing options...
infinity Posted August 12, 2016 Author Share Posted August 12, 2016 Thanks for your replies So.... I know for sure it was Jessie terminal version with Vanilla Kernel (now called Jessie Server) I'm not sure which version-number exactly, but either it was v4.81 / 28.12.2015 or... v5.00 / 12.2.2016 According to my notes I setup this system on 10. February 2016, so it must have been v4.81, which I must have updated (apt-get upgrade) to v5.00 some days later. Then a month ago again and on monday the last time. Ramlog was working till monday. Actually I'm just a bit confused, no panic here on my side As I'm not a experienced Linux user I'm not sure now whether my system has systemd enabled or not. I thought it is disabled, because I had the exact same problem back in february: http://forum.armbian.com/index.php/topic/342-dbus-error-when-using-systemctl/http://forum.armbian.com/index.php/topic/342-dbus-error-when-using-systemctl/ and because Igors reply in #2 of the mentioned thread was that systemd was disabled, I was satisfied with this answer. Now I'm confused as it turns out that systemd was apparently enabled haha. My rsyslog is this: #! /bin/sh ### BEGIN INIT INFO # Provides: rsyslog # Required-Start: $remote_fs $time ramlog # Required-Stop: umountnfs $time ramlog # X-Stop-After: sendsigs # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: enhanced syslogd # Description: Rsyslog is an enhanced multi-threaded syslogd. # It is quite compatible to stock sysklogd and can be # used as a drop-in replacement. ### END INIT INFO # # Author: Michael Biebl <biebl@debian.org> # # PATH should only include /usr/* if it runs after the mountnfs.sh script PATH=/sbin:/usr/sbin:/bin:/usr/bin DESC="enhanced syslogd" NAME=rsyslog RSYSLOGD=rsyslogd DAEMON=/usr/sbin/rsyslogd PIDFILE=/var/run/rsyslogd.pid SCRIPTNAME=/etc/init.d/$NAME # Exit if the package is not installed [ -x "$DAEMON" ] || exit 0 # Read configuration variable file if it is present [ -r /etc/default/$NAME ] && . /etc/default/$NAME # Define LSB log_* functions. . /lib/lsb/init-functions do_start() { # Return # 0 if daemon has been started # 1 if daemon was already running # other if daemon could not be started or a failure occured start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- $RSYSLOGD_OPTIONS } do_stop() { # Return # 0 if daemon has been stopped # 1 if daemon was already stopped # other if daemon could not be stopped or a failure occurred start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --exec $DAEMON } # # Tell rsyslogd to close all open files # do_rotate() { start-stop-daemon --stop --signal HUP --quiet --pidfile $PIDFILE --exec $DAEMON } create_xconsole() { XCONSOLE=/dev/xconsole if [ "$(uname -s)" != "Linux" ]; then XCONSOLE=/run/xconsole ln -sf $XCONSOLE /dev/xconsole fi if [ ! -e $XCONSOLE ]; then mknod -m 640 $XCONSOLE p chown root:adm $XCONSOLE [ -x /sbin/restorecon ] && /sbin/restorecon $XCONSOLE fi } sendsigs_omit() { OMITDIR=/run/sendsigs.omit.d mkdir -p $OMITDIR ln -sf $PIDFILE $OMITDIR/rsyslog } case "$1" in start) log_daemon_msg "Starting $DESC" "$RSYSLOGD" create_xconsole do_start case "$?" in 0) sendsigs_omit log_end_msg 0 ;; 1) log_progress_msg "already started" log_end_msg 0 ;; *) log_end_msg 1 ;; esac ;; stop) log_daemon_msg "Stopping $DESC" "$RSYSLOGD" do_stop case "$?" in 0) log_end_msg 0 ;; 1) log_progress_msg "already stopped" log_end_msg 0 ;; *) log_end_msg 1 ;; esac ;; rotate) log_daemon_msg "Closing open files" "$RSYSLOGD" do_rotate log_end_msg $? ;; restart|force-reload) $0 stop $0 start ;; status) status_of_proc -p $PIDFILE $DAEMON $RSYSLOGD && exit 0 || exit $? ;; *) echo "Usage: $SCRIPTNAME {start|stop|rotate|restart|force-reload|status}" >&2 exit 3 ;; esac : As ramlog is mentioned there, this means that rsyslog was not modified due to an update, right? So I'm still wondering why ramlog fails since monday. [...] If you want to run without systemd and ramlog, check this: http://without-systemd.org/wiki/index.php/How_to_remove_systemd_from_a_Debian_jessie/sid_installation Well I was actually happy you had ramlog integrated, but on the other hand I don't mind if it's not there anymore. But it is apparently installed and it worked 4 days ago for half a year, so I'm curious what might break it. Perhaps it's something serious and this is just the first symptom. [...] Ramlog on Wheezy and Trusty (also on old Armbian Jessie release without systemd) relies on "hack" to start before rsyslog (it modifies rsyslog init script), so rsyslog update might mess this up. [...] Looking at the posted rsyslog it is apparently not modified, as ramlog is mentioned in there. So something else might be the reason for breaking ramlog function since monday. [...] Armbian currently uses init system from upstream distribution without any changes. Systemd was disabled in Jessie due to another reasons, not related to ramlog. [...] Does that mean that systemd is enabled now or not? I'm a bit confused (not experienced in Linux unfortunately) [...] Only release upgrade (wheezy -> jessie or trusty -> xenial) could install systemd if it was not installed previously, but this doesn't happen via apt-get upgrade or even apt-get dist-upgrade. Clearly not the case here. This setup was never wheezy, nor did I ever make apt-get dist-upgrade. My thoughts: Perhaps everybody, who is coming from 4.81 or 5.00 originally, has this errormessage that I mentioned in my opening post, but most don't notice, because they don't have a mailer daemon installed on their system? I mean... I also wouldn't notice this if my system didn't send me this email since I installed sSMTP for fail2ban back then. I did not setup those status messages intentionally, they just came after sSMTP installation. I kind of began to like to see these daily mails, because they showed me that the server is running properly. My further questions: How can I make sure systemd is running properly? Is Ramlog still integrated / enabled in current v5.16 / 5.7.2016 release, if I make a v5.16 fresh install? Assuming I uninstall ramlog via http://forum.armbian.com/index.php/topic/342-dbus-error-when-using-systemctl/#entry8104, how bad is this for the SD-Card then? Do you have any bad experience? I mean... it's clear that sd-cards don't like frequent writes... but is this really a big issue then? Are there any plans to implement Ram2Log instead of ramlog in the future? I saw that Ram2Log is working on filesize basis (40MB I think is default there): Which value would be better for a SD-Card on ARMbian, 2MB? I can't imagine that logs can collect 40MB within 24 hours or so. If I'd leave Ram2Log at 40MB, then logs would be written within half a year or so? Link to comment Share on other sites More sharing options...
zador.blood.stained Posted August 12, 2016 Share Posted August 12, 2016 Does that mean that systemd is enabled now or not? I'm a bit confused (not experienced in Linux unfortunately) This means that all new Jessie and Xenial releases are with systemd, Wheezy still uses sysvinit and Trusty uses upstart. How can I make sure systemd is running properly? Hm. systemctl status systemctl --failed should provide some info Is Ramlog still integrated / enabled in current v5.16 / 5.7.2016 release, if I make a v5.16 fresh install? Ramlog is still present in Wheezy and Trusty configurations, but there are no prebuilt images for these releases anymore AFAIK. Assuming I uninstall ramlog via http://forum.armbian.com/index.php/topic/342-dbus-error-when-using-systemctl/#entry8104, how bad is this for the SD-Card then? Do you have any bad experience? I mean... it's clear that sd-cards don't like frequent writes... but is this really a big issue then? Are there any plans to implement Ram2Log instead of ramlog in the future? I saw that Ram2Log is working on filesize basis (40MB I think is default there): Which value would be better for a SD-Card on ARMbian, 2MB? I can't imagine that logs can collect 40MB within 24 hours or so. If I'd leave Ram2Log at 40MB, then logs would be written within half a year or so? Depends on what services are running on your system. If you have anything that writes huge amount of logs, then it's recommended to use ramlog or its systemd-compatible alternatives like log2ram. Adding this by default is a low priority unless someone implements and provides working package. Link to comment Share on other sites More sharing options...
infinity Posted August 12, 2016 Author Share Posted August 12, 2016 [...] Depends on what services are running on your system. If you have anything that writes huge amount of logs, then it's recommended to use ramlog or its systemd-compatible alternatives like log2ram. Adding this by default is a low priority unless someone implements and provides working package. Oh I thought Ram2Log from the linked post was some kind of working package. Nevermind, I can try it myself then Actually it is obviously better to not add such things by default, as everybody can choose then to use it or not. Was looking a bit around in RamLog documentation and found a report command: root@bananapi:~# more /var/log/ramlog Aug 07 06:36:23 Restarting ramlog = saving logs to hdd: [ OK ] Aug 08 06:42:36 Restarting ramlog = saving logs to hdd: [ OK ] Aug 08 16:37:30 Saving logs to hdd: [ OK ] Aug 08 16:37:30 Stopping ramlog: [ OK ] Aug 08 16:38:24 Starting ramlog-tmpfs 2.0.0: Error: /var/log is in use... [fail] Aug 08 16:38:25 The list of open files: (You need to close below daemons if you want to start/stop ramlo g manually) COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME lsusb 1537 root 1w REG 179,1 20448 2168 /var/log/armhwinfo.log Aug 08 16:38:25 Test result: ramlog cannot be started or stopped at the moment. Aug 08 17:01:38 ramlog: Cannot find shutdown flag file. [warning] Aug 08 17:01:38 Stopping ramlog: Error: ramlog is not running [fail] Aug 08 17:02:29 Starting ramlog-tmpfs 2.0.0: Error: /var/log is in use... [fail] Aug 08 17:02:30 The list of open files: (You need to close below daemons if you want to start/stop ramlo g manually) COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME lsusb 1533 root 1w REG 179,1 47278 2168 /var/log/armhwinfo.log Aug 08 17:02:30 Test result: ramlog cannot be started or stopped at the moment. Aug 09 06:53:23 Error: ramlog is not running... [fail] Aug 10 06:48:01 Error: ramlog is not running... [fail] Aug 11 06:43:18 Error: ramlog is not running... [fail] Aug 11 18:36:06 ramlog: Cannot find shutdown flag file. [warning] Aug 11 18:36:06 Stopping ramlog: Error: ramlog is not running [fail] Aug 11 18:36:47 Starting ramlog-tmpfs 2.0.0: Error: /var/log is in use... [fail] Aug 11 18:36:48 The list of open files: (You need to close below daemons if you want to start/stop ramlo g manually) COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME lsusb 1540 root 1w REG 179,1 20064 2287 /var/log/armhwinfo.log Aug 11 18:36:48 Test result: ramlog cannot be started or stopped at the moment. Aug 11 18:41:42 Error: ramlog is not running... [fail] Aug 11 18:41:49 Starting ramlog-tmpfs 2.0.0: Error: /var/log is in use... [fail] So the reason is lsusb if I'm not wrong? How come? This definitely occured the first time after the last apt-get upgrade. System is working properly besides this error message with Ramlog. Did not notice anything suspicious. [...] systemctl status systemctl --failed [...] root@bananapi:~# systemctl status Failed to get D-Bus connection: Unknown error -1 root@bananapi:~# systemctl --failed Failed to get D-Bus connection: Unknown error -1 root@bananapi:~# So in my case systemd is actually disabled. Strange... As I am coming from 4.81 for sure (perhaps even 5.00). I think I should set up this system from scratch with a new prebuilt image. Is there something new on the horizon to wait for? Any fundamental changes in ARMbian (e.g. like systemd on or off) Link to comment Share on other sites More sharing options...
lupus Posted September 16, 2016 Share Posted September 16, 2016 I realised today that I have the same issue - here is how to solve it (I am running Jessie with Kernel 4.7.3 and sysv-init - not systemd) The problem is that /etc/init.d/armhwinfo starts before /etc/init.d/ramlog. You can change that in the following way: 1.) add ramlog to Required-Start and Required-Stop in /etc/init.d/armhwinfo: #!/bin/bash ### BEGIN INIT INFO # Provides: armhwinfo # Required-Start: ramlog # Required-Stop: glibc ramlog # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Armbian gathering hardware information ### END INIT INFO 2.) enter the following commands as root: dpkg-reconfigure insserv update-rc.d armhwinfo defaults shutdown -r now # reboot at least two times Cheers, lupus 1 Link to comment Share on other sites More sharing options...
infinity Posted September 23, 2016 Author Share Posted September 23, 2016 Lupus! Nice!! Seems to work Well... this implicates that most are affected by this issue (if updated from Armbian <=v5.0), but most don't notice, because they didn't set up a mailer in their system. Thank you very much Lupus Link to comment Share on other sites More sharing options...
Recommended Posts