Changing to Chromium

[tkaiser]

'Upstream' the idea to ship OS images with Firefox/Iceweasel has been dropped in the meantime due to horribly low performance in favour of chromium-browser. As far as I understood that's partially related to firefox acting single threaded (while Chromium uses all CPU cores).

 

Here's longsleep's latest config and the mostly accepted 'fix' for Chromium complaining about running insecure due to missing sandboxing is disabling exactly this. (that's Hardkernel's method to 'fix' Chromium on ODROID-C2). Users will be happy with a fast(er) browser not throwing annoying error messages at them they click away without reading them anyway and surfing the web on Pinebook with crappy BSP kernel will get a little bit more insecure too. Will we follow or waste our time trying to find out why seccomp/sandboxing is not working on arm64?

[Igor]

11 minutes ago, tkaiser said:

Will we follow or waste our time trying to find out why seccomp/sandboxing is not working on arm64?

I haven't done any research in this area but I suspect this is digging into the mud. What about http://midori-browser.org/ ? But I don't have much experiences neither ... Let's stick to Firefox for now and check little more about this and / or provide both browsers for now?

 

BTW. After few hours, no change with my battery. Still not powering on.

[tkaiser]

57 minutes ago, Igor said:

Let's stick to Firefox for now and check little more about this and / or provide both browsers for now?

 

Well, if you compare Chromium with Firefox side by side it's impossible to use FF any more since it's too slow to not get immediately mad. But Chromium would require weakening security (further, I'm not that convinced that the BSP kernel everyone uses on A64 devices is suitable for connecting devices to the Internet with Linux anyway).

 

Midori looks nice and fast especially after:

root@pinebook:~# cat /etc/profile.d/xdg_cache_home.sh
#!/bin/bash
export XDG_CACHE_HOME="/dev/shm/.cache"

 

 

[hojnikb]

is midori even working stable on arm ? last time i check it segfaulted on me almost immediately.

[tkaiser]

3 hours ago, tkaiser said:

/etc/profile.d/xdg_cache_home.sh

 

Setting this improves everything a lot with FF, Midori and Chromium. But it's not a 'real' solution since this way most probably all available RAM will be eaten up over time unless the browsers have a configurable cache size. Also everything is lost after a reboot (so setting XDG_CACHE_HOME="/var/log/.cache" might be an idea since log2ram then does the job). 

 

I will stop now testing with the following results:

root@pinebook:/dev/shm/.cache# du -sh *
4.0K	blueman-applet-1000
27M	chromium
12K	event-sound-cache.tdb.aab709d4dc8efb47e12e29ac5910e6a8.aarch64-unknown-linux-gnu
100K	fontconfig
276K	gstreamer-1.0
616K	mate
3.6M	midori
5.7M	mozilla
0	obexd
0	tilda
4.0K	webkit
root@pinebook:/dev/shm/.cache# cat /etc/chromium-browser/default 
# Default settings for chromium-browser. This file is sourced by /bin/sh from
# /usr/bin/chromium-browser

# Options to pass to chromium-browser
CHROMIUM_FLAGS="--disable-smooth-scrolling \
--disable-low-res-tiling \
--enable-low-end-device-mode \
--num-raster-threads=4 \
--disable-seccomp-filter-sandbox"

With '--disable-seccomp-filter-sandbox' every time Chromium will be started a security/stability warning will appear with our current kernel config which might be better than fooling users by disabling sandboxing without notice? Then on Pinebook installation of this Extension is recommended: https://chrome.google.com/webstore/detail/no-mousewheel-zoom/ckhlfagjncfmdieecfekjbpnnaekhlhd

 

For anyone doing further testing I strongly recommend to always run htop and 'iostat 5' in 2 separate terminals in the background to watch for multithreading behaviour and Pinebook being bottlenecked by slow IO (or not when caches live in memory and not on slow storage). Testing with https://www.theguardian.com/international seems like a good idea (@Xaliusidea -- see today's http://irc.pine64.uk log)

 

BTW: using LZO compression my chromium cache folder shrinks only to ~75% so most probably it's useless to further explore ways to use compressed tmpfs. I did just a quick static test and already gave up on the idea.

[Xalius]

There is a switch to limit Chromium's disk cache size:

 

--disk-cache-size=n

n in Bytes, I use 300MB atm...

[tkaiser]

48 minutes ago, Xalius said:

There is a switch to limit Chromium's disk cache size

 

Thank you. Now this works surprisingly well with Chromium (with 'export XDG_CACHE_HOME="/var/log/.cache"' and /var/log/.cache created with 777 mode manually before):

root@pinebook:~# cat /etc/chromium-browser/default 
# Default settings for chromium-browser. This file is sourced by /bin/sh from
# /usr/bin/chromium-browser

# Options to pass to chromium-browser
CHROMIUM_FLAGS="--disable-smooth-scrolling \
--disable-low-res-tiling \
--enable-low-end-device-mode \
--num-raster-threads=4 \
--disk-cache-size=314572800 \
--profiler-timing=0 \
--disable-composited-antialiasing \
--disable-seccomp-filter-sandbox"
root@pinebook:~# cat /etc/default/log2ram 
# configuration values for the log2ram service
#
# enable the log2ram service?
ENABLED=true
#
# size of the tmpfs mount
SIZE=350M
#
# use rsync instead of cp -r
# requires rsync installed, may provide better performance
# due to copying only new and changed files
USE_RSYNC=true

 

[Igor]

I did few tests on Odroid C2 since my Pinebook is still dead and have nothing against to switch to Chromium. Firefox can't match. 

There are still some writing to SD but minimal.
 

6 hours ago, tkaiser said:

Will we follow or waste our time trying to find out why seccomp/sandboxing is not working on arm64?

Follow, waste later. We know about this problem and eventually will be fixed.

[tkaiser]

2 hours ago, Igor said:

have nothing against to switch to Chromium. Firefox can't match

 

I agree and prepared that with this commit. Prerequisits: 

• we replace firefox with chromium on desktop images
• we add /etc/chromium-browser/default on desktop images with @Xalius' contents: http://pastebin.ubuntu.com/24575990/
• we add /etc/profile.d/xdg_cache_home.sh defining 'XDG_CACHE_HOME="/var/log/.cache"' on desktop images and do an 'mkdir -m 777 /var/log/.cache' as part of image creation (otherwise it will fail)
• we disable SECCOMP on all 64-bit platforms in kernel config just like this (though I've no idea about H5/mainline now)

IMO some testing needs to be done on

• 64-bit board with H5/mainline
• 32-bit board with low memory (H2+/H3 board with 512 MB)
• 32-bit board with just 2 CPU cores (any A20 thing should do)

log2ram syncing every hour will be 'funny' on systems running desktop images off crappy SD cards but to be honest: I don't care at all

 

TODO:

• check security implications of setting $XDG_CACHE_HOME to a non unique location on multi user systems (to be honest: I don't care since it's already insane to 'browse the web' with smelly vendor kernels)
• resolve Chromium sandboxing on 64-bit platforms

[zador.blood.stained]

9 minutes ago, tkaiser said:

• we add /etc/profile.d/xdg_cache_home.sh defining 'XDG_CACHE_HOME="/var/log/.cache"' on desktop images and do an 'mkdir -m 777 /var/log/.cache' as part of image creation (otherwise it will fail)

I don't like the idea of putting a browser cache to the logs directory. We could make another tmpfs mount  for the cache that doesn't have to be saved at all (or saved only on shutdown) - who cares if browser cache is cleared on reboot

[Igor]

I just committed package change

 

What about adding to armhwinfo?

# create chromium cache directory
[[ ! -d /var/log/.cache ]] && mkdir /var/log/.cache && chgrp netdev /var/log/.cache && chmod g+w /var/log/.cache

You can add scripts to:

/etc/chromium-browser/customizations

export XDG_CACHE_HOME="/var/log/.cache"

works.

[tkaiser]

7 minutes ago, zador.blood.stained said:

We could make another tmpfs mount  for the cache that doesn't have to be saved at all (or saved only on shutdown) - who cares if browser cache is cleared on reboot

 

Chromium and users will care (since message 'unsafe shutdown, do you want to restore previous session?' or something like that). But I agree, /var/log/.cache is somewhat dirty, let's implement something different that gets only synced on shutdown/boot. Do you think log2ram script is sufficient for that (when dealing with OMV the last weeks I came across 'folder2ram' which might be worth a look for such scenarios)?

[Igor]

OK, I add only things that it's at least functional now, than we change this asap.

[zador.blood.stained]

5 minutes ago, tkaiser said:

Chromium and users will care (since message 'unsafe shutdown, do you want to restore previous session?' or something like that).

AFAIK this is only for the cache, and browser cache != browser profile, so settings, bookmarks, extensions and other things still live on persistent storage and only cached web files are on tmpfs.

 

5 minutes ago, tkaiser said:

Do you think log2ram script is sufficient for that (when dealing with OMV the last weeks I came across 'folder2ram' which might be worth a look for such scenarios)?

Current log2ram version deals only with 1 hardcoded directory, we need to check if folder2ram is more universal.

[tkaiser]

14 minutes ago, zador.blood.stained said:

browser cache != browser profile, so settings, bookmarks, extensions and other things still live on persistent storage and only cached web files are on tmpfs

 

Hmm... Chromium started to complain on my installation after defining XDG_CACHE_HOME pointing to DRAM but XDG_CACHE_HOME="/var/log/.cache" didn't fix it. Needs more investigations. I won't do that much here within the next week(s) since performing an 'Aunt Tilly' test: handing my Pinebook over to an elderly lady used to OS X since her MacBook died. Currently backing up the installation, then trying to upgrade to 16.10 and installing GNOME OS X II theme.

[tkaiser]

13 hours ago, zador.blood.stained said:

Current log2ram version deals only with 1 hardcoded directory

 

With the following minor changes we could add a symlink /usr/sbin/xdgcache2ram (or maybe also /usr/sbin/profile2ram later) and then it's just adding /etc/defaults/xdgcache2ram with own settings and a xdgcache2ram.service without an hourly cron job and we would already be done: https://pastebin.com/raw/Pw48Nrqp

 

 

[Igor]

I can implement this later if we agree @zador.blood.stained @tkaiser

/usr/sbin/logram # changed #11

Spoiler

#!/bin/sh

# Source: https://github.com/azlux/log2ram
# License: MIT
# License file: /usr/share/log2ram/LICENSE

SIZE=40M
USE_RSYNC=false
ENABLED=false
ME="${0##*/}"
DIRNAME="$(echo ${ME} | sed 's/2ram$//')"

[ -f /etc/default/${ME} ] && . /etc/default/${ME}

[ "$ENABLED" != true ] && exit 0

# don't touch anything below here.

HDD_LOG=/var/${DIRNAME}.hdd/
RAM_LOG=/var/${DIRNAME}/
LOG2RAM_LOG="${HDD_LOG}${ME}.log"
LOG_OUTPUT="tee -a $LOG2RAM_LOG"

isSafe () {
    [ -d $HDD_LOG ] || echo "ERROR: $HDD_LOG doesn't exist!  Can't sync."
    [ -d $HDD_LOG ] || exit 1
}

syncToDisk () {
    isSafe

    if [ "$USE_RSYNC" = true ]; then
        rsync -aXWv --delete --exclude ${ME}.log --links $RAM_LOG $HDD_LOG 2>&1 | $LOG_OUTPUT
    else
        cp -rfup $RAM_LOG -T $HDD_LOG 2>&1 | $LOG_OUTPUT
    fi
}

syncFromDisk () {
    isSafe

    if [ "$USE_RSYNC" = true ]; then
        rsync -aXWv --delete --exclude ${ME}.log --links $HDD_LOG $RAM_LOG 2>&1 | $LOG_OUTPUT
    else
        cp -rfup $HDD_LOG -T $RAM_LOG 2>&1 | $LOG_OUTPUT
    fi
}


case "$1" in
  start)
      [ -d $HDD_LOG ] || mkdir $HDD_LOG
      rm -f $LOG2RAM_LOG
      mount --bind $RAM_LOG $HDD_LOG
      mount --make-private $HDD_LOG
      mount -t tmpfs -o nosuid,noexec,nodev,mode=0755,size=$SIZE ${ME} $RAM_LOG
      syncFromDisk
      ;;

  stop)
      syncToDisk
      umount -l $RAM_LOG
      umount -l $HDD_LOG
      ;;

  write)
      syncToDisk
      ;;
  *)
      echo "Usage: ${ME} {start|stop|write}" >&2
      exit 1
      ;;
esac

 

 

[zador.blood.stained]

IMO for now we can just move/symlink Chromium cache to /tmp (which should be tmpfs already)

[tkaiser]

Well, IMO that's not the same if you look above: https://forum.armbian.com/index.php?/topic/4285-changing-to-chromium/&do=findComment&comment=31759

 

Either we get things sorted before next major release (and then a persistent cache living in RAM with a dynamic size depending on DRAM that gets synced is IMO mandatory) or we can leave everything as is and drop the idea of a Pinebook image for now.

 

I thought reusing the log2ram skeleton would be a good idea since it requires just minimal changes when we add another xdgcache2ram.service unit and an own config file so testing efforts are minimal since log2ram is confirmed to work on a huge number of Armbian installations. But I agree that it's somewhat ugly.

 

[zador.blood.stained]

We can push an image for the Pinebook at any time, it doesn't have to come with this stable release.

[Igor]

Problems with 32bit Chromium ... I just made some tests with Cubietruck and Chromium is not working - "Aw - snap! Something went wrong ..." 

[TonyMac32]

Same here, I just mentioned it in the other thread, Chromium is "Aw Snap!" on Tinker Board as well.

[Igor]

Wonderful. We have broken Firefox in arm64 (but solvable) and broken Chromium in armhf . In this case is better to have Firefox which at least works. We would need extensive research and patching for which we don't have time ATM so rolling back to Firefox seems to be the only viable solution?

 

If you do some quick search, you can see lot's of problems with Chromium on Rpi, Odroids, ... 

[tkaiser]

Well, maybe it's time to slow down (again) and do some research first? 

 

What do we 'know'? Chromium stability might be related to settings (I'm not sure whether this commit missing trailing double quote works or not -- tried to fix it yesterday but my whole commit has been reverted for good reasons). Firefox/arm64 stability might be related to... what? We don't know, we never explored (maybe it's extensive memory usage, maybe not). Installing armhf FF on arm64 blows up installation size by ~140MB. Also not that great. 

 

Based on the way we currently build images we could easily differentiate between armhf and arm64 at image creation time (not ideal from a maintainance point of view but at least we would get experiences/feedback on both browser variants). Luke shared some ideas how to improve FF performance here. And Zador outlined what needs to be done to solve storage performance dependency problems (cache/profile in RAM) in a more general way here.

 

I still wonder whether there's some sort of a 'real world' benchmark suite for webbrowsers on Linux (the usual benchmarks are crap since testing irrelevant stuff for our situation). Something like a script that can be used to instruct FF and Chromium to visit 10 defined web sites so we can measure performance, IO load and memory requirements (to get more ideas about armhf vs. arm64 on 64-bit platforms!). On macOS I would already be done but no idea how to do this with Linux

[umiddelb]

seleniumhq ?

[tkaiser]

To all: Even after https://github.com/armbian/build/commit/d78d95622aab0a20a0b532410644a886c3c44e55 there's way too much write activity. Please check yourself using

find $HOME/.config . -mtime 0
armbianmonitor -d mmcblk0

Now that we've addressed the cache problem we should take a deeper look into 'profile sync daemon' and necessary configuration bits: http://irc.pine64.uk --> 03-05-2017 --> 08:45:32 (won't do it since it's sunny today  )

[zador.blood.stained]

We have to make sure that all dependencies for psd (like overlayfs) are satisfied on systems where it is intalled

[tkaiser]

I implemented dynamic detection of overlayfs with https://github.com/armbian/build/commit/50a6c0791e243b227a541f7f4163d2288d98c445

 

Missing part: installation of the daemon itself as part of desktop.sh. Won't touch this since not sure whether it's ok to add a 3rd party PPA like here or we should integrate the scripts in build system or repo?

 

IMO we should use our rather useless nightly builds within the next days to ask users to test through stuff...

 

BTW: to check efficiency it's always just 'psd p[review]':

pine64@pinebook:~$ psd preview
Profile-sync-daemon v6.31 on Ubuntu 16.04.2 LTS

 Systemd service is currently active.
 Systemd resync-timer is currently active.
 Overlayfs v22 is currently active.

Psd will manage the following per /home/pine64/.config/psd/.psd.conf:

 browser/psname:  firefox/firefox
 owner/group id:  pine64/1000
 sync target:     /home/pine64/.mozilla/firefox/7ebocrc0.default
 tmpfs dir:       /run/user/1000/pine64-firefox-7ebocrc0.default
 profile size:    14M
 overlayfs size:  0
 recovery dirs:   none

find: ‘/home/pine64/.config/htop’: Permission denied
 browser/psname:  midori/midori
 owner/group id:  pine64/1000
 sync target:     /home/pine64/.config/midori
 tmpfs dir:       /run/user/1000/pine64-midori
 profile size:    292K
 overlayfs size:  0
 recovery dirs:   none

find: ‘/home/pine64/.config/htop’: Permission denied
 browser/psname:  chromium/chromium
 owner/group id:  pine64/1000
 sync target:     /home/pine64/.config/chromium
 tmpfs dir:       /run/user/1000/pine64-chromium
 profile size:    32M
 overlayfs size:  4.6M
 recovery dirs:   none

 

[tkaiser]

https://github.com/agiz/youtube-mpv

[hojnikb]

I'd suggest using chromium:armhf on both arm and arm64 builds, as there really is no reason to use arm64 build of chromium, because it's more ram hungry and not faster on our ram limited boards. I've been using armhf version on PC2 for some time now and works just fine. Needs a tweak to get it working though.