R2D2_C3PO Posted June 12, 2016 Posted June 12, 2016 While this idea is not on any list it might still be interesting for some people. There is already an option to build an image with btrfs as the root filesystem. However this requires rebuilding the whole image and you have to know the SD card size in advance. I will write a script to convert an already existing image (or even an already used system on a SD-Card) to btrfs. This script will also handle the related problems (creating an ext4 boot partition, updating fstab, etc.). I will also try to create a script to set up automatic btrfs snapshots before and after any package installations (especially for the automated updates). This is the main reason why I want to switch to btrfs. Is allows for easy rollbacks if something goes wrong and provides a nice mechanism for incremental backups.
tkaiser Posted June 12, 2016 Posted June 12, 2016 I will write a script to convert an already existing image (or even an already used system on a SD-Card) to btrfs. This script will also handle the related problems (creating an ext4 boot partition, updating fstab, etc.). I think this would be a great addition. But unfortunately this seems to be the wrong way (since you will waste enourmous amounts of time with this 'change things later with a script' approach -- BTDT many times). I believe it would be better to combine the benefits of btrfs and being able to use SD cards of any size in another way: check the target audience: people who know why btrfs/snapshots rock and assume they want to do something serious with Armbian and their device(s) assume installation media requirements are understood (it's 2016 and there's no reason to use an old crappy and ultra slow SD card with 4 GB that has been found in the drawer) make use of partitions That being said I think we should make using at least a 16GB SD card a prerequisit (we have already a list of recommended SD cards and good performance needs larger cards anyway). Then we can add another build variant: choose btrfs for the rootfs, choose 8 GB in size (so we end up with a small ext4 partition and a large btrfs partition) enhance the firstrun script to automatically partition the remaining size on the card enhance the check_first_login.sh script to ask the user whether he wants to use this partition for /home or for something else -- /var/www for example adjust fstab and settings accordingly Since we're talking here about a H3 board to give away unfortunately this won't work (now) since using btrfs with our 3.4.112 legacy kernel on H3 boards would be simply insane (tons of unfixed bugs, btrfs stability is even with most recent kernels not that great), mainline kernel is a requirement for btrfs anyway. But since we're not that far away from using mainline on H3 boards and development is time consuming (especially when modifying partitions in a realiable way when 4 different distros are involved!) it's worth to start now.
Gravelrash Posted June 12, 2016 Posted June 12, 2016 How do i get the .deb to you when i have completed it? Id better do something constructive as you have selected me to receive a free board Since I'm currently trying to integrate @lex' latest patches to support the cheap GC2035 Orange Pi camera as driver into our sun8i legacy kernel another useful task would be to package @lex' adopted fswebcam version so we can deliver this armbian-gc2035-fswebcam package through Armbian repo. Anyone?
zador.blood.stained Posted June 12, 2016 Posted June 12, 2016 How do i get the .deb to you when i have completed it? This is not about providing .deb file, this is about creating packaging scripts (again). Once you (or anyone else) makes something that works, just open a pull request to https://github.com/igorpecovnik/lib
zador.blood.stained Posted June 12, 2016 Posted June 12, 2016 For people who wanted to help with documentation, here is your chance: This was added recently, so it's a good idea to extend this to more or less universal HOWTO for different boards. Things to add: Generic prologue about kernel version and configuration files(TL;DR - fex/bin for legacy, Device Tree for mainline); Decompiling/editing/compiling back .bin/.fex files with some practical examples; Decompiling/editing/compiling back .dtb/.dts files, matching original and decompiled syntax, some practical examples; Compiling DT from scratch if changes are too big, some examples for this case too; Troubleshooting (how to check that changes were correctly applied, where to look for error logs (dmesg), etc); I'm not saying that one person should do all of this, so feel free to claim topics you are most familiar with.
lanefu Posted June 12, 2016 Posted June 12, 2016 ---snip-- @lanefu Issues are scattered around, but most of them are under lib -> issues We don't label or arrange them since it takes extra effort, but if you are willing to take issues on another level, we will save lot's of time on a long run. Both ideas are valuable and welcomed, just start If you need any extra tools or permissions, please send me a PM. Okay since i'm pretty much doing the same thing with my team at work, I'll take on establishing clearer process / procedures for issue tracking. Since both the forum and github support a concept of tagging there's some opportunities there for integration. We want to make issue/task creation is easy, but deliberate enough that we don't end up with having to filter a bunch of noise. 1
xcasex Posted June 12, 2016 Posted June 12, 2016 Well, ping me on irc (same nick on freenode, or dm me if you need help with packaging it up) also, writing howto's is equally important. because without good docs, we have bad routines/processes. kick ass! @all I have had a go at what you suggested and i need to spend a LOT more time getting to grips with the fundamentals, so I wont be claiming any of the deb and packaging side. I think that @Xer0 & @madilabs & @xcasex & @rella would be more suited to this task I will resign myself to doing HOWTO's for armbian as and when i spot the need for one.
Xer0 Posted June 12, 2016 Posted June 12, 2016 @Xer0 https://github.com/igorpecovnik/lib/blob/master/desktop.sh We need implementation there, within build process. Current way is outdated, a bit dirty and need an upgrade. It looks like it's still working on H3 but failing on A20 since certain libs needs updated versions. We need a solution which is build from sources to get rid of outdated prepacked libs and pack output into .deb that we can provide an update. Only complete solution makes sense since we already have some temporally solution. So, do you want media_build drivers to be _compiled_ during build process, or just included? media_build is tricky to build reproduceably, as it pulls latest sources from the upstream tree, while beeing quite slow in adapting its supplied backport-patches, which frequently breaks it for example, a new source tar is up since friday, with it fails at apply patches stage but manually wget'ing and untaring the previous one will work normally also, some drivers wont compile for arm even if seemingly supported by the kernel so we would need to "freeze" a working media_build+linux-media combo and fork it somewhere
tkaiser Posted June 13, 2016 Posted June 13, 2016 I just added another probably interesting task (mainline kernel images for H3 devices) and started an own thread for it: http://forum.armbian.com/index.php/topic/1403-finishing-vanilla-kernel-support-for-h3/
Gravelrash Posted June 13, 2016 Posted June 13, 2016 I claim the HOWTO for configuring NFS and shareing it it to alow for NFS booting. I have also shared my quick and dirty build script for fswebcam if its suitable to be included for your use. https://github.com/gravelrash/lib/blob/master/extras/fswebcam.sh
Gravelrash Posted June 13, 2016 Posted June 13, 2016 untaring the previous one will work normally also, some drivers wont compile for arm even if seemingly supported by the kernel so we would need to "freeze" a working media_build+linux-media combo and fork it somewhere +1 If we freeze at a set working point then we at least know that this will work, and can modify the variable for its download to match the next known working one when its stable.
dimag0g Posted June 16, 2016 Posted June 16, 2016 Hi, I've done some tweaks to make OpenGL games work on my Mali boards (simple games like billard-gl work, openarena does start but it's a strech to call it fully playable). It's not finished yet, but the results are already usable despite occasional bugs. Would you consider adding a package which is not 100% stable? I don't have any experience with packaging scripts, but I can try. I'm taking 2 weeks of holiday in July, I'm pretty sure I will have time to do it by the end of the month. Another question is about changing the kernel configuration. Since OpenGL is implemented via GLES, it's necessary to give a share of RAM to Mali GPU. How should my hypothetical package go about changing /boot/boot.scr? Disclamer: I'm not the author of this GL library, those guys are.
Gravelrash Posted June 16, 2016 Posted June 16, 2016 @all preamble - i can quite straightforwardly craft a script that clones a repository, compiles the software and creates a .deb package Question : to include this for your builds as a standalone package for downloading at a future time, what is your preferred directory structure? I presume that this wont be at the build image stage of the script or even in the image itself. So where would your preferred locations be so i can mirror them into the build script? have i missed a documentation piece of a link somewhere? e.g. clone into /src/"packagename" -> compile -> deb creation -> build output to /?? i can then contribute successfully
zador.blood.stained Posted June 16, 2016 Posted June 16, 2016 @Gravelrash Please take a look at updated hostapd compilation script for example. Directory structure is easy to fix afterwards, but if you want to do it right from the start, here is recommended setup: If package can be cross-compiled, it's better to put in into "$SOURCES/<package_name>" directory and compile & pack it in-place. If package needs to be compiled in chroot, put in into "$CACHEDIR/sdcard/root/<package_name>", so you can access it in chroot in "/root/<package_name>". Don't forget to remove this directory after you successfully built the package. After building a package copy it to "$DEST/debs" If there are no dependencies between packages, create new script in "extras" for each package. If some packages needs to be built in specific order, you can build several packages in one script. 2
sysitos Posted June 16, 2016 Posted June 16, 2016 Hi, I also want to claim a task. Imho, from all the available arm boards the Orange Pi+ 2E could be one of the best mini servers. That's why, I would provide a usb automount/autoshare udev/systemd script for it to simplify some server tasks. In the meaning of simple plugin USB drives and forget about it. (I know, there are already some automount tools, but none of them meet my needs.) So what exactly does it do: for plug in: mounts USB drives with their label under specified mount point, ignore drives on black list and fstab, mount drives with uuid if label is already mounted optional: share the new mounted drive with samba and nfs (create a special exports file), other servers as ftp are possible too for unplug unshare and unmount the drive, clean up the changed files To be fair, all this is already done and is running here fine with OMV and my ODROID C2. (OMV was the first working image for the C2). I have only tested the new C2 armbian image. So what must be done until this goes public on armbian? 1. clean up the code 2. create a deb package optional 3. use external default/custom ini files (better than changing the whole script) 4. extend the script for other servers and other purposes Ok, and some docu could be helpfull Maybe it's useful. Any thoughs? PS: script is working only with systemd cu sysitos 1
Gravelrash Posted June 17, 2016 Posted June 17, 2016 One thing i have come across while looking into packages and packaging is "post install scripts". I would look to use this method something along the lines of convert file from y -> x format, use sed or awk or whatever method you deem fit to find and replace/modify the string you are interested in then convert the file back x -> y format. Hi, Another question is about changing the kernel configuration. Since OpenGL is implemented via GLES, it's necessary to give a share of RAM to Mali GPU. How should my hypothetical package go about changing /boot/boot.scr?
R2D2_C3PO Posted June 20, 2016 Posted June 20, 2016 I looked a bit into your idea and tried to estimate if is is possible in a reasonable time. While at this task I already submitted a patch to the check_first_login script (https://github.com/igorpecovnik/lib/pull/366). My new task proposal would be this: Write a script which asks the user at the first login what he wants to do with the unpartitioned space on the SD-Card. Options: a) Use whole SD-Card for root (as it currently happens automatically) Partition disk and use space for a configurable mountpoint with any supported filesystem (perhaps with some options to choose from, e.g. /home, /var/www, ...) c) Leave space unpartitioned (options: ask again at next boot; never ask again) When selecting option fstab will be updated automatically. The amount reserved for root is user selectable (with reasonable limits). The current mechanism to resize the root fs at first boot would be disabled. Doing this at first login is as good as during first boot. Does this proposal sound good? I think this would be a great addition. But unfortunately this seems to be the wrong way (since you will waste enourmous amounts of time with this 'change things later with a script' approach -- BTDT many times). I believe it would be better to combine the benefits of btrfs and being able to use SD cards of any size in another way: check the target audience: people who know why btrfs/snapshots rock and assume they want to do something serious with Armbian and their device(s) assume installation media requirements are understood (it's 2016 and there's no reason to use an old crappy and ultra slow SD card with 4 GB that has been found in the drawer) make use of partitions That being said I think we should make using at least a 16GB SD card a prerequisit (we have already a list of recommended SD cards and good performance needs larger cards anyway). Then we can add another build variant: choose btrfs for the rootfs, choose 8 GB in size (so we end up with a small ext4 partition and a large btrfs partition) enhance the firstrun script to automatically partition the remaining size on the card enhance the check_first_login.sh script to ask the user whether he wants to use this partition for /home or for something else -- /var/www for example adjust fstab and settings accordingly Since we're talking here about a H3 board to give away unfortunately this won't work (now) since using btrfs with our 3.4.112 legacy kernel on H3 boards would be simply insane (tons of unfixed bugs, btrfs stability is even with most recent kernels not that great), mainline kernel is a requirement for btrfs anyway. But since we're not that far away from using mainline on H3 boards and development is time consuming (especially when modifying partitions in a realiable way when 4 different distros are involved!) it's worth to start now.
dimag0g Posted June 21, 2016 Posted June 21, 2016 Write a script which asks the user at the first login what he wants to do with the unpartitioned space on the SD-Card.Options: a) Use whole SD-Card for root (as it currently happens automatically) Partition disk and use space for a configurable mountpoint with any supported filesystem (perhaps with some options to choose from, e.g. /home, /var/www, ...) c) Leave space unpartitioned (options: ask again at next boot; never ask again) That would certainly be useful for me. I'm not sure if covering all the cases is necessary, but at least there should be an option to refrain from extending root, or choose its future size. I had to shrink it back to 12 GB on my 32 GB card to accomodate the other partition I needed (/opt)
tkaiser Posted June 21, 2016 Posted June 21, 2016 Does this proposal sound good? Well, IMO a few things have to be considered: the firstrun concept started back at a time where no user interaction after installation happened, so while we're now enforcing user creation we could really move tasks from firstrun to check_first_login.sh But we have to keep in mind that some of our users already skip check_first_login.sh since they use customized images that should start totally unattended (also skipping user creation process). So we need to support a trigger or configuration file to be able to revert back to defaults (resizing partition as its currently implemented without any user interaction) At least from my point of view we should also think about integrating nand-sata-install.sh at this early point (after these issues are resolved). Given an Armbian image on a 64 GB SD card inserted into any of the eMMC equipped newer boards I would like to be able to choose a variant where rootfs (including $HOME) end up on eMMC afterwards, the SBC is able to boot with inserted SD card and this card is then empty, re-partitioned and included in /etc/fstab afterwards after the necessary reboot Allowing btrfs should only happen with most recent kernel versions (and needs a warning that switching back to legacy kernel isn't an option) Given different boot priorities on different SBC we support and that partitioning is always 'fun' since tool behaviour changes from time to time the whole process is quite challenging but can be done step by step (needing some coordination eg. through github issues from time to time) and even if implementing the whole thing might take some time, tested code snippets known to work can be integrated in current scripts to enhance behaviour
vlad59 Posted June 21, 2016 Posted June 21, 2016 For people who wanted to help with documentation, here is your chance: This was added recently, so it's a good idea to extend this to more or less universal HOWTO for different boards. Things to add: Generic prologue about kernel version and configuration files(TL;DR - fex/bin for legacy, Device Tree for mainline); Decompiling/editing/compiling back .bin/.fex files with some practical examples; Decompiling/editing/compiling back .dtb/.dts files, matching original and decompiled syntax, some practical examples; Compiling DT from scratch if changes are too big, some examples for this case too; Troubleshooting (how to check that changes were correctly applied, where to look for error logs (dmesg), etc); I'm not saying that one person should do all of this, so feel free to claim topics you are most familiar with. Hi I'm currently working on this and while I'm fluent (or at least I hope so) in FEX modification (two commits were merged), DT is newer for me. I'll certainly need some guidance here. My first question is how do you know which DTB file you're currently using. For now my method it to use cat /proc/device-tree/model and try to guess the file in /boot/dtb. It worked, but there's certainly a better method. boot.cmd is using ${fdtfile} so no help here boot.src is compiled so no help there either. Thanks in advance.
martinayotte Posted June 22, 2016 Posted June 22, 2016 I've received my OPi-Plus-2E this afternoon and done a fresh new Mainline build 5.14 kernel 4.6.2 I've soon discovered that 8189fs was not part of my fresh build. I've done the same thing I did few days ago with my OPi-Lite : patch DTS and build rtl8189fs drivers borrowed from Legacy after apply Hans equivalent patch suggested to me by @jernej, and got it working. I will probably probably have to prepare real patches for the official builds soon. I've seen that eth0 is also not working, but I didn't investigated yet... There is also the eMMC not present, but I think this one is related to new patches needed, I've seen several post on linux-sunix mailing list. BTW, does someone have a link to schematic/board-layout for the OPi-Plus-2E board ? (I didn't find it, I have all of them for others board I have, PC, One, Lite, but I wish to get this one as well as OPi-PC+ that I should received next week) 1
tkaiser Posted June 22, 2016 Posted June 22, 2016 I've seen that eth0 is also not working, but I didn't investigated yet... Should be working as soon as the .dts bits for Plus or BPI M2+ are applied (I prepared them as patch 2 months ago). And yes, it would be cool if you could provide real patches to get everything working. I tested yesterday some other stuff (btrfs installation on eMMC with mainline kernel -- for eMMC all that's needed should be a an additional .dts node and the correct u-boot settings we added with 5.14) just to realize that exchanging u-boot with the one for OPi Plus 2E from apt.armbian.com (built on June 1st) led to strange symptoms compared to before (I used my preliminary 5.07 image back from April that works also flawlessly with OPi Plus 2E since .dts differences are only one led and one USB port left). So I gave up again EDIT: Schematics aren't released yet AFAIK
zador.blood.stained Posted June 22, 2016 Posted June 22, 2016 My first question is how do you know which DTB file you're currently using. For now my method it to use cat /proc/device-tree/model and try to guess the file in /boot/dtb. It worked, but there's certainly a better method. This is the right approach if you don't know exact name of DTB file. ${fdtfile} variable in u-boot is defined at compile time (in defconfig file for the board) and this value is not accessible outside of u-boot.
vlad59 Posted June 22, 2016 Posted June 22, 2016 This is the right approach if you don't know exact name of DTB file. ${fdtfile} variable in u-boot is defined at compile time (in defconfig file for the board) and this value is not accessible outside of u-boot. Thanks for the confirmation. Another quick question, currently I can install the kernel headers (patched Armbian version). Would it be possible to have the complete patched kernel sources through apt.armbian.com ? If that's possible, we could have access to the real DTS file (with comments, unprocessed) and use dtc on them (like shown here). It would make the job easier most of the time and there would be no need to recompile the full kernel. I always find it hard to manually edit decompiled DTB.
zador.blood.stained Posted June 22, 2016 Posted June 22, 2016 Another quick question, currently I can install the kernel headers (patched Armbian version). Would it be possible to have the complete patched kernel sources through apt.armbian.com ? If that's possible, we could have access to the real DTS file (with comments, unprocessed) and use dtc on them (like shown here). It would make the job easier most of the time and there would be no need to recompile the full kernel. I always find it hard to manually edit decompiled DTB. While it's possible to pack full kernel source into a .deb package, installing a package with such number of small files on a device will be a nightmare (especially on a slow SD card). It may be better to provide ony "arch/arm/boot/dts" and "include" directories - this should be enough for compiling DT from scratch.
vlad59 Posted June 22, 2016 Posted June 22, 2016 While it's possible to pack full kernel source into a .deb package, installing a package with such number of small files on a device will be a nightmare (especially on a slow SD card). It may be better to provide ony "arch/arm/boot/dts" and "include" directories - this should be enough for compiling DT from scratch. I'll try that tonight. Thanks for the idea.
vlad59 Posted June 22, 2016 Posted June 22, 2016 This was actually easy, I only need the dts directory and the kernel headers (they were already installed) and that's it. The compile script look like that : #!/bin/sh IDE=sun7i-a20-bananapro SRC=$IDE.dts TMP=$IDE.tmp.dts DST=$IDE.dtb cpp -nostdinc -I /usr/src/linux-headers-4.5.2-sunxi/include -undef -x assembler-with-cpp $SRC > $TMP dtc -O dtb -b 0 -o $DST $TMP #rm $TMP I'll check tonight if the DTB I compiled is exactly the same as the one from Armbian. EDIT : An additional interest of having all the DTS files is that you can quickly find the DTB file you're currently using : root@bananapipro:~/test/dts# grep "`cat /proc/device-tree/model`" *.dts sun7i-a20-bananapro.dts: model = "LeMaker Banana Pro"; 1
martinayotte Posted June 23, 2016 Posted June 23, 2016 I've been working on the rtl8189fs patches for Mainline. I think it is better that I start new thread for it ...
Gravelrash Posted June 24, 2016 Posted June 24, 2016 Just a note to confirm i received my board yesterday morning. 1
Recommended Posts