[RfD] Make Armbian more Etcher friendly

[tkaiser]

Hi all,

 

please be aware that Pine folks in the meantime understood that recommendations to use WinDisk32Imager, dd and other non verifying burning tools are wrong, tried to recommend Etcher instead (as usual sabotaged by at least their most famous forum moderator) and now use finally a 'branded' Etcher 2.0 beta: https://forum.pine64.org/showthread.php?tid=4481&pid=27520#pid27520

 

I don't think it's correct to do it this way (and at least the macOS application sucks since not digitally signed and using a wrong so called 'bundle identifier') but anyway. Let's take this for a foresight.

 

Etcher 2.0 uses catalog files in such a way: http://files.pine64.org/sw/pine64_installer/json/default/pine_a64_1_2gb.json

 

Pine folks added 3 Armbian images but unfortunately hosted on their server and also in another format with other checksums than ours. It's us to blame here since the compression scheme we use is not supported by Etcher's 'stream+decompress+burn' mode (and I doubt it will anytime soon).

 

Currently on Xenial xz-utils is 5.1.1alpha+20120614-2ubuntu2 (multithreaded compression came with 5.2) so let's use 7-zip instead. Testing on a dual core machine with HT enabled (so 4 cores seen by OS) with pretty fast SSD (no storage bottlenecks) and using '7za a -t7z -bd -m0=lzma2 -mx=3 -mfb=64 -md=32m -ms=on' vs '7za a -txz -mx=3 -mfb=64 -md=32m':

Bananapi_Ubuntu_xenial_next_4.10.14.7z: 1m14.631s
Bananapi_Ubuntu_xenial_next_4.10.14.xz: 1m10.530s
Pine64_Ubuntu_xenial_default_3.10.104_desktop.7z: 2m56.346s
Pine64_Ubuntu_xenial_default_3.10.104_desktop.xz: 3m5.232s

231438878  5 Mai 08:19 Bananapi_Ubuntu_xenial_next_4.10.14.7z
231438596  5 Mai 08:22 Bananapi_Ubuntu_xenial_next_4.10.14.xz
569026080  5 Mai 08:34 Pine64_Ubuntu_xenial_default_3.10.104_desktop.7z
569025792  5 Mai 08:30 Pine64_Ubuntu_xenial_default_3.10.104_desktop.xz

Looks good but Etcher doesn't like these files

 

So let's try xz-utils (since the only machine later facing problems will be @Igor's build host that might need an upgrade to xz-utils 5.2 or above to benefit from multithreading): 'xz -c -F xz -C crc32 -3 -T 0' -- results look nice but Etcher still doesn't like those files.

 

So prior to spending more time on this... what do you others think? Is switching from .7z to .xz worth the efforts or will it cause too much confusion?

 

Besides that we should have Etcher development in mind. Both regarding changes on our side (provide checksums, generating the OS image catalog automagically by build-all.sh for example) and on their side (discussing the use of torrents for example since central download servers might become a bottleneck -- at least that's true for files.pine64.org). In the past they always listened to suggestions and constructive feedback  

[tkaiser]

Forgot. Worth a read too: https://github.com/resin-io/etcher/blob/bdf920f86459d4ab63e62bd3b380f6c2d336ceac/docs/EXTENDED-ARCHIVES.md

[Igor]

Downloading from within Etcher would be a step back ... it's possible to have one URL and mirrors behind that, but that's bringing costs and yet another technology. I doubt they will implement torrent client soon

 

If I understand correctly, it's possible to load and burn .zip file with image if it contains proper meta data. But they don't accept 7z? XZ format contain only one stream

Than it would be better to compress everything to a .zip with aproproate .jsone file. This this zip is thrown directly into the etcher ...

 

Do we need our branded version of Etcher ? 

[tkaiser]

7 hours ago, Igor said:

I doubt they will implement torrent client soon

 

I guess it's Alexandro himself who mentions/mentioned torrents the most. Just check torrent site:github.com/resin-io

 

For me 3 things should be considered:

• The 'Pine case' (that means compatiblity to Etcher 2.0 requirements -- switching back to .zip or .xz so Pine folks and other vendors who start to use branded versions of Etcher can put Armbian images into their catalogue that point to original download location with original checksum and both $instructionsUrl and $releaseNotesUrl pointing to our servers. At the same time we should check our publishing process to generate both automatically)
• Talking to Etcher/Resin folks how to get into their catalogue (automatically submitting metadata with new releases)
• Let download integrity be checked automagically by Etcher (first steps are done but not implemented yet)

7 hours ago, Igor said:

Do we need our branded version of Etcher ?

 

I almost hate the idea (really, I prefer to fix things upstream instead of doing 'our own thing').

 

IMO a short-term goal should be to switch to a compression format that can be used with Etcher's streaming mode (.xz provides way better compression ratio but it's an 'Etcher 2.0' only solution due to just containing the image itself and all metadata being part of the 'catalogue', AFAIK same problem with the ZIP container format now since not supported by current Etcher version).

 

What do we package now and in the past in our  [7-]zip files?

Armbian_5.25_Pine64_Ubuntu_xenial_default_3.10.104_desktop.img
Armbian_5.25_Pine64_Ubuntu_xenial_default_3.10.104_desktop.img.asc
armbian.txt
armbian.txt.asc
sha256sum.sha

IMO the only interesting thing is a mechanism to check authenticity and download integrity. With Etcher 2.0 'catalogue' scenarios that's easy and as soon as Etcher implements checksum verification if it's part of the filename then we're also done for torrent scenarios now. What am I'm missing (besides the future potential of .etch format and ZIP containers)?

[Igor]

On 5. 5. 2017 at 6:06 PM, tkaiser said:

The 'Pine case' (that means compatiblity to Etcher 2.0 requirements -- switching back to .zip or .xz so Pine folks and other vendors who start to use branded versions of Etcher can put Armbian images into their catalogue that point to original download location with original checksum and both $instructionsUrl and $releaseNotesUrl pointing to our servers. At the same time we should check our publishing process to generate both automatically)

Than the best way is still using .zip files ... even compressed files are almost 1/3 larger than alternative,  .xz can't store multiple files so it has limited usage, while .zip can be dumped directly to Etcher and used for "Pine Etcher 2.0 JSON" implementation and our end user download.  

 

I am not sure what to do 

[zador.blood.stained]

6 minutes ago, Igor said:

I am not sure what to do 

I'm in favor of leaving everything as is for now. This Etcher implementation is still in beta, we have enough issues and migrations to deal with, so jumping from one compression format to another IMO is not something urgent.

[tkaiser]

Just as a reference: https://github.com/resin-io/etcher/issues/711#issuecomment-307168759