research Need your help - what else beside Etcher

[Tido 187]

On 3/21/2020 at 9:31 AM, Tido said:

A must was - standard writing verification.

 

3 hours ago, legogris said:

verify sha1sum:

I don't know if this is the same, do you?

 

[legogris 10]

10 hours ago, Tido said:

 

I don't know if this is the same, do you?

 

Yes - if those two commands return the same output, it means the data in img file and on the sdcard have the same sha1 hash (and therefore contain the exact same bytes).

[Tido 187]

2 hours ago, legogris said:

Yes -

Well, I don't think so. Etcher for example writes the SDcard and then, then it reads it and checks if it is the same. You only look at the checksum - which is better than not doing it.

You do a checksum verification.

We (at armbian) talk about: writing verification (bit by bit) on the SDcard.  @OrangePee, do you know which of the two USBimager does?  I am a bit uncertain right now.

 

[OrangePee 4]

16 hours ago, Tido said:

 

We (at armbian) talk about: writing verification (bit by bit) on the SDcard

 

This it what usbimager does if I interpret the docs right

 

Quote

The last option, the selection box selects the buffer size to use. The image file will be processed in this big chunks. Keep in mind that the actual memory requirement is threefold, because there's one buffer for the compressed data, one for the uncompressed data, and one for the data read back for verification.

 

[legogris 10]

On 4/17/2020 at 3:44 PM, Tido said:

Well, I don't think so. Etcher for example writes the SDcard and then, then it reads it and checks if it is the same. You only look at the checksum - which is better than not doing it.

You do a checksum verification.

We (at armbian) talk about: writing verification (bit by bit) on the SDcard.  @OrangePee, do you know which of the two USBimager does?  I am a bit uncertain right now.

 

So, this is how it works (and advance apologies if I misread the conversation and you're talking about something else):

 

First off, if you want to talk semantics, sha1 is technically a cryptographic hash function, not a checksum algorithm. One could even argue that it's overkill for this use-case and that an algortithm made for checksums, such as CRC, is more appropriate (less secure but by far enough to detect random write or read errors, but more performant).

 

The data on the card is read, bit-by-bit (well, on a lover level the reads are buffered for performance reasons, but it makes no practical difference otherwise). The data is then put through the sha1 hash function, which produces a 160-bit hash. A single bit being changed produces a completely different hash.

 

The odds of the hashes matching up in case every single bit doesn't is 1 in ~10^18. For practical purposes, low enough odds to be practically impossible. When we're at those odds, you might as well start thinking about the probability that the read during your verification steps gives you the expected output due to a read error exactly matching the inverse of the write error during the write, RAM errors and cosmic radiation. If you're concerned about well-funded adversaries with a huge dedicated compute cluster deliberately giving you a fake image with the same matching hash, then you might want to look at more cryptographically secure hash functions - in which case replace the `sha1sum` command above with the slower but more secure `sha512sum` (512 bits, or 1 in 10^154). The chances of a collision are small enough that on average, with all the worlds current computing power, we'd be talking millions of years. The number of atoms in the entire universe is believed to be around 10^80. Pick two random atoms in the known universe and the odds of those being the same atom is still vastly higher than SHA-2 producing the same hash after a write error.

 

TLDR; For your purposes, if the checksum validates, so does the written data (given that the checksum verification here is made between data read back from the written card vs the source file and not just between the source file and a website). Etcher did CRC32 until 2018, and is now on SHA512. Looking at the source, usbimager actually reads back the source and the target into memory and does a memcmp (straight-up byte-by-byte comparison): https://gitlab.com/bztsrc/usbimager/-/blob/master/src/main_libui.c#L147

These are all valid approaches, only caveat with the last one being that you need to be able to fit 2x the image size into RAM during the validation

[Tido 187]

To get the picture, you have start read here: https://github.com/armbian/documentation/issues/66  and then the first post in this thread. Because I moved the discussion from Github into the forum.

 

11 hours ago, legogris said:

If you're concerned about well-funded adversaries with a huge dedicated compute cluster deliberately giving you a fake image

and if you are patient enough to read the above, you understand that it is not because of a tin-foil hat, but  TL;DR to improve the user experience.

That said we have a section Board doesn't start: https://forum.armbian.com/forum/36-board-doesnt-start/ and in the documentation: https://docs.armbian.com/User-Guide_Getting-Started/#how-to-prepare-a-sd-card  these weren't there from begin, but if you fly over the topics. You see why we are looking for an automatic fool proof SW

 

Thank you for the detailed explanation, really interesting, especially the part of Etcher vs memcmp of USBimager. As you see sometimes the reasons are much simpler.

 

My AMD Ryzen 5 2600 Six-Core, 16GB RAM writes thoes card easily on the fly, as written above in about  60 seconds. I know, not everybody has that, but until last year I had 4GB and Etcher-1.2.1-x86_64.AppImage, it worked flawlessly too.

 

[guidol 280]

USB-IT did get some updates and has "now" (since v1.75beta) a GUI-option for verify after a Backup/Restore.
The .ZIP of v1.80 ( at https://www.alexpage.de/ ) is - nice small - with under 1MB  

 

Maybe we can recommend this tool also?
Download USB Image Tool 1.80 (ZIP, 848 KB):
http://www.alexpage.de/download/usbit/usbit.zip
 

[Tido 187]

This could be an option for Windows users if...

4 hours ago, guidol said:

a GUI-option for verify after a Backup/Restore

this is set when you download it and it is meant that it re-reads to verify that the writing is error free. Just like discussed before in this thread.  Plus this function...

 

On 4/8/2020 at 7:36 AM, Werner said:

Windows,  what I have noticed so far is that USBimager does not hide non-sdcard devices

https://forum.armbian.com/topic/13421-need-your-help-what-else-beside-etcher/?do=findComment&comment=98830

 

[lanefu 363]

I've updated our canned response to below.... I saw no need to revoke Etcher's status.. as it is easier to install, but I updated it's description to be more....accurate

 

-----------------------------

 

Armbian's archives can be uncompressed with 7-Zip on Windows, Keka on OS X and 7z on Linux.

 

Images shall only be written with imaging tools that validate burning results.  This saves you from corrupted SD card contents.

 

Approved Tools:

• USBImager a lightweight cross-platform imaging tool
• Balena Etcher an electron / node.js based cross-platform imaging tool

[Igor 2347]

1 hour ago, lanefu said:

Armbian's archives can be uncompressed with 7-Zip on Windows, Keka on OS X and 7z on Linux.

This has to be adjusted since with next release we only have xz https://armbian.atlassian.net/browse/AR-213 Odroid C1 is already made on new standards https://dl.armbian.com/odroidc1/ ... what bothers me is:

https://dl.armbian.com/odroidc1/Focal_current_minimal_nightly
https://dl.armbian.com/odroidc1/Focal_current_minimal_nightly.asc (signature)

https://dl.armbian.com/odroidc1/archive/Armbian_20.02.14_Odroidc1_buster_current_5.4.39_minimal.img.xz

https://dl.armbian.com/odroidc1/archive/Armbian_20.02.14_Odroidc1_buster_current_5.4.39_minimal.img.asc (signed is image, not compressed image. Is that alright?)

[OrangePee 4]

On 5/16/2020 at 3:42 PM, lanefu said:

• Balena Etcher an electron / node.js based cross-platform imaging tool

 

I'm missing information here that this tool comes bundled with adware+spyware, probably violates european law (gdpr) and should be heavily avoided when using a metered network connection due to it's insane size (almost half a gigabyte till the slow ui starts).

 

  

On 4/7/2020 at 11:33 AM, Igor said:

and leave Etcher there as "also usable, but includes spyware"  

 

[Igor 2347]

54 minutes ago, OrangePee said:

I'm missing information here that this tool comes bundled with adware+spyware, probably violates european law (gdpr) and should be heavily avoided when using a metered network connection due to it's insane size (almost half a gigabyte till the slow ui starts).

 

https://docs.armbian.com/Process_Contribute/

https://github.com/armbian/documentation/blob/master/docs/User-Guide_Getting-Started.md

also this https://armbian.atlassian.net/browse/AR-213 needs to be integrated

 

Thank you for your help.

[OrangePee 4]

 

So if I understand right, this:

 

On 5/16/2020 at 3:42 PM, lanefu said:

 

Armbian's archives can be uncompressed with 7-Zip on Windows, Keka on OS X and 7z on Linux.

 

Images shall only be written with imaging tools that validate burning results.  This saves you from corrupted SD card contents.

 

Approved Tools:

• USBImager a lightweight cross-platform imaging tool
• Balena Etcher an electron / node.js based cross-platform imaging tool

will be the new version of https://github.com/armbian/documentation/blob/master/docs/User-Guide_Getting-Started.md, correct?

 

It would be really nice @lanefu to add the "anti-features" too.

 

Thank's @Igor for your links. I will try to contribute a little... let's see if my pull requests get accepted

 

BTW: On the website on every site of board (for example: https://www.armbian.com/orange-pi-4/) there is also a direct link to the etcher site (SD writing tool).

Would be maybe good to change this and link directly to the user guide (section: writing tools) instead?

[Tido 187]

34 minutes ago, OrangePee said:

a direct link to the etcher site

A good idea to link it to the user-guide.

@Igor, I tried to help, but my permission within WordPress are limited. So, this is up to you. 

 

@OrangePee ,  in regard to you PR, there are two sections to address, if you don't mind:

https://docs.armbian.com/User-Guide_Getting-Started/#how-to-prepare-a-sd-card  

https://docs.armbian.com/User-Guide_Basic-Troubleshooting/#writing-images-to-the-sd-card

 

[lanefu 363]

added annotation that it may container spyware

[Igor 2347]

On 5/22/2020 at 5:27 PM, OrangePee said:

Would be maybe good to change this and link directly to the user guide (section: writing tools) instead?

Changed. But now we need a big download for our default tool and smaller for optional tool. When user clicks "Tool" on the download, he already expects to download the tool without lecturing about SD cards and finding links in the text ... https://www.goodreads.com/book/show/18197267-don-t-make-me-think-revisited

[xwiggen 6]

other alternative; raspberry pi imager.

 

https://www.raspberrypi.org/downloads/

 

[Tido 187]

1 hour ago, xwiggen said:

raspberry pi imager

IIRC we already discussed this SW in this thread.

[guidol 280]

On 3/23/2020 at 8:44 PM, OrangePee said:

 

Newest Version 1.0.2 has this automatically turned on 

USBImager V1.0..4 is now 2 months old, but I today recommend by cnx-software:
https://www.cnx-software.com/2020/09/08/usbimager-a-lightweight-alternative-to-balenaetcher/

 

Filter out drives on the same disk as C:

https://gitlab.com/bztsrc/usbimager/-/commit/1775cdc628a3d28ce454c181952aa283334d0991

[guidol 280]

On 3/21/2020 at 11:31 AM, Tido said:

The question  and  tests  should be. Are there alternatives to Etcher?

A must was - standard writing verification.

and now etcher so:

v1.5.111 - (2020-11-23)

Remove "Validate write on success" setting. Validation is always enabled, press the "skip" button to skip it. [Alexis Svinartchouk]

[Tido 187]

On 12/10/2020 at 12:24 PM, guidol said:

and now etcher so:

I don't know how to understand that. However, I am more than happy with USBImager

 

I read in the PDF manual about adding my user its group and now not even sudo is necessary to launch it. By the way, new release and this time with the .deb image https://gitlab.com/bztsrc/usbimager/-/releases  and if you run  man usbimager  you see some switches for more functions.

[Ash Martin 0]

Hi there, I’ve just deployed a fresh Windows 10 x64, installed Etcher 1.5.109 and flashed an image to USB. I started Etcher (etcher.net) normally and got an elevation prompt after hitting the flash button.

Edited March 13 by Werner
link disabled

[Werner 427]

2 hours ago, Ash Martin said:

Hi there, I’ve just deployed a fresh Windows 10 x64, installed Etcher 1.5.109 and flashed an image to USB. I started Etcher normally and got an elevation prompt after hitting the flash button.

Never heard of etcher.net. Maybe scam?

Make sure you grab the tool from https://www.balena.io/etcher/

[guidol 280]

6 hours ago, Werner said:

Never heard of etcher.net. Maybe scam?

Make sure you grab the tool from https://www.balena.io/etcher/

 

Info on Reddit
 

Quote

DO NOT download etcher from etcher.NET!

I made this mistake while going from windows 10 to pop os. If you're going to use etcher (balena etcher) to flash a usb go to [balena.io/etcher/]. This is their real site! etcher.NET gives you nothing but a bogus installer that tries to get you to install several unwanted software, before proceeding to NOT install etcher. This is what I get for trying Bing, which recommended this above the legitimate site.

 

[TRS-80 145]

Maybe I will try and find a few minutes to add a short note to Docs about this.  Unfortunately I am pretty busy though.  If someone else can get to it, that would be awesome.

[Tido 187]

2 hours ago, TRS-80 said:

to add a short note

@Ash Martin @Werner @guidol  You do know that this thread is about a BETTER software than Etcher, right?

And you do know we recommend USBimager OVER Etcher, right?

 

If not, I recommend to read the first couple posts of this thread. Instead of misleading people @Ash Martin  just be quiet. It is not as if you had to comment every post.

[Tido 187]

On 4/8/2020 at 12:20 PM, Werner said:

until there are some ways implemented to make sure that no internal device can be killed by accident.

IIRC, this is now the case. Please test on your Windows box and report. Thank you.

[TRS-80 145]

Sorry, no, I did not realize the thread we were in at all (I got here from All Activity stream and was replying to the two posts above mine).

 

Anyway, back to the topic.  Just today I came across bmap-tools, which not only appears to perform the all-important write verification, but also apparently have some advantages over tools like dd because it is capable of writing sparse file maps, and thus can save a lot of time while writing (depending on the image).  Another interesting feature, you can give it a URL as an argument and it will download the image and then flash it all in one go.  Not sure how it handles checksum verification in that case (or if it does at all).

 

I had never heard of it before today, and have not used it yet.  Heard about it while reading install instructions for PinePhone at Mobian wiki.  I am thinking I might try it out to burn Mobian to an SD card here shortly.  Apparently it's available in Debian as package bmap-tools.

 

EDIT:  I am not sure if this is available on Windows or not.  I forget that was one of the original criteria, as this is irrelevant for me, personally.  But I know the topic is about something which can be recommended for Armbian users overall.

 

EDIT2: OK, I found what I was looking for about checksums a little further down (under "Usage scenarios" section):

 

Quote

The additional benefit of using bmap for flashing is the checksum verification.
Indeed, the "bmaptool create" command generates SHA256 checksums for all mapped
block ranges, and the "bmaptool copy" command verifies the checksums while
writing. Integrity of the bmap file itself is also protected by a SHA256
checksum and bmaptool verifies it before starting flashing.

On top of this, the bmap file can be signed using OpenPGP (gpg) and bmaptool
automatically verifies the signature if it is present. This allows for
verifying the bmap file integrity and authoring. And since the bmap file
contains SHA256 checksums for all the mapped image data, the bmap file
signature verification should be enough to guarantee integrity and authoring of
the image file.

 

So, there is no direct checksum verification of the downloaded file (AFAICT), however their special bmap file format (which is some XML) includes checksums of the actual original contents, which are checked against the final written result.

Edited March 20 by TRS-80
second edit

[Tido 187]

On 3/20/2021 at 5:47 PM, TRS-80 said:

if this is available on Windows or not. 

As it is a Python SW I guess it can run everywhere. USBimager is written in C and runs everywhere, has a nice PDF manual in the repo. I am happy with USBimager