piknew Posted October 7, 2016 Author Share Posted October 7, 2016 If you have it on SD then put card in SD Card reader and make an backup (by for example Win32DiskImager). If you have it on emmc then you may use script from this thread. Of course you must boot your device from SD card (it can be fresh Armbian, with pv tool installed and enabled authorization by public key). BTW. I think I will adjust my script to support also "local" image storage (not only by SSH). "local" == locally mounted file system. Link to comment Share on other sites More sharing options...
tkaiser Posted October 7, 2016 Share Posted October 7, 2016 If you have it on emmc then you may use script from this thread. Of course you must boot your device from SD card BTW: http://forum.armbian.com/index.php/topic/2033-question-h3-unified-imaging/ (Start reading from post #8 -- eMMC can be mounted as USB thumb drive. On a related note: WinDiskImager does no verify so it's a bad tool. Fortunately the Etcher folks want to add backup/clone functionality to their tool in the future) Link to comment Share on other sites More sharing options...
earth08 Posted October 16, 2016 Share Posted October 16, 2016 How to backup my whole os installed on emmc So when ever I need a fresh install I use the backup Link to comment Share on other sites More sharing options...
piknew Posted October 16, 2016 Author Share Posted October 16, 2016 How to backup my whole os installed on emmc So when ever I need a fresh install I use the backup I answered you already, but you have asked the same question again... If something is not clear you have to point "what" exactly should be elaborated. Link to comment Share on other sites More sharing options...
earth08 Posted October 19, 2016 Share Posted October 19, 2016 I answered you already, but you have asked the same question again... If something is not clear you have to point "what" exactly should be elaborated.I am not clear, thereI am on emmc and I would like to back it on my sdcard/pendrive and than copy it to my pc. Please guide. Thanks in adv. Link to comment Share on other sites More sharing options...
tkaiser Posted December 19, 2016 Share Posted December 19, 2016 Addendum: New tools that look promising: http://www.cnx-software.com/2016/12/16/compress-decompress-files-faster-with-lbzip2-multi-threaded-version-of-bzip2/ http://www.cnx-software.com/2016/12/19/facebook-zstandard-zstd-pzstd-data-compression-tools-deliver-high-performance-efficiency/ Use case: A eMMC equipped SBC that boots from SD card to make an eMMC backup (clone) to a remote location. The 8GB eMMC is a rather fresh Armbian installation only using 15% of the eMMC's capacity: /dev/mmcblk1p1 7.1G 1012M 6.0G 15% / On the remote host we start 'mbuffer -I 9090 >sbc.img.zst'. On the H3 device we now do: root@bananapim2plus:~# dd if=/dev/mmcblk1 | mbuffer | zstd - -c | mbuffer -O 192.168.83.146:9090 in @ 0.0 KiB/s, out @ 0.0 KiB/s, 871 MiB total, buffer 0% full15269888+0 records in 15269888+0 records out 7818182656 bytes (7.8 GB, 7.3 GiB) copied, 221.039 s, 35.4 MB/s summary: 7456 MiByte in 3min 41.0sec - average of 33.7 MiB/s summary: 871 MiByte in 3min 41.0sec - average of 4037 KiB/s The whole stuff was both bottlenecked by CPU and I/O (tested on BPi M2+ that uses a pretty slow eMMC). On boards with faster eMMC (every Orange Pi for example) things would be faster. On average we read from eMMC with 35.4 MB/s. After piping through zstd on the other host just 871MB arrive as a result of the compression (which means we had a network throughput of 4037 KiB/s) and the whole process took 3:41 minutes. To optimize this we could simply split the whole /dev/mmcblk1 in 500 MB chunks and let two of them being processed in parallel. One process starts from beginning of the device, the other from the end. Please remember: As long as the zstd command is fed with real data CPU is the bottleneck and as soon as the 'empty' parts are processed reading from eMMC becomes the bottleneck. With the aforementioned approach it should be ensured that as long as possible one zstd process is fed with real data and the other running in parallel processing 'empty space'. So on a fast eMMC (read as: not on BPi M2+ or the other newer SinoVoip devices that chose the cheapest Samsung eMMC available) this parallelism should speed up things further. For more details (eg. how to 'restore' such a bunch of .zst file to an SD card for example) please read through the first linked comments thread above. Link to comment Share on other sites More sharing options...
tkaiser Posted December 19, 2016 Share Posted December 19, 2016 And now the fun part! My first try was to make a device backup from an eMMC with a fresh Armbian installation on it. But the eMMC itself was quite often in use before (I use the BPi M2+ only for tests in the meantime). So it doesn't really matter how 'full' the eMMC seems to be when looking at it from the filesystem perspective since due to the prior usage the unused space isn't just zeros (that compress pretty good) but on the eMMC is data from the former installations that compresses not that great since 'real data'. How to improve that? Zero out all the unused space first. So I mounted the ext4 fs on the eMMC and let simply all 'free space' overwrite with zeroes: root@bananapim2plus:~# dd if=/dev/zero of=zeroes bs=10M || rm zeroes dd: error writing 'zeroes': No space left on device 617+0 records in 616+0 records out 6468685824 bytes (6.5 GB, 6.0 GiB) copied, 1077.24 s, 6.0 MB/s OMFG, really just 6.0 MB/s write speed. Did I already mention that SinoVoip uses really cheap/slow eMMC on their devices? (to be honest: as long as you write just a few MB this doesn't matter but such an excercise as above where you want to write +6GB takes some time on such an ultra slow eMMC -- every good SD card is almost four times faster here!) Now let's repeat this: root@bananapim2plus:~# dd if=/dev/mmcblk1 | mbuffer | zstd - -c | mbuffer -O 192.168.83.146:9090 in @ 8.0 KiB/s, out @ 8.0 KiB/s, 317 MiB total, buffer 0% full15269888+0 records in 15269888+0 records out 7818182656 bytes (7.8 GB, 7.3 GiB) copied, 207.588 s, 37.7 MB/s summary: 7456 MiByte in 3min 27.5sec - average of 35.9 MiB/s summary: 317 MiByte in 3min 27.5sec - average of 1563 KiB/s Processing time got only slightly better but we have huge savings in disk space and transmittion: Before we had to transfer 871 MB compressed data but now it's just 317 MB any more. Since read speeds with this cheap eMMC are limited to ~40 MB/s we got only a slight increase of average read speed (on Orange Pi's it would be twice as much for example). Lessons to learn: In case you want to do device backups/clones at least once zero out every unneeded space. This will greatly reduce disk space and network bandwidth needed since zeroes compress pretty good compared to -- existing but unused -- real data patterns (TODO: Have a look whether eMMC implementation support fstrim) Know your bottlenecks and act accordingly. In our case that means setting up some parallelisms to process different data patterns in parallel After I zeroed out the empty space the following assumption is both easy and true when using a fresh Armbian installation: If the FS in question is filled with 1012 MB according to df then the first 1012 1MB blocks contain data and the all blocks behind only zeroes. Let's take a look. I started to receiving mbuffer instances on different ports and then process both parts individually: The data part (1012 MB before compression become 289 MB after) finishes pretty fast: root@bananapim2plus:~# dd if=/dev/mmcblk1 bs=1M count=1012 | mbuffer | zstd - -c | mbuffer -O 192.168.83.146:9089 in @ 4417 KiB/s, out @ 4417 KiB/s, 289 MiB total, buffer 0% full1012+0 records in 1012+0 records out 1061158912 bytes (1.1 GB, 1012 MiB) copied, 71.4225 s, 14.9 MB/s summary: 1012 MiByte in 1min 11.4sec - average of 14.2 MiB/s summary: 289 MiByte in 1min 11.4sec - average of 4146 KiB/s Now the 'empty space' (6444 MB become just 27.6 MB after compression and the whole process is mainly bottlenecked by slow eMMC): root@bananapim2plus:~# dd if=/dev/mmcblk1 bs=1M skip=1012 | mbuffer | zstd - -c | mbuffer -O 192.168.83.146:9090 in @ 40.3 MiB/s, out @ 40.2 MiB/s, 6433 MiB total, buffer 0% full6444+0 records in 6444+0 records out 6757023744 bytes (6.8 GB, 6.3 GiB) copied, 179.819 s, 37.6 MB/s summary: 6444 MiByte in 2min 59.8sec - average of 35.8 MiB/s summary: 27.6 MiByte in 2min 59.8sec - average of 157 KiB/s Link to comment Share on other sites More sharing options...
tkaiser Posted December 19, 2016 Share Posted December 19, 2016 Last update: By increasing compression level for 'normal data' (zstd defaults to 3 and now I used '-5') the time to process increased from 1:11 minutes to 3:03 (therefore as long as processing the zeroes needs) but we end up with a smaller backup image now: root@bananapim2plus:~# dd if=/dev/mmcblk0 bs=1M count=1012 | mbuffer | zstd -5 - -c | mbuffer -O 192.168.83.146:9089 in @ 3826 KiB/s, out @ 3826 KiB/s, 254 MiB total, buffer 0% full1012+0 records in 1012+0 records out 1061158912 bytes (1.1 GB, 1012 MiB) copied, 183.417 s, 5.8 MB/s summary: 1012 MiByte in 3min 03.4sec - average of 5651 KiB/s summary: 255 MiByte in 3min 03.4sec - average of 1422 KiB/s So compared to an archive that needs 871 MB and took 3:41 min to finish we are now at 283 MB archive size and ~3 minutes processing time. If this board would feature faster eMMC and by using more parallelization processing time could further improve. Link to comment Share on other sites More sharing options...
Recommended Posts