Creating an img from sd card

[bedna]

Hi!

 

Have limited knowledge in linux, all self learned so bear with me if I sometimes sound like an i**ot in my wording.

 

I have used lots of different Raspberry pi's but since they are near to impossible to get your hands on I figured I'd try out some of the alternate boards.

 

Got myself a nanoPiNeo2, orangePiZero2 and an orangePiPC2 just for the fun of it. I have tried this on the nanoPi and the opiZero2 and run into the same problems, havent even opened the opiPC2 yet but I suspect it will be the same problem on that board.

 

Downloaded armbian for each separate board (instead of debian) because I want to use the same system on all my devices so its easier for me to learn.

Got everything working as I wanted (just making a simple pihole to give away as xmas presents) but when it came to backing up, I ran into a wall.

 

With the raspberries I use a script called image-backup located on the raspberrypi forums (awesome script made by RonR) but that obv wont work here since the rpi uses a fat32 as a boot partition.

There is a flag you can run when using the script called "ubuntu", but it didn't work.

 

Ok then, lets use win32 imager instead to just rip the card, its only 8gb, not a huge deal, I can probably even shrink the size with e2fsck and parted (as I understand it armbian autoexpands the card on mount anyway?) but I get an i/o error in win32 after a few seconds, this might actually be my computer, going to try it on another computer in a day or two, (I only have one). I tried other programs too, but none seems to work so I decided to try to dd from the mounted card. The backup directory is on a mounted cif on a windows computer if that has to do with anything?

 

sudo dd if=/dev/mmcblk0 of=/path/to/backup/ddbackup.img bs=4M conv=sync status=progress

 

That didn't work, I get (the numbers are bigger I just randomly took a number, the important thing is the +1 that gets added to the "normal" number in "records out", I suspect that is the boot sector?)

 

15874+1 records in
15875+0 records out

 

I figured it had to do with the card being mounted so I made ANOTHER identical install on another sdcard and inserted the original sd in the usb with a card reader and doing the dd from /dev/sda instead (without it being mounted), but I get the exact same thing and when trying to flash the image and use that card, the board just never comes to life. Network port keeps flashing but nothing else.

 

My plan was to make a backup.img file, mount that with a loop and shrink the "partition" with shrink2fs and parted. But even trying to mount the created image files gives me errors about bad superblocks etc, probably because of what I described above.

Running e2fsck also gives me a TON of errors.

 

I have tried for DAYS, searching all over the internet (figuring things out for myself is the best way to learn) but I get nowhere. Is there a simple solution to this?

 

The only time I actually got a working .img file is if I dd a blank img file first from /dev/zero and set the file to become a size (with bs and count) where all data can fit and then rsync / into it. I can mount it with a loop (losetup) and access the files, but naturally it wont boot on the board (if I flash it to an sd card) since I read that the boot sector is "out of bonds" or whatever its called. Same error, just flashing network port.

I'm at a complete loss here, I REALLY want to have an img file that the xmas gift holders get that they can just flash onto an sdcard in case of a breakdown of the software somehow.

 

Please have pity on my soul!

[denni_isl]

   

Use at your own risk and practice with less important drive (of course).

    ----------------- copy hard drive to .img(.gz) and .img(.gz) hard drive ----

 

* boot your computer up with a live usb or sd wit gparted > unmount the disk you want to copy >

 gparted > make the last partition smaller but leave some small additional space > check filesystem

 

* to copy used partitions  over to your_hard_drive_copy .img.gz ;
  - calculate >  count= (gparted last sector of the alst partition (/)  right mouse button, information, last sector  * 512 / bs=4096  >

 

* calculate in console;

expr 22171647 \* 512 / 4096 = 2771455   -(2771456)) (+1 sector).  - last sector +1 * 512 / 4096 = last sector in bs=4096

 

*  copy used partitions  /dev/sdc á .img og .img.gz;  

               sudo dd if=/dev/sdc bs=4096 count=2771456 conv=sync,noerror status=progress of=manjaro_vim3_38okt20.img && gzip manjaro_khad_aml_mini.img

 

* .img over to /dev/sda;

             dd if=hard_drive_backup.img of=/dev/sda bs=1m status=progress ; sync,
 

* when restored one need to boot again with a live usb or cd with gparted and umount the disk > resize >  check filesystem

    ------------------------------------------------------

Edited November 4, 2022 by denni_isl

[bedna]

I think the reason it didn't work with my first tries I suspect is because I: 1. tried to mount the whole disk, not the partition. 2. Did not probe the "disk".

 

This is the solution I found to best working for me.

 

Backing up and mounting image:

 

Use gnu version of ddrescue (you can back up mounted system drive with it)

$ sudo apt install gddrescue -y
$ sudo ddrescue -f -n /dev/mmcblk0 /path/to/backup.img

GNU ddrescue 1.23
Press Ctrl-C to interrupt
     ipos:   15634 MB, non-trimmed:        0 B,  current rate:  14352 kB/s
     opos:   15634 MB, non-scraped:        0 B,  average rate:  19066 kB/s
non-tried:        0 B,  bad-sector:        0 B,    error rate:       0 B/s
  rescued:   15634 MB,   bad areas:        0,        run time:     13m 39s
pct rescued:  100.00%, read errors:        0,  remaining time:         n/a
                              time since last successful read:         n/a
Finished

Find out where the boot sector/partition ends/starts, ie: start(32768) x blocksize(512) = offset(16777216), and mount it (make sure you have directories to mount to, in this case sudo mkdir /mnt/loop)

You can also divide the offset by 8 to make sure it's in the right position, should end up with a whole number, in this case 16777216/8=2 097 152. This tells me, even though the bootsector seems to be way bigger than needed, it probably will work.

$ sudo fdisk -l /mnt/backup/nanoPiNeo3/piHole/backup.img
Disk /path/to/backup.img: 14.56 GiB, 15634268160 bytes, 30535680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xb90578af

Device              Boot  Start End      Sectors  Size  Id Type
/path/to/backup.img1      32768 30212096 30179329 14.4G 83 Linux

 

Mount the img with a loop

$ sudo mount -o loop,offset=16777216 /path/to/backup.img /mnt/loop
$ ls /mnt/loop -l
total 84
-rw-r--r--   1 root root  3128 Oct 22 18:11 armbian.key
lrwxrwxrwx   1 root root     7 Oct 22 08:41 bin -> usr/bin
drwxr-xr-x   3 root root  4096 Nov  1 23:50 boot
drwxr-xr-x   2 root root  4096 Oct 22 18:11 dev
drwxr-xr-x 103 root root  4096 Nov  4 19:19 etc
drwxr-xr-x   3 root root  4096 Nov  1 16:34 home
lrwxrwxrwx   1 root root     7 Oct 22 08:41 lib -> usr/lib
drwx------   2 root root 16384 Oct 22 18:19 lost+found
drwxr-xr-x   2 root root  4096 Oct 22 08:41 media
drwxr-xr-x   5 root root  4096 Nov  4 19:19 mnt
drwxr-xr-x   3 root root  4096 Nov  1 17:26 opt
drwxr-xr-x   2 root root  4096 Oct 22 08:41 proc
drwx------   5 root root  4096 Nov  4 19:18 root
drwxr-xr-x   3 root root  4096 Oct 22 18:21 run
lrwxrwxrwx   1 root root     8 Oct 22 08:41 sbin -> usr/sbin
drwxrwxr-x   2 root root  4096 Oct 22 18:11 selinux
drwxr-xr-x   2 root root  4096 Oct 22 08:41 srv
drwxr-xr-x   2 root root  4096 Sep  3 14:10 sys
drwxrwxrwt   2 root root  4096 Oct 22 18:11 tmp
drwxr-xr-x  11 root root  4096 Oct 22 08:41 usr
drwxr-xr-x  13 root root  4096 Nov  1 17:23 var

 

Now you can rsync your whole disk to keep files up to date.

First create a file with paths to exclude, ie so you don't create a loop and backs the directory you are backing up to, I simply created exclude.txt in my home directory.

nano ~/exclude.txt

 

Content of my exclude.txt

#directories to exclude when backing up with rsync
/proc/*
/sys/*
/dev/*
/tmp/*
/run/*
/mnt/*
/media/*

 

rsync to the mounted drive to update backup

sudo rsync -ahv --info=progress2 --stats --delete --force --partial exclude-from=/home/$USER/exclude.txt --delete-excluded / /mnt/loop/

 

umount /mnt/loop so we can shrink the img file

sudo umount /mnt/loop

 

 

Shrinking the img file

 

Change to superuser (sudo su) so you don't have to type sudo before each line and run following commands. I will use parted.

$ sudo su
$ modprobe loop
$ losetup -f
$ losetup /dev/loop0 /path/to/backup.img
$ partprobe /dev/loop0

 

Check how much space is actually used, since it's a copy of my system I can just df that

$ df -h /dev/mmcblk0p1
Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p1   15G  1.7G   13G  12% /

 

Lets make the drive 2500MB, just to be a bit safer and leave space for future rscync backups in case they size increases of what you want to back up

$ e2fsck -f /dev/loop0p1
e2fsck 1.46.2 (28-Feb-2021)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
armbi_root: 49784/916864 files (0.1% non-contiguous), 502310/3772416 blocks

$ resize2fs /dev/loop0p1 2500M
resize2fs 1.46.2 (28-Feb-2021)
resize2fs: New size smaller than minimum (659029)

 

Nope, it didn't like that, lets try with 2600M

$ resize2fs /dev/loop0p1 2600M
resize2fs 1.46.2 (28-Feb-2021)
Resizing the filesystem on /dev/loop0p1 to 665600 (4k) blocks.
The filesystem on /dev/loop0p1 is now 665600 (4k) blocks long.

 

Nice, it worked. Then recheck it

e2fsck -f /dev/loop0p1
e2fsck 1.46.2 (28-Feb-2021)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
armbi_root: 49784/165984 files (0.2% non-contiguous), 454731/665600 blocks

 

Looking good.
Now lets shrink the partition using parted. Since I'm not sure how to get all units in the same way I'd rather make the partition a little bigger than the filesystem and then expand the filesystem it up to the partition size, ie lets make it 2700MB. If someone could clarify this part I would be very happy

$parted /dev/loop0

GNU Parted 3.4
Using /dev/loop0
Welcome to GNU Parted! Type 'help' to view a list of commands.

(parted) print
Model: Loopback device (loopback)
Disk /dev/loop0: 15.6GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      16.8MB  15.5GB  15.5GB  primary  ext4

(parted) resizepart 1
End?  [15.5GB]? 2700MB
Warning: Shrinking a partition can cause data loss, are you sure you want to continue?
Yes/No? y

(parted) quit
Information: You may need to update /etc/fstab.

 

Then lets resize the filesystem with resize2fs

$ resize2fs /dev/loop0p1
resize2fs 1.46.2 (28-Feb-2021)
Resizing the filesystem on /dev/loop0p1 to 655083 (4k) blocks.
The filesystem on /dev/loop0p1 is now 655083 (4k) blocks long.

 
And one last check before shrinking the img file

$ e2fsck -f /dev/loop0p1
e2fsck 1.46.2 (28-Feb-2021)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
armbi_root: 49784/158080 files (0.2% non-contiguous), 454235/655083 blocks

 

Looking good. Now let's see how much we can shave off the file, we are looking for the end sector (5273437) and sector size (512), we use fdisk

$ fdisk -l /dev/loop0
Disk /dev/loop0: 14.56 GiB, 15634268160 bytes, 30535680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x83f3ee4c

Device       Boot Start     End Sectors  Size Id Type
/dev/loop0p1      32768 5273437 5240670  2.5G 83 Linux

 

The calculation goes: end sector+1 and then multiply that number with the sector size (512), that gives us 5273438 x 512 = 2 700 000 256bytes, about 2,7GB

We use truncate to trim the file

$ truncate --size=$[(5273437+1)*512] /path/to/backup.img

 

Lets mount the img file again and see if we can access the files

$ mount /dev/loop0p1 /mnt/loop
$ ls /mnt/loop -l
total 84
-rw-r--r--   1 root root  3128 Oct 22 18:11 armbian.key
lrwxrwxrwx   1 root root     7 Oct 22 08:41 bin -> usr/bin
drwxr-xr-x   3 root root  4096 Nov  1 23:50 boot
drwxr-xr-x   2 root root  4096 Nov  1 23:50 dev
drwxr-xr-x 103 root root  4096 Nov  4 19:19 etc
drwxr-xr-x   3 root root  4096 Nov  1 16:34 home
lrwxrwxrwx   1 root root     7 Oct 22 08:41 lib -> usr/lib
drwx------   2 root root 16384 Oct 22 18:19 lost+found
drwxr-xr-x   2 root root  4096 Oct 22 08:41 media
drwxr-xr-x   2 root root  4096 Nov  4 19:19 mnt
drwxr-xr-x   3 root root  4096 Nov  1 17:26 opt
dr-xr-xr-x   2 root root  4096 Jan 21  2016 proc
drwx------   5 root root  4096 Nov  4 19:18 root
drwxr-xr-x   2 root root  4096 Nov  4 19:28 run
lrwxrwxrwx   1 root root     8 Oct 22 08:41 sbin -> usr/sbin
drwxrwxr-x   2 root root  4096 Oct 22 18:11 selinux
drwxr-xr-x   2 root root  4096 Oct 22 08:41 srv
dr-xr-xr-x   2 root root  4096 Jan 21  2016 sys
drwxrwxrwt   2 root root  4096 Nov  4 20:09 tmp
drwxr-xr-x  11 root root  4096 Oct 22 08:41 usr
drwxr-xr-x  13 root root  4096 Nov  1 17:23 var

 

We can also see that the file is WAY smaller than the 15GB file we started out with.

$ df -h /mnt/loop/
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0p1    2.5G  1.7G  740M  70% /mnt/loop

 

If you want to, you can once again rsync into the img file to back up your system. Just make sure that you don't install massive amounts so the file gets filled, the file will not get bigger by itself. You have to redo these steps here but START with increasing in parted THEN resize it with e2fsck.

 

Finally let's umount and remove the loop

$ umount /mnt/loop
$ losetup -d /dev/loop0

 

Don't forget to exit SU by simply typing

$ exit

 

You can now flash this img file to an sdcard using win32DiskImager, etcher or whatever your fav prog is. If you want it to expand the partition after you have flashed it to a new card, simply boot up with new card in board, enable armbian-resize-filesystem.service then reboot and disable it again. Your sd card will be max size.

$ sudo systemctrl enable armbian-resize-filesystem.service
$ sudo reboot now
$ sudo systemctrl disable armbian-resize-filesystem.service

 

If someone knows if it's possible to put a file on the sdcard that tells it to autoexpand on boot I would love to know about it. That would remove the whole reboot process when expanding the card.

 

Hope this wall of text can help someone else.

Edited November 5, 2022 by bedna