Alternative setup

[DiegoBM]

I've a set of Orange Pi 5 Max boards, all of them mounting fast NVME units. My plan was to use micro SD card to host the OS, but delegate all intensive writing (log, tmp) to a special partition on the NVME unit which will be used for OS and Application data (both for performance, and to extend the lifetime of the SD card). That way I can flash the OS disk as needed without worrying about flashing the data disk with it.

 

I've seen that Armbian uses zram by default, but I'd like to keep the entire RAM available for the applications that will be running in the boards (NVME should be fast enough to not notice a massive impact vs using the RAM). As I have seen disabling it is not a problem. I've also seen that Armbian can use zswap but I presume that zswap also uses memory before writing back to the designated storage unit, so it would be a similar situation.

 

I reckon that the best solution for me would be to have the OS in the SD card but make it search the for the entire "/var" folder (that should include log, tmp, and swap. Am I missing any other intensive writes?) into the NVME device. So my question is: is there any way to configure Armbian after flashing, and before the first boot, so that it automatically goes to the NVME device (which should be available) for "/var"? 

 

If a good article exists on the topic, could you please point me in its direction? Thank you in advance!

Edited Saturday at 04:08 PM by DiegoBM

[IBV]

Hi,

 

theoretically if you knew the UUID of the partition on the NVME disk that you want to use as /var partition, you could do the following:

- after flashing the SD card with the Armbian image, put it in a reader (or a usb card reader) connected to  a Linux PC.

- the / partition on the SD card should be auto mounted

- you can go and then edit the etc/fstab file on the SD card and add a line like this:

UUID=<your UUID goes here> /var           ext4    defaults        0       2

put the UUID of the partition you want to use as /var in the above line.

- boot your SD card

 

I am not sure how this will work with the ram logging, you should try it yourself.

 

If it does not work I can show you how to move the /var partition on the NVME disk after Armbian boots the first time (I have done this myself and it works). This involves

- ram log disabling

- reboot in rescue mode

- identify the UUID of your NVME partition for new /var

- move the contents of the /var/ to the new /var

- modify the /etc/fstab to mount the new /var on boot.

[DiegoBM]

Thank you for your reply @IBV!

 

Can I not use the device name (something along the lines of "/dev/nvme0n1p1") which I know upfront, and will always be the same, since I have prepared all the NVME devices? That will be known before boot, although initially the NVME won't have any "/var" contents on it (can those be extracted from the image before boot?). Also how can you edit the /etc/fstab file on the sd card? I'm working on Windows where I can't see the sd card contents after flashing. Would it mount automatically if I used Linux?

 

It might be more reasonable to follow your method, since I will also need to disable ram log and copy the "/var" contents. Maybe I could follow your process with the first card. Create an image out of the process, capture the /var contents inside nvme. And reutilize that with all the other cards?

 

Could you please show me how to complete your process or indicate me where could I read more about it? Thanks in advance!

Edited Sunday at 02:23 PM by DiegoBM

[IBV]

47 minutes ago, DiegoBM said:

Can I not use the device name (something along the lines of "/dev/nvme0n1p1") which I know upfront, and will always be the same, since I have prepared all the NVME devices?

Yes you could. I gave the UUID example out of habit, I do this on VM's where the disks are /dev/sdX and I want to make sure I get exactly the partition I want on boot in case the kernel/udev is messing the device ordering. In your case /dev/nmve0n1p1 should be fine for example.

50 minutes ago, DiegoBM said:

That will be known before boot, although initially the NVME won't have any "/var" contents on it (can those be extracted from the image before boot?)

You are right, you will need to move anyway the old /var contents to the new partition. In case you are able to mount the SD card on a Linux PC/VM, you could do that too before first boot. But maybe better to follow the procedure I describe later.

52 minutes ago, DiegoBM said:

I'm working on Windows where I can't see the sd card contents after flashing.

Probably the easiest would be to use a Linux VM (eg. VirtualBox). You can then use a usb SD card reader and pass it through to the VM.

54 minutes ago, DiegoBM said:

Would it mount automatically if I used Linux?

My distribution does it (Ubuntu). 

55 minutes ago, DiegoBM said:

Could you please show me how to complete your process

Yes.

After flash the Armbian image, boot it. Allow it to reboot and expand the filesystem. Then:

- disable ram log (by editing the /etc/default/armbian-ramlog and setting ENABLED=false) then reboot

- make sure you have the root password (you can set one with "sudo passwd" if you don't have it)

- reboot in rescue mode (use the root password when prompted)

sudo init 1

- mount the NVME partition you want to use as /var in /mnt. FYI: This step is to move the existing /var data to the new location. You need rescue mode so no services run and write stuff in /var when you do the move. Examples below assume that the partition you want to use is /dev/nvme0n1p1.

mount /dev/nvme0n1p1 /mnt

- move the data

mv /var/* /mnt/

- edit /etc/fstab and add a line with the /var mount. 

/dev/nvme0n1p1 /var           ext4    defaults        0       2

If you want to use UUID's you can run "blkid" command and you will get the UUID's of all disks/partitions. You can then take the one of the nvme0n1p1 and use it with UUID=<your uuid> like I showed you in the first answer.

- Reboot

 

[DiegoBM]

@IBV I can't thank you enough for taking the time to put together such an elaborate response! You're a rockstar!

 

I'll make sure to try actually try both options over the week, and I will come back to report how it went.

 

Once again, thank you!

[DiegoBM]

Unfortunately it seems like I won't be able to test the first method since, no matter how I write the image into the sd card, Ubuntu is not able to mount it when using an SD card reader. It might seem like there's something wrong with the partition information in the sd card according to gparted. Maybe it's something specific of the orange pi 5 max image....?

[IBV]

Are you sure you properly prepared the SD card?

 

https://docs.armbian.com/User-Guide_Getting-Started/#preparing-sd-card

 

I just did a test write of the Orange Pi 5 max image, using dd on a usb stick. Granted, I used the Debian Bookworm minimal image.

 

https://dl.armbian.com/orangepi5-max/Bookworm_current_minimal

 

When I inserted the usb stick in my Ubuntu PC, it was automatically mounted.

Can you post the output of "dmesg" command after you insert the SD card?

[DiegoBM]

I did follow the instructions except for the part where they recommend using USBImager, since it won't work on Windows 11 (it won't display the USB devices)

 

Absolutely, here's the output:

 

[Mar24 12:46] usb 1-4: new high-speed USB device number 14 using xhci_hcd
[  +0.131864] usb 1-4: New USB device found, idVendor=05e3, idProduct=0751, bcdDevice=14.02
[  +0.000007] usb 1-4: New USB device strings: Mfr=3, Product=4, SerialNumber=0
[  +0.000002] usb 1-4: Product: USB Storage
[  +0.000002] usb 1-4: Manufacturer: USB Storage
[  +0.001897] usb-storage 1-4:1.0: USB Mass Storage device detected
[  +0.000495] scsi host3: usb-storage 1-4:1.0
[  +1.027333] scsi 3:0:0:0: Direct-Access     Generic  STORAGE DEVICE   1402 PQ: 0 ANSI: 6
[  +0.000552] sd 3:0:0:0: Attached scsi generic sg1 type 0
[  +0.181714] sd 3:0:0:0: [sdb] 31116288 512-byte logical blocks: (15.9 GB/14.8 GiB)
[  +0.001269] sd 3:0:0:0: [sdb] Write Protect is off
[  +0.000007] sd 3:0:0:0: [sdb] Mode Sense: 21 00 00 00
[  +0.001264] sd 3:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[  +0.043992] GPT:Primary header thinks Alt. header is not at the end of the disk.
[  +0.000005] GPT:3571711 != 31116287
[  +0.000002] GPT:Alternate GPT header not at the end of the disk.
[  +0.000002] GPT:3571711 != 31116287
[  +0.000002] GPT: Use GNU Parted to correct GPT errors.
[  +0.000009]  sdb: sdb1
[  +0.000209] sd 3:0:0:0: [sdb] Attached SCSI removable disk

 

Edited yesterday at 11:48 AM by DiegoBM
added extra information

[IBV]

OK, so it did not auto mount.

Can you try to mount it manually ?

 

sudo mount /dev/sdb1 /mnt

 

If it works, you should see the contents in /mnt

[IBV]

First method: even if you manage to mount the partition, you will be able to edit the /etc/fstab file, but moving the data in /var partition will be more difficult. You would need also the NVME disk available to the Ubuntu machine, then you will have to mount the new NVME partition and move the data.

 

Maybe is simpler to boot the SD card and follow the steps I described as the 2 method.

[DiegoBM]

I reckon I know what was I doing wrong. I had decompressed the ".img" file from within the ".img.xz" file and that's what I was trying to flash. I don't know why I assumed that the .xz file was just some form of zip and extracting the image would speed things up for multiple flashing, but I guess that's not the case and that there must be more going on with the.xz file.

 

You're right that will probably be the most sensible choice and the one that I'll finally use. I will give the first option a try just for the sake of checking how wrongly it can turn. Having the NVME available won't be an issue, I have a NVMe-to-USB adapter that I was using anyway to encrypt one of the partitions within the NVME

Edited yesterday at 12:17 PM by DiegoBM