After further debugging, the DTS from above is not enough. Crosschecking between Armbian's banana-pi-m5.dtb and the one provided by Sinovo's Armbian, I noticed the following change in "serial@24000" node.
- phandle = <0x128>;
+ pinctrl-0 = <0x27 0x28>;
+ pinctrl-names = "default";
+ uart-has-rtscts;
+ phandle = <0x12a>;
I'm not an expert in DTS definitions, but as far as I understand, the phandle is a reference. The reference doesn't seem to be used anywhere in the main DTS, so I haven't changed it.
Instead I've modified the overlay to add the other changes:
/dts-v1/;
/ {
compatible = "amlogic,sm1";
fragment@0 {
target-path = "/aliases";
__overlay__ {
serial1 = "/soc/bus@ffd00000/serial@24000";
};
};
fragment@1 {
target-path = "/soc/bus@ffd00000/serial@24000";
__overlay__ {
pinctrl-0 = <0x27 0x28>;
pinctrl-names = "default";
uart-has-rtscts;
status = "okay";
};
};
};
I confirm that after these changes, UART finally works. Hope this helps anyone in search of a solution. I hope the Armbian maintainers can address these problems related to the Banana Pi M5 board.
EDIT:
After reverting the previous DTS, as I pasted in the previous post and doing a full power cycle, the UART works still. So it's likely that I didn't power cycle the board properly. I just kept pressing the reset button, but that doesn't seem to be enough apparently?