I think there are some major misunderstanding here about the USB-C power supply.
1. The problem about the under voltage at the input of the board are the thin wires of the USB cable...
Do not use USB cable to power the M4 board, just use a descent barrel Jack DC power supply (for example: Meanwell)
2. Do we have to power the board over the pin header?
No, not necessary.
The USB-C connector is rated for 5A, physically! Not 3A. The 3A limitation is a question of wiring limitation of the standard USB-C cable, not the connector itself (and hypothetically the PCB traces of the board)
The missing support for USB-PD isn't an issue either. Don't use USB-C power supplies, it's that simple as that. Again, just use a descent barrel Jack DC power supply! And if possible a 5.1V or 5.2V one.
You only need a cheap USB-C DC jack adapter (which causes also probably a weak voltage drop):
Now, about the USB 3.0 connectors. Well indeed, at load there is a voltage drop on the board, especially on heavy load, and they are partially caused by the mosfets on the power rail on the board I think.
VDD_5V goes through the AO3415A mosfet to power the board. It has a 42mOhm Rds(on). which caused alone a 0.1V drop at full load of the M4 board (2A).
The USB 3.0 port uses a RT9724GQW load switch which has a Rds(on) of 100mOhm. That causes another 0.1V loss at 1A load.