So I figured it out (and wasted 5 hours or so ) ...
Support for bridge flags for the mv88e6xxx chips was introduced here: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/drivers/net/dsa/mv88e6xxx?h=v5.1&id=4f85901f0063e6f435125f8eb54d12e3108ab064
One of these flags is flooding. Which seems to be enabled by default, and leads to incoming packets on port A to be replayed on all other ports. This naturally lowers the receive speed on port A to the minimum transmit speed of all other ports.
Easy way to fix this is to issue 'bridge link set dev lan4 flood off' for all the lan ports on the clearfog.
I confirmed this also works on Lk5.8
Does it make sense to add a family tweak which does this for all lan ports? Or is this possible via device tree?
EDIT: Nevermind, this is not an issue with the dsa but with the default settings of a linux bridge. Let's not touch it, because not everyone will put the lan ports on the clearfog in a bridge. If they do, they hopefully stumble upon this and check the man page: https://www.man7.org/linux/man-pages/man8/bridge.8.html
Heisath's post in After 1st shutdown, can't get past "Enter Auth Username"/"Enter Auth Password" was marked as the answer