In the end the fix seemed to be fairly simple.
The function fusb302_extcon_register that was added by this patch was just called to early and was checking for usb_ss_present and dp_present to advertise the capabilities, but at that point those values where not initialized yet. So the extcon device did not have any capabilities resulting in -EPERM / "get superspeed property failed".
The following patch moves the initialization until after the values are initialized.
This now successfully works with a usb-c to DisplayPort cable. Although it seems to work only with some monitors. My Dell U2414H did not work, while it worked with LG 27GL850.
The Dell would cause this:
However that sounds like its now a displayport problem, no longer a usb alt mode problem.
Getting my type-c monitor with touch interface to work took some more work. This monitor prefers to connect in a usb+DP mode. And the code in drivers/usb/typec/altmodes/displayport.c could not get a matching pin configuration between port and device.
The display returned a vdo of 0xC0045 for the displayport alt mode. According to this 3rd party description https://www.st.com/resource/en/technical_article/ta0356-usb-typec-and-power-delivery-displayport-alternate-mode-stmicroelectronics.pdf this essentially means:
Type-C receptacle with pin modes C and D supported as a UFP_D device.
The port was registered with a vdo of 0x1C1C46
Type-C receptacle with pin modes C, D and E supported as a DFP_D device.
The current code of displayport.c was not able to make a match between these even though they are compatible.
It seemed to not correctly interpret the DFP_D and UFP_D bitmasks and ignoring the bit indicating receptacle or plug, which flips the meaning of those masks.
With the following patch the MB16AMT monitor does now correctly get a pinmode of D (2 lanes USB + 2 lanes DP) since it also has the DP_STATUS_PREFER_MULTI_FUNC set.
The USB3 interface shows up as a hub and seems to provide a DisplayLink usb device, as backup to DP I guess...
The touch HID device is connected through the USB2 connection.
With the fix from above, patches (like this one) that check the reverse mapping and or like this fuse302-add-extcon.patch does, registering a port with identical pinmaps for UFP_D and DFP_D even though only one mode is actually supported (~line 292 fuse302-add-extcon.patch), are no longer necessary?
Anyway, so far everything seems to work now regarding the alt mode switching . Just some weirdness with DP and specific monitors.