So, apparently switching to PPP didn't solve the issue, just made it less worse.
My kern.log is now being spammed with -
May 14 09:34:09 pikachu3 kernel: [18693.614909] dwc2 c9100000.usb: dwc2_hc_chhltd_intr_dma: Channel 1 - ChHltd set, but reason is unknown
May 14 09:34:09 pikachu3 kernel: [18693.618514] dwc2 c9100000.usb: hcint 0x00000002, intsts 0x04200009
May 14 09:34:16 pikachu3 kernel: [18700.943811] dwc2 c9100000.usb: dwc2_hc_chhltd_intr_dma: Channel 12 - ChHltd set, but reason is unknown
May 14 09:34:16 pikachu3 kernel: [18700.947505] dwc2 c9100000.usb: hcint 0x00000002, intsts 0x04200009
May 14 09:34:20 pikachu3 kernel: [18705.084439] dwc2 c9100000.usb: dwc2_hc_chhltd_intr_dma: Channel 6 - ChHltd set, but reason is unknown
May 14 09:34:20 pikachu3 kernel: [18705.088048] dwc2 c9100000.usb: hcint 0x00000002, intsts 0x04200009
May 14 09:34:23 pikachu3 kernel: [18708.092196] dwc2 c9100000.usb: dwc2_hc_chhltd_intr_dma: Channel 12 - ChHltd set, but reason is unknown
May 14 09:34:23 pikachu3 kernel: [18708.095887] dwc2 c9100000.usb: hcint 0x00000002, intsts 0x04200009
May 14 09:34:26 pikachu3 kernel: [18711.067310] dwc2 c9100000.usb: dwc2_hc_chhltd_intr_dma: Channel 7 - ChHltd set, but reason is unknown
May 14 09:34:26 pikachu3 kernel: [18711.070917] dwc2 c9100000.usb: hcint 0x00000002, intsts 0x04200009
May 14 09:34:28 pikachu3 kernel: [18712.597679] dwc2 c9100000.usb: dwc2_hc_chhltd_intr_dma: Channel 11 - ChHltd set, but reason is unknown
May 14 09:34:28 pikachu3 kernel: [18712.601374] dwc2 c9100000.usb: hcint 0x00000002, intsts 0x04200009
When I download / upload something.