First of all, thank you @leeson for pointing out that jellyfin image, transcoding seems to work.
However, I also want home decoding in client app, since this board sometimes is connected to Armbian from orangepi website, hwaccel was "enabled" from their instructions, but mpv was not using it/neither did jellyfin client.
When I tried to compile modified ffmpeg from nyanmisaka github the screen goes blank, and gnome session fails to start. There were error messages in journalctl about mpp, and something along these lines recently:
Jan 09 22:04:01 orangepi5 gnome-shell[34647]: Running GNOME Shell (using mutter 42.9) as a Wayland display server
Jan 09 22:04:01 orangepi5 gnome-shell[34647]: Added device '/dev/dri/card0' (rockchip) using atomic mode setting.
Jan 09 22:04:01 orangepi5 gnome-shell[34647]: g_hash_table_destroy: assertion 'hash_table != NULL' failed
Jan 09 22:04:01 orangepi5 gnome-shell[34647]: Failed to open gpu '/dev/dri/card1': No suitable mode setting backend found
Jan 09 22:04:01 orangepi5 gnome-shell[34647]: g_str_has_prefix: assertion 'str != NULL' failed
Jan 09 22:04:01 orangepi5 gnome-shell[34647]: g_str_has_prefix: assertion 'str != NULL' failed
Jan 09 22:04:01 orangepi5 gnome-shell[34647]: g_str_has_prefix: assertion 'str != NULL' failed
Jan 09 22:04:01 orangepi5 gnome-shell[34647]: Failed to get string: No error has been recorded.
Jan 09 22:04:01 orangepi5 systemd[7299]: org.gnome.Shell@wayland.service: Main process exited, code=killed, status=11/SEGV
Jan 09 22:04:01 orangepi5 systemd[7299]: org.gnome.Shell@wayland.service: Failed with result 'signal'.
Jan 09 22:04:01 orangepi5 systemd[7299]: Failed to start GNOME Shell on Wayland.
For the reference I was running Armbian 23.11.1 Jammy with Linux 5.10.160-legacy-rk35xx
downloaded from the website not so long ago, not sure if this version involves all required kernel patches or not.
What would be the general direction to troubleshoot? Reinstall everything gpu related, then try compiling ffmpeg again?