I've build ffmpeg from sources with suport for hw acceleration for h264 for rpi3
It is not possible to achive same for plex because their Plex Transcoder is modified ffmpeg. And sources are closed...
I've created topic on a plex forums maybe they can give their sources https://forums.plex.tv/discussion/302965/plex-transcoder-actual-sources