Thanks -- turns out the current limit is 16 as defined by https://github.com/jernejsk/FFmpeg/blob/v4l2-request-hwaccel-4.4/libavcodec/v4l2_request_hevc.c#L24. I increased that in my ffmpeg build to 32 and that made HEVC work with moonlight-qt! At I was able to stream for a few minutes; I didn't see much difference between HEVC and H.264 in moonlight streaming (and had no luck forcing HDR), so for now I went back to the better supported H.264 codec. I'll definitely give this another try once the kernel APIs + userspace code stabilizes.
hege got a reaction from gounthar in Understanding Hardware-Accelerated Video Decoding