jernej Posted September 15, 2021 Posted September 15, 2021 1 hour ago, usual user said: So I will stay at the status quo of LE patches on 5.15.0-rc1 for now and wait till they rebase. Rebase usually happens around the time of stable release. You'll wait for more than a month. 0 Quote
jock Posted September 15, 2021 Posted September 15, 2021 1 hour ago, usual user said: With gstreamer framework it is working, even for bbb_sunflower_2160p_30fps_normal.mp4 without any problem. It is mpv that seems not be able to cope with the hight data rate. Useing --hwdec=drm-copy makes the CPU utilization ramp up go away and the video plays smoothly, but to slow and the sound get out of sync. I don't think I am missing anything kernel wise. Also on my tests drm-copy is consuming more cpu power. My guess that is because the decoded video buffer is copied for use as EGL image/texture, because the larger the video (and higher framerate) the bigger the cpu usage. Plus mpv uses a single thread for decoding when hardware decoding is in place, so when you saturate a core the video plays slow. I didn't try gstreamer yet, but is it working smooth for you in X11 window? 0 Quote
usual user Posted September 17, 2021 Posted September 17, 2021 On 9/15/2021 at 10:39 PM, jernej said: You'll wait for more than a month. Thank you for sharing the schedule. But no hurry, my daily use cases are already working anyway. I'm only doing this for my own education and as a preview of maybe landing features in Mainline. On 9/15/2021 at 10:50 PM, jock said: Also on my tests drm-copy is consuming more cpu power. For me, it's the other way around. hwdec=drm yield high CPU utilisation for bbb_sunflower_2160p_30fps_normal.mp4 and hwdec=drm-copy yield low CPU utilisation. As the values are fluctuating heavyly it is difficult to provide absolut numbers. But anyway here are some rough values: bbb_sunflower_1080p_30fps_normal.mp4: gst-play-1.0 ~35% mpv --hwdec=none --hwdec-codecs=all ~40% mpv --hwdec=drm --hwdec-codecs=all ~38% mpv --hwdec=drm-copy --hwdec-codecs=all ~28% bbb_sunflower_2160p_30fps_normal.mp4: gst-play-1.0 ~35% mpv --hwdec=none --hwdec-codecs=all ~98% playing slow, sound out of sync mpv --hwdec=drm --hwdec-codecs=all ~98% playing jerky, sound out of sync mpv --hwdec=drm-copy --hwdec-codecs=all ~34% playing slow, sound out of sync This is on plasma desktop with wayland backend. Desktop CPU usage fluctuates around 8 to 15% without playing a video, so it's uncertain which CPU cycles are really associated with video playback. The utilization is always distributed almost evenly over all cores. Frequency scaling is not considered here. On 9/15/2021 at 10:50 PM, jock said: I didn't try gstreamer yet, but is it working smooth for you in X11 window? For reference here the values for lxqt desktop with native xorg backend: bbb_sunflower_1080p_30fps_normal.mp4: gst-play-1.0 ~25% mpv --hwdec=none --hwdec-codecs=all ~30% mpv --hwdec=drm --hwdec-codecs=all ~28% mpv --hwdec=drm-copy --hwdec-codecs=all ~18% bbb_sunflower_2160p_30fps_normal.mp4: gst-play-1.0 ~18% paying as diashow mpv --hwdec=none --hwdec-codecs=all ~98% playing slow, sound out of sync mpv --hwdec=drm --hwdec-codecs=all ~98% playing jerky, sound out of sync mpv --hwdec=drm-copy --hwdec-codecs=all ~22% playing slow, sound out of sync Conlusion: The VPU decoder is not the bottleneck, but setting up an efifcient video pipeline with proper interaction of the several involved hardware acellerators. For Xwindow with the modeset driver this seems not realy possible. 0 Quote
ScottP Posted September 23, 2021 Posted September 23, 2021 Working for me on kernel 5.14.5 with ffmpeg https://github.com/jernejsk/FFmpeg/tree/v4l2-request-hwaccel-4.3.2 h264 1600*900 15fps stream from my cams use 17-22% of one CPU about 5% of which is pxfmt conversion because it reduces by that amount without conversion - no further kernel patches and ffmpeg dependencies all installed from default repos. This is awesome I've been wanting this for a while, my use case is Frigate NVR which is currently running on an old intel system with a nvidia gpu doing the decoding. I can now revert my nanopc t4 to the task and save some electricity. My ffmpeg command that emulates approximately what Frigate does: ffmpeg -loglevel warning -hwaccel drm -i rtsp://192.168.50.144:8554/unicast -pix_fmt yuv420p -f rawvideo pipe: Many thanks @jernejkwiboo and everyone else who made this possible Now need to find out why get block i/o errors and corruption on emmc on all 5.13 and 5.14 kernels I have tried, doing testing from sdcard for now. 2 Quote
ATK Posted September 25, 2021 Posted September 25, 2021 @ScottP... will definitely like to hear more from you on your NVR stuff... 0 Quote
ScottP Posted October 9, 2021 Posted October 9, 2021 On 9/25/2021 at 4:46 PM, ATK said: @ScottP... will definitely like to hear more from you on your NVR stuff... Hear is a lengthy reply I put on the Frigate NVR github for someone asking about hardware decoding for Frigate NVR. I would be interested if I am doing anything wrong here or I have missed a step. TL;DR It does not work reliably for me ATM but this is the closest to working I have seen so far. Work is ongoing in linux kernel and FFmpeg, it may work reliably sometime in the future. When the kernel drivers are moved out of staging and the interface to them is stable I expect to see a pull request on the main FFmpeg git. This is a long reply with information to test because I am giving up at this point and moving to a different platform. I would be interested if you find a solution though, or that I have missed something - hence the detailed reply. For testing you can try this fork of ffmpeg https://github.com/jernejsk/FFmpeg It has v4l2-request and libdrm stateless VPU decoding built in using hantro and rockchip_vdec/rkvdec. use kernel 5.14.9, armbian is a convenient way to change kernels - sudo armbian-config -> system -> Other kernels. FFmpeg from the above github has private headers for kernel interfaces and they are updated about a month after each release. You must install the correct userspace kernel headers, I just get the kernel source from https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.14.9.tar.xz and then do `make -j6 headers_install INSTALL_HDR_PATH=/usr` Do not use amrbian-config to install kernel headers - it installs the wrong version. Then install FFmpeg dependencies: `sudo apt install libudev-dev libfreetype-dev libmp3lame-dev libvorbis-dev libwebp-dev libx264-dev libx265-dev libssl-dev libdrm2 libdrm-dev pkg-config libfdk-aac-dev libopenjp2-7-dev` Run configure, this is a minimal set of options, frigate includes many more though, I removed many of them to build faster and save memory (I actually think there are a lot of redundant ffmpeg components in frigates default build files, some X11 frame grabber stuff and codecs nobody uses anymore, but thats for a separate discussion): ``` ./configure \ --enable-libdrm \ --enable-v4l2-request \ --enable-libudev \ --disable-debug \ --disable-doc \ --disable-ffplay \ --enable-shared \ --enable-libfreetype \ --enable-gpl \ --enable-libmp3lame \ --enable-libvorbis \ --enable-libwebp \ --enable-libx265 \ --enable-libx264 \ --enable-nonfree \ --enable-openssl \ --enable-libfdk_aac \ --enable-postproc \ --extra-libs=-ldl \ --prefix="${PREFIX}" \ --enable-libopenjpeg \ --extra-libs=-lpthread \ --enable-neon ``` Then `make -j6` I dont know if this next bit is correct, but it works for me, I dont want to do `make install` just run the ffmpeg tests from the build directory, to run tests you must run `sudo ldconfig $PWD $PWD/lib*` first otherwise linker will not find libraries. If you want to try a different kernel version run `make distclean` in FFmpeg and run ./configure again. If FFmpeg fails to build it will be because private headers do not match kernel headers. errors like V4L... undefined etc Then you can do some tests and see if you get valid output, for example, this decodes 15s from one of my cams: `./ffmpeg -benchmark -loglevel debug -hwaccel drm -i rtsp://192.168.50.144:8554/unicast -t 15 -pix_fmt yuv420p -f rawvideo out.yuv` Checks to make during and after decoding: Observe CPU usage, on my system rk3399 with 1.5Ghz little core and 2Ghz big core overclock I get between 17 and 25% cpu on one core, it varies if it runs on a53 little core or a72 big core. It should be better than that, I think its the way that the data is copied around in memory. Gstreamer or mpv attempt to do zero copy decoding so its more efficient. With software decoding CPU use is about 70% of one core. RK3328 does not have the two a72 cores and four a53 cores that RK3399 has, just four a53 cores so rk3328 about half as powerful as RK3399 as the a72 cores are about twice as powerful as the a53 cores. You should see in the debug output for ffmpeg where it tries each of the /dev/video interfaces to find the correct codec for decoding. Be warned that ffmpeg will sometimes just fall back to software decode, if that happens you will see much higher CPU usage and often ffmpeg will spawn a number of threads to use all cores in your system. Your user should be a member of the "video" group in /etc/group to access without sudo. Log snippet of that section below: ```[h264 @ 0xaaab06cd9070] Format drm_prime chosen by get_format(). [h264 @ 0xaaab06cd9070] Format drm_prime requires hwaccel initialisation. [h264 @ 0xaaab06cd9070] ff_v4l2_request_init: avctx=0xaaab06cd9070 hw_device_ctx=0xaaab06c549a0 hw_frames_ctx=(nil) [h264 @ 0xaaab06cd9070] v4l2_request_probe_media_device: avctx=0xaaab06cd9070 ctx=0xffff8804df20 path=/dev/media1 driver=hantro-vpu [h264 @ 0xaaab06cd9070] v4l2_request_probe_video_device: avctx=0xaaab06cd9070 ctx=0xffff8804df20 path=/dev/video1 capabilities=69222400 [h264 @ 0xaaab06cd9070] v4l2_request_try_format: pixelformat 875967059 not supported for type 10 [h264 @ 0xaaab06cd9070] v4l2_request_probe_video_device: try output format failed [h264 @ 0xaaab06cd9070] v4l2_request_probe_video_device: avctx=0xaaab06cd9070 ctx=0xffff8804df20 path=/dev/video2 capabilities=69222400 [h264 @ 0xaaab06cd9070] v4l2_request_try_format: pixelformat 875967059 not supported for type 10 [h264 @ 0xaaab06cd9070] v4l2_request_probe_video_device: try output format failed [h264 @ 0xaaab06cd9070] v4l2_request_probe_media_device: avctx=0xaaab06cd9070 ctx=0xffff8804df20 path=/dev/media0 driver=rkvdec [h264 @ 0xaaab06cd9070] v4l2_request_probe_video_device: avctx=0xaaab06cd9070 ctx=0xffff8804df20 path=/dev/video0 capabilities=69222400 [h264 @ 0xaaab06cd9070] v4l2_request_init_context: pixelformat=842094158 width=1600 height=912 bytesperline=1600 sizeimage=2918400 num_planes=1 [h264 @ 0xaaab06cd9070] ff_v4l2_request_frame_params: avctx=0xaaab06cd9070 ctx=0xffff8804df20 hw_frames_ctx=0xffff8804faa0 hwfc=0xffff8804e530 pool=0xffff8805e910 width=1600 height=912 initial_pool_size=3 ``` Check that the output file contains valid video data, try playing it using vlc: `vlc --rawvid-fps 10 --rawvid-width 1600 --rawvid-height 900 --rawvid-chroma I420 out.yuv` adjust the command to what height/width/fps your cameras record in. If all this is working then try doing longer decodes in parallel, eg is you have 3 cams run the ffmpeg command for each of them in a separate window and increase the time. What happens to me is that at some point ffmpeg will start reporting "resource not available/busy" or similar, rebooting will make it work for a while again. You can check what codecs are supported by each of the interfaces /dev/video[012] by `v4l2-ctl --all -d0` change d0 to d1 d2 etc to view the other decoders/encoders You can monitor the state of kernel development https://patchwork.kernel.org/project/linux-rockchip/list/ Most of the work on this is being done by Andrzej Pietrasiewicz. My suggestion is monitor both the ffmpeg github and kernel commits/patches, find out when they rebase ffmpeg. Pull that version and install the current kernel for it plus headers and retest. I have all the frigate docker files already created. I basically created a new set of dockerfiles with an arch of aarch64rockchip and added those to Makefile. I'll upload them to my github at some point, I see little point to a pull request since rockchip is a niche platform with not many users in home assistant or frigate, and it does not currently work for me reliably anyway. I have been trying to get this working for some time now, at kernel 5.4.* there were a bunch of kernel patches you had to apply. Nothing worked for me then. Often FFmpeg complained about the pixel format. There were some people on Armbian forums who claimed to have it working, but I had my doubts, maybe it was wishful thinking and ffmpeg was really using software decode. Most of the effort around this is for video playback so people can play 1080p and 2/4k videos on desktop and kodi. There is little information about straight decoding to a pipe like frigate. So in research ignore stuff to do with patched libva etc. For now I am using an old ~2013 i5-4670 four core/thread Haswell with Nvidia GT640 GPU for Frigate and Home Assistant. For three cams at 1600*900 10fps Frigate uses 6% CPU as reported by Home Assistant supervisor. It is very stable. With that in mind and wanting to use a more power efficient system I caved and ordered a Nvidia Jetson 4GB developer kit yesterday. I have confidence I can build Frigate docker containers for that system and it has a similar hardware decoder as their GPUs, I can also try out using CUDA filters and scaling to reduce CPU load for Frigate detector. A start would be to copy the amd64nvidia dockerfiles and create aarch64nvidia arch and modify from there it should be mostly the same. 2 Quote
ATK Posted October 9, 2021 Posted October 9, 2021 Can you try with " Xeoma " CCTV/IP camera NVR software ? give it a try .... 0 Quote
VyacheslavS Posted October 11, 2021 Posted October 11, 2021 09.10.2021 в 21:09, ScottP сказал: ...about hardware decoding for Frigate NVR.. Very interesting. I also want to test it in the near future. 0 Quote
Gustavo A. Díaz Posted October 19, 2021 Posted October 19, 2021 Hi, Does repo https://github.com/jernejsk/FFmpeg compiles against kernel 5.14.13 (5.14.13-rockchip64)? I am testing unstable Armbian for Rock64 (Armbian 21.11.0-trunk.38 Hirsute) and gives me: Quote libavcodec/v4l2_request_mpeg2.c:24:37: error: field ‘sequence’ has incomplete type 24 | struct v4l2_ctrl_mpeg2_sequence sequence; | ^~~~~~~~ libavcodec/v4l2_request_mpeg2.c:25:36: error: field ‘picture’ has incomplete type 25 | struct v4l2_ctrl_mpeg2_picture picture; | ^~~~~~~ libavcodec/v4l2_request_mpeg2.c:26:41: error: field ‘quantisation’ has incomplete type 26 | struct v4l2_ctrl_mpeg2_quantisation quantisation; | ^~~~~~~~~~~~ libavcodec/v4l2_request_mpeg2.c: In function ‘v4l2_request_mpeg2_start_frame’: libavcodec/v4l2_request_mpeg2.c:39:10: error: ‘struct v4l2_ctrl_mpeg2_sequence’ has no member named ‘horizontal_size’ 39 | .horizontal_size = s->width, | ^~~~~~~~~~~~~~~ libavcodec/v4l2_request_mpeg2.c:39:28: warning: excess elements in struct initializer 39 | .horizontal_size = s->width, | ^ libavcodec/v4l2_request_mpeg2.c:39:28: note: (near initialization for ‘(anonymous)’) libavcodec/v4l2_request_mpeg2.c:40:10: error: ‘struct v4l2_ctrl_mpeg2_sequence’ has no member named ‘vertical_size’ 40 | .vertical_size = s->height, | ^~~~~~~~~~~~~ libavcodec/v4l2_request_mpeg2.c:40:26: warning: excess elements in struct initializer 40 | .vertical_size = s->height, | ^ libavcodec/v4l2_request_mpeg2.c:40:26: note: (near initialization for ‘(anonymous)’) libavcodec/v4l2_request_mpeg2.c:41:10: error: ‘struct v4l2_ctrl_mpeg2_sequence’ has no member named ‘vbv_buffer_size’ 41 | .vbv_buffer_size = req->output.size, | ^~~~~~~~~~~~~~~ libavcodec/v4l2_request_mpeg2.c:41:28: warning: excess elements in struct initializer 41 | .vbv_buffer_size = req->output.size, | ^~~ libavcodec/v4l2_request_mpeg2.c:41:28: note: (near initialization for ‘(anonymous)’) libavcodec/v4l2_request_mpeg2.c:44:10: error: ‘struct v4l2_ctrl_mpeg2_sequence’ has no member named ‘profile_and_level_indication’ 44 | .profile_and_level_indication = 0, | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~ libavcodec/v4l2_request_mpeg2.c:44:41: warning: excess elements in struct initializer 44 | .profile_and_level_indication = 0, | ^ libavcodec/v4l2_request_mpeg2.c:44:41: note: (near initialization for ‘(anonymous)’) libavcodec/v4l2_request_mpeg2.c:45:10: error: ‘struct v4l2_ctrl_mpeg2_sequence’ has no member named ‘chroma_format’ 45 | .chroma_format = s->chroma_format, | ^~~~~~~~~~~~~ libavcodec/v4l2_request_mpeg2.c:45:26: warning: excess elements in struct initializer 45 | .chroma_format = s->chroma_format, | ^ libavcodec/v4l2_request_mpeg2.c:45:26: note: (near initialization for ‘(anonymous)’) libavcodec/v4l2_request_mpeg2.c:37:60: error: invalid use of undefined type ‘struct v4l2_ctrl_mpeg2_sequence’ 37 | controls->sequence = (struct v4l2_ctrl_mpeg2_sequence) { | ^ libavcodec/v4l2_request_mpeg2.c:49:37: error: ‘V4L2_MPEG2_SEQ_FLAG_PROGRESSIVE’ undeclared (first use in this function); did you mean ‘V4L2_DV_BT_CAP_PROGRESSIVE’? 49 | controls->sequence.flags |= V4L2_MPEG2_SEQ_FLAG_PROGRESSIVE; | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | V4L2_DV_BT_CAP_PROGRESSIVE libavcodec/v4l2_request_mpeg2.c:49:37: note: each undeclared identifier is reported only once for each function it appears in libavcodec/v4l2_request_mpeg2.c:53:10: error: ‘struct v4l2_ctrl_mpeg2_picture’ has no member named ‘picture_coding_type’ 53 | .picture_coding_type = s->pict_type, | ^~~~~~~~~~~~~~~~~~~ libavcodec/v4l2_request_mpeg2.c:53:32: warning: excess elements in struct initializer 53 | .picture_coding_type = s->pict_type, | ^ libavcodec/v4l2_request_mpeg2.c:53:32: note: (near initialization for ‘(anonymous)’) libavcodec/v4l2_request_mpeg2.c:56:10: error: ‘struct v4l2_ctrl_mpeg2_picture’ has no member named ‘f_code’ 56 | .f_code[0][0] = s->mpeg_f_code[0][0], | ^~~~~~ libavcodec/v4l2_request_mpeg2.c:56:25: warning: excess elements in struct initializer 56 | .f_code[0][0] = s->mpeg_f_code[0][0], | ^ libavcodec/v4l2_request_mpeg2.c:56:25: note: (near initialization for ‘(anonymous)’) libavcodec/v4l2_request_mpeg2.c:57:10: error: ‘struct v4l2_ctrl_mpeg2_picture’ has no member named ‘f_code’ 57 | .f_code[0][1] = s->mpeg_f_code[0][1], | ^~~~~~ libavcodec/v4l2_request_mpeg2.c:57:25: warning: excess elements in struct initializer 57 | .f_code[0][1] = s->mpeg_f_code[0][1], | ^ libavcodec/v4l2_request_mpeg2.c:57:25: note: (near initialization for ‘(anonymous)’) libavcodec/v4l2_request_mpeg2.c:58:10: error: ‘struct v4l2_ctrl_mpeg2_picture’ has no member named ‘f_code’ 58 | .f_code[1][0] = s->mpeg_f_code[1][0], | ^~~~~~ libavcodec/v4l2_request_mpeg2.c:58:25: warning: excess elements in struct initializer 58 | .f_code[1][0] = s->mpeg_f_code[1][0], | ^ libavcodec/v4l2_request_mpeg2.c:58:25: note: (near initialization for ‘(anonymous)’) libavcodec/v4l2_request_mpeg2.c:59:10: error: ‘struct v4l2_ctrl_mpeg2_picture’ has no member named ‘f_code’ 59 | .f_code[1][1] = s->mpeg_f_code[1][1], | ^~~~~~ libavcodec/v4l2_request_mpeg2.c:59:25: warning: excess elements in struct initializer 59 | .f_code[1][1] = s->mpeg_f_code[1][1], | ^ libavcodec/v4l2_request_mpeg2.c:59:25: note: (near initialization for ‘(anonymous)’) libavcodec/v4l2_request_mpeg2.c:60:10: error: ‘struct v4l2_ctrl_mpeg2_picture’ has no member named ‘picture_structure’ 60 | .picture_structure = s->picture_structure, | ^~~~~~~~~~~~~~~~~ libavcodec/v4l2_request_mpeg2.c:60:30: warning: excess elements in struct initializer 60 | .picture_structure = s->picture_structure, | ^ libavcodec/v4l2_request_mpeg2.c:60:30: note: (near initialization for ‘(anonymous)’) libavcodec/v4l2_request_mpeg2.c:61:10: error: ‘struct v4l2_ctrl_mpeg2_picture’ has no member named ‘intra_dc_precision’ 61 | .intra_dc_precision = s->intra_dc_precision, | ^~~~~~~~~~~~~~~~~~ libavcodec/v4l2_request_mpeg2.c:61:31: warning: excess elements in struct initializer 61 | .intra_dc_precision = s->intra_dc_precision, | ^ libavcodec/v4l2_request_mpeg2.c:61:31: note: (near initialization for ‘(anonymous)’) libavcodec/v4l2_request_mpeg2.c:51:58: error: invalid use of undefined type ‘struct v4l2_ctrl_mpeg2_picture’ 51 | controls->picture = (struct v4l2_ctrl_mpeg2_picture) { | ^ libavcodec/v4l2_request_mpeg2.c:65:36: error: ‘V4L2_MPEG2_PIC_FLAG_TOP_FIELD_FIRST’ undeclared (first use in this function) 65 | controls->picture.flags |= V4L2_MPEG2_PIC_FLAG_TOP_FIELD_FIRST; | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ libavcodec/v4l2_request_mpeg2.c:68:36: error: ‘V4L2_MPEG2_PIC_FLAG_FRAME_PRED_DCT’ undeclared (first use in this function) 68 | controls->picture.flags |= V4L2_MPEG2_PIC_FLAG_FRAME_PRED_DCT; | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ libavcodec/v4l2_request_mpeg2.c:71:36: error: ‘V4L2_MPEG2_PIC_FLAG_CONCEALMENT_MV’ undeclared (first use in this function) 71 | controls->picture.flags |= V4L2_MPEG2_PIC_FLAG_CONCEALMENT_MV; | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ libavcodec/v4l2_request_mpeg2.c:74:36: error: ‘V4L2_MPEG2_PIC_FLAG_INTRA_VLC’ undeclared (first use in this function) 74 | controls->picture.flags |= V4L2_MPEG2_PIC_FLAG_INTRA_VLC; | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ libavcodec/v4l2_request_mpeg2.c:77:36: error: ‘V4L2_MPEG2_PIC_FLAG_Q_SCALE_TYPE’ undeclared (first use in this function) 77 | controls->picture.flags |= V4L2_MPEG2_PIC_FLAG_Q_SCALE_TYPE; | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ libavcodec/v4l2_request_mpeg2.c:80:36: error: ‘V4L2_MPEG2_PIC_FLAG_ALT_SCAN’ undeclared (first use in this function) 80 | controls->picture.flags |= V4L2_MPEG2_PIC_FLAG_ALT_SCAN; | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~ CC libavcodec/vble.o libavcodec/v4l2_request_mpeg2.c:83:36: error: ‘V4L2_MPEG2_PIC_FLAG_REPEAT_FIRST’ undeclared (first use in this function) 83 | controls->picture.flags |= V4L2_MPEG2_PIC_FLAG_REPEAT_FIRST; | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ libavcodec/v4l2_request_mpeg2.c:86:36: error: ‘V4L2_MPEG2_PIC_FLAG_PROGRESSIVE’ undeclared (first use in this function); did you mean ‘V4L2_DV_BT_CAP_PROGRESSIVE’? 86 | controls->picture.flags |= V4L2_MPEG2_PIC_FLAG_PROGRESSIVE; | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | V4L2_DV_BT_CAP_PROGRESSIVE libavcodec/v4l2_request_mpeg2.c: In function ‘v4l2_request_mpeg2_end_frame’: libavcodec/v4l2_request_mpeg2.c:121:19: error: ‘V4L2_CID_STATELESS_MPEG2_SEQUENCE’ undeclared (first use in this function); did you mean ‘V4L2_CID_STATELESS_VP8_FRAME’? 121 | .id = V4L2_CID_STATELESS_MPEG2_SEQUENCE, | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | V4L2_CID_STATELESS_VP8_FRAME libavcodec/v4l2_request_mpeg2.c:126:19: error: ‘V4L2_CID_STATELESS_MPEG2_PICTURE’ undeclared (first use in this function); did you mean ‘V4L2_CID_STATELESS_VP8_FRAME’? 126 | .id = V4L2_CID_STATELESS_MPEG2_PICTURE, | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | V4L2_CID_STATELESS_VP8_FRAME libavcodec/v4l2_request_mpeg2.c:131:19: error: ‘V4L2_CID_STATELESS_MPEG2_QUANTISATION’ undeclared (first use in this function) 131 | .id = V4L2_CID_STATELESS_MPEG2_QUANTISATION, | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ make: *** [ffbuild/common.mak:59: libavcodec/v4l2_request_mpeg2.o] Error 1 make: *** Waiting for unfinished jobs.... Configured with: Quote ./configure --prefix=/usr --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --enable-libdrm --enable-v4l2-request --enable-libudev --di sable-debug --disable-doc --disable-ffplay --enable-shared --enable-libfreetype --enable-gpl --enable-libmp3lame --enable-libvorbis --enable-libwebp --enable-libx265 --enable-libx264 --enable-nonfree --enable-ope nssl --enable-libfdk_aac --enable-postproc --extra-libs=-ldl --enable-libopenjpeg --extra-libs=-lpthread --enable-neon Thanks in advance. 0 Quote
ScottP Posted November 4, 2021 Posted November 4, 2021 On 10/19/2021 at 9:43 PM, Gustavo A. Díaz said: Hi, Does repo https://github.com/jernejsk/FFmpeg compiles against kernel 5.14.13 (5.14.13-rockchip64)? I am testing unstable Armbian for Rock64 (Armbian 21.11.0-trunk.38 Hirsute) and gives me: Configured with: Thanks in advance. Your kernel headers dont match FFmpeg. 0 Quote
jock Posted November 10, 2021 Posted November 10, 2021 Hello, made this tutorial that may be helpful to test hardware video decoding easily using latest edge kernels armbian builds. @jernej is it currently possible to use directly map decoded frames into EGL image handles and then use OpenGL to draw frames as textures in X11 or Wayland? I mean, if you take a look to the tutorial, I have to use drm-copy when I want hardware acceleration in a X11 window, so memory copy happens and slow CPUs could not cope with it. I can't use drm because the device is kept busy by X11. Is mpv missing a "drm-egl" thing or the capability is not (yet) in the kernel? Thanks! 0 Quote
jernej Posted November 11, 2021 Posted November 11, 2021 I don't use any desktop environment (X11 or Wayland) on SBCs, so I can't tell you what works and what not. However, I can tell you that v4l2 request drivers decode video frames to dma-buf, which can be directly imported into DRM or OpenGL ES (mesa exposes functions to do that, but afaik it convert colour space with GPU, still much quicker than CPU). Later option should work everywhere where OpenGL ES is supported, so under X11 and Wayland too. At first glance, it seems that mpv doesn't have this implemented, but I'm not sure. Kodi has both rendering methods implemented, but only when using GBM variant (nobody bothered to enabled that for other environments). 1 Quote
rubenvb Posted November 19, 2021 Posted November 19, 2021 I am currently running kernel 5.14.9 with the LibreELEC patches applied that were at that time created against that version. I then compiled Kodi with FFmpeg from the v4l2-request-hwaccel-4.3.2 branch of https://github.com/jernejsk/FFmpeg The result can play MPEG2 and H264 video with DRM prime accel turned on, has 4k resolution available, and HDMI-CEC which is all quite awesome. Only thing not working for me is 10-bit HEVC, which proceeds to play audio but freezes on the Kodi menu. I'm unsure of where the problem lies, but it seems like an unsupported color format: Quote 2021-11-19 19:54:26.505 T:692608 INFO <general>: [WHITELIST] Searching the whitelist for: width: 1920, height: 1080, fps: 23.976, 3D: false 2021-11-19 19:54:26.506 T:692608 INFO <general>: Display resolution ADJUST : 3840x2160 @ 23.976025 Hz (27) (weight: 0.000) 2021-11-19 19:54:26.773 T:610 INFO <general>: VideoPlayer: OnLostDisplay received 2021-11-19 19:54:26.773 T:610 WARNING <general>: CDVDMessageQueue(audio)::Put MSGQ_NOT_INITIALIZED 2021-11-19 19:54:26.773 T:610 WARNING <general>: CDVDMessageQueue(video)::Put MSGQ_NOT_INITIALIZED 2021-11-19 19:54:26.773 T:610 INFO <general>: GLES: Maximum texture width: 8192 2021-11-19 19:54:26.775 T:692608 INFO <general>: Creating video codec with codec id: 173 2021-11-19 19:54:26.776 T:692608 INFO <general>: CDVDVideoCodecDRMPRIME::Open - using decoder HEVC (High Efficiency Video Coding) 2021-11-19 19:54:26.779 T:692608 INFO <general>: Creating video thread 2021-11-19 19:54:26.779 T:692616 INFO <general>: running thread: video_thread 2021-11-19 19:54:26.780 T:692608 INFO <general>: Opening stream: 1 source: 256 2021-11-19 19:54:26.780 T:692608 INFO <general>: Finding audio codec for: 86018 2021-11-19 19:54:26.781 T:692608 INFO <general>: CDVDAudioCodecFFmpeg::Open() Successful opened audio decoder aac 2021-11-19 19:54:26.781 T:692608 INFO <general>: Creating audio thread 2021-11-19 19:54:26.782 T:692617 INFO <general>: running thread: CVideoPlayerAudio::Process() 2021-11-19 19:54:26.864 T:610 INFO <general>: VideoPlayer: OnResetDisplay received 2021-11-19 19:54:26.885 T:692617 INFO <general>: CVideoPlayerAudio: display reset occurred, checking for passthrough 2021-11-19 19:54:26.887 T:692617 INFO <general>: CDVDAudioCodecFFmpeg::Open() Successful opened audio decoder aac 2021-11-19 19:54:26.887 T:692617 INFO <general>: Creating audio stream (codec id: 86018, channels: 6, sample rate: 48000, no pass-through) 2021-11-19 19:54:26.888 T:692610 ERROR <general>: CDVDVideoCodecDRMPRIME::GetFormat - unsupported pixel format 2021-11-19 19:54:26.889 T:637 INFO <general>: CActiveAESink::OpenSink - initialize sink 2021-11-19 19:54:26.927 T:692616 ERROR <general>: CDVDVideoCodecDRMPRIME::AddData - send packet failed: Operation not permitted (-1) 2021-11-19 19:54:26.983 T:637 INFO <general>: CAESinkALSA::Initialize - Attempting to open device "default" 2021-11-19 19:54:26.988 T:637 INFO <general>: CAESinkALSA::Initialize - Opened device "default" 2021-11-19 19:54:26.990 T:637 INFO <general>: CAESinkALSA::InitializeHW - Your hardware does not support AE_FMT_FLOAT, trying other formats 2021-11-19 19:54:26.990 T:637 INFO <general>: CAESinkALSA::InitializeHW - Using data format AE_FMT_S24NE4 2021-11-19 19:54:27.046 T:692616 ERROR <general>: CDVDVideoCodecDRMPRIME::GetPicture - videoBuffer:nullptr format:yuv420p10le 2021-11-19 19:54:27.047 T:692616 ERROR <general>: Skipped 2 duplicate messages.. 2021-11-19 19:54:27.047 T:692616 ERROR <general>: CDVDVideoCodecDRMPRIME::AddData - send packet failed: Invalid data found when processing input ( -1094995529) After this and the AE_FMT_FLOAT support is fixed, this thing will be golden. I notice this is the color format I reported previously that was completely borked when played by FFmpeg, but subsequently reported as working somehow in Kodi itself by myself. So this seems like some weird/transient regression against a previous state of all the code involved (somewhere between kernel/ffmpeg/kodi). Are there any patches I can test for either the yuv420p10le or AE_FMT_FLOAT support? It's no problem for me to recompile any component. Anything to help each other out :). For now, the powerful rk3399 CPU can just about decode the HEVC content all by itself, if the scenes aren't overly complex or fast-moving. 0 Quote
usual user Posted November 23, 2021 Posted November 23, 2021 VP9 support has just landed in media staging, i.e. is in flight for 5.17.0. I moved to 5.16.0-rc2 with media staging patches applied. LE 0000-linux-0011-v4l2-from-list.patch and 0000-linux-1001-v4l2-rockchip.patch where also applied after resolving small merge conflicts or dropping already applied commits. 0000-linux-2001-v4l-wip-rkvdec-hevc.patch can no longer be applied, I will wait patiently until LE rebases. I'm not even sure if hevc works at all, because with 5.15.0 and the patch, mpv does not pick drm-copy as it does like for VP9. Spoiler 0 Quote
jernej Posted November 24, 2021 Posted November 24, 2021 Note that final VP9 API, which landed in media repo, is slightly different than before and new ffmpeg patches will be needed. We only have WIP variant for new VP9 API. It works, but breaks SW VP9 decoding. We have some time untill it lands in stable kernel... 0 Quote
usual user Posted November 24, 2021 Posted November 24, 2021 2 hours ago, jernej said: Note that final VP9 API, which landed in media repo, is slightly different than before and new ffmpeg patches will be needed. I already use these patches since I build 5.15.0. I switched to 5.17.0 preview to see what's currently missing when it comes out. AFAIS I have only to wait for a rebase of "Draft: v4l2codecs: Implement VP9 v4l2 decoder" for gstreamer and LE 0000-linux-2001-v4l-wip-rkvdec-hevc.patch for kernel to have all the basics for further experiments in place. For now only mpv works to some extend. 0 Quote
jernej Posted November 24, 2021 Posted November 24, 2021 Ah, yes, that should work with upstream VP9 API. I'm not sure what could broke with kernel update. There are few HEVC patches that were upstreamed. Maybe order of the fields changed? You can try simple experiment. Copy hevc-ctrl.h from patched kernel sources to ffmpeg sources (replace existing file) and rebuild ffmpeg. If it fails to build, you know some, probably minor, updates are needed. If it goes through, it might just work. 0 Quote
usual user Posted November 25, 2021 Posted November 25, 2021 7 hours ago, jernej said: Copy hevc-ctrl.h from patched kernel sources to ffmpeg sources (replace existing file) and rebuild ffmpeg. hevc-ctrl.h does not exist, but hevc-ctrls.h are identical: --- hevc-ctrls.h-kernel 2021-09-08 20:51:20.802970171 +0200 +++ hevc-ctrls.h-ffmpeg 2021-10-11 23:43:54.894127584 +0200 @@ -210,7 +210,7 @@ struct v4l2_ctrl_hevc_slice_params { __u16 short_term_ref_pic_set_size; __u16 long_term_ref_pic_set_size; - + __u32 num_entry_point_offsets; __u32 entry_point_offset_minus1[256]; __u8 padding[8]; 0 Quote
iamdrq Posted November 25, 2021 Posted November 25, 2021 On 11/12/2021 at 12:37 AM, jernej said: I don't use any desktop environment (X11 or Wayland) on SBCs, so I can't tell you what works and what not. However, I can tell you that v4l2 request drivers decode video frames to dma-buf, which can be directly imported into DRM or OpenGL ES (mesa exposes functions to do that, but afaik it convert colour space with GPU, still much quicker than CPU). Later option should work everywhere where OpenGL ES is supported, so under X11 and Wayland too. At first glance, it seems that mpv doesn't have this implemented, but I'm not sure. Kodi has both rendering methods implemented, but only when using GBM variant (nobody bothered to enabled that for other environments). I tested on the gnome 40 with wayland and the orangepi4 kernel is armbian apt 5.15 without apply any patch. I used gstreamer 1.19.3 from source code build(I use dt-overylay disable rockchip-vpu,retain rockchip-rkvdec,if not do that gstreamer can't use v4l2.I known 1.20 haven't release but I need webkit2gtk that use gstreamer ). gst-play-1.0 --use-playbin3 --videosink="gtksink" ~/demo.mp4 video show slow and cpu cost about 70%,I know it used hardware decode video from gstreamer log, but I don't understand cpu cost too big. gst-play-1.0 --use-playbin3 --videosink="glimagesink" ~/demo.mp4 Press 'k' to see a list of keyboard shortcuts. Now playing /home/drq/ggxs.mp4 Redistribute latency... Redistribute latency... Redistribute latency.. 0:00:00.0 / 0:04:04.0 PIPE_FORMAT_B10G10R10A2_UNORM PIPE_FORMAT_B10G10R10A2_UNORM PIPE_FORMAT_B10G10R10A2_UNORM PIPE_FORMAT_B10G10R10A2_UNORM PIPE_FORMAT_B10G10R10A2_UNORM PIPE_FORMAT_B10G10R10A2_UNORM PIPE_FORMAT_B10G10R10A2_UNORM PIPE_FORMAT_B10G10R10A2_UNORM ... no video show and crazy print PIPE_FORMAT_B10G10R10A2_UNORM 0 Quote
jernej Posted November 25, 2021 Posted November 25, 2021 @usual userno idea then. Best to wait on RK devs to update patches, but that will take some time... @iamdrqIt looks you're using 10-bit video sample. It's always good idea to try 8-bit first. 0 Quote
iamdrq Posted November 26, 2021 Posted November 26, 2021 6 hours ago, jernej said: @usual userno idea then. Best to wait on RK devs to update patches, but that will take some time... @iamdrqIt looks you're using 10-bit video sample. It's always good idea to try 8-bit first. No, I sure the video is 8-bit. I notice support 8-bit now. demo320x240.mp4 0 Quote
usual user Posted November 26, 2021 Posted November 26, 2021 Plays flawless for me with this video-pipeline.pdf. 0 Quote
iamdrq Posted November 26, 2021 Posted November 26, 2021 4 hours ago, usual user said: Plays flawless for me with this video-pipeline.pdf. I found you use xvImageSink, Could you test this glimagesink ? gst-play-1.0 --use-playbin3 --videosink="glimagesink" ~/demo.mp4 0 Quote
usual user Posted November 26, 2021 Posted November 26, 2021 6 hours ago, iamdrq said: Could you test this glimagesink ? As expected, no valid video-pipeline.pdf is created. [plasma@trial-01 video]$ gst-play-1.0 --use-playbin3 --videosink="glimagesink" demo320x240.mp4.149ba2bb88584b89814a1c41b5feef77.mp4 Press 'k' to see a list of keyboard shortcuts. Now playing /home/plasma/workbench/video/demo320x240.mp4.149ba2bb88584b89814a1c41b5feef77.mp4 ERROR Driver did not report framing and start code method. for file:///home/plasma/workbench/video/demo320x240.mp4.149ba2bb88584b89814a1c41b5feef77.mp4 ERROR debug information: ../sys/v4l2codecs/gstv4l2codech264dec.c(212): gst_v4l2_codec_h264_dec_open (): /GstPlayBin3:playbin/GstURIDecodeBin3:uridecodebin3-0/GstDecodebin3:decodebin3-0/v4l2slh264dec:v4l2slh264dec0: gst_v4l2_decoder_get_controls() failed: Invalid argument Reached end of play list. Useing a mem to mem 3D accellerator in arm world with Wayland environment in a video-pipeline is a bad idea. 0 Quote
iamdrq Posted November 27, 2021 Posted November 27, 2021 16 hours ago, usual user said: As expected, no valid video-pipeline.pdf is created. [plasma@trial-01 video]$ gst-play-1.0 --use-playbin3 --videosink="glimagesink" demo320x240.mp4.149ba2bb88584b89814a1c41b5feef77.mp4 Press 'k' to see a list of keyboard shortcuts. Now playing /home/plasma/workbench/video/demo320x240.mp4.149ba2bb88584b89814a1c41b5feef77.mp4 ERROR Driver did not report framing and start code method. for file:///home/plasma/workbench/video/demo320x240.mp4.149ba2bb88584b89814a1c41b5feef77.mp4 ERROR debug information: ../sys/v4l2codecs/gstv4l2codech264dec.c(212): gst_v4l2_codec_h264_dec_open (): /GstPlayBin3:playbin/GstURIDecodeBin3:uridecodebin3-0/GstDecodebin3:decodebin3-0/v4l2slh264dec:v4l2slh264dec0: gst_v4l2_decoder_get_controls() failed: Invalid argument Reached end of play list. Useing a mem to mem 3D accellerator in arm world with Wayland environment in a video-pipeline is a bad idea. I can use gst-play-1.0 --videosink="waylandsink" play 4k 30fps video normal and keep cpu rest when I change gnome to weston, so maybe gnome has some problem. but weston only has a terminal not like a desktop. what is your desktop ? on the other hand, don't need disable rockchip-vpu node, gstreamer auto choose rkvdec when rockchip-vpu fail, and I found rkvdec decode h264 output format is NV12, not need convert when use xvimagesink, need convert NV12 to RGB when use waylandsink (cpu convert) or glimagesink (gpu convert, but it has problem now). 0 Quote
usual user Posted November 27, 2021 Posted November 27, 2021 2 hours ago, iamdrq said: so maybe gnome has some problem I don't know if you run Gnome with Xwindow or Wayland backend 2 hours ago, iamdrq said: what is your desktop ? With Xwindow backend I am running LXQt and with Wayland backend I use Plasma. 2 hours ago, iamdrq said: on the other hand, don't need disable rockchip-vpu node, gstreamer auto choose rkvdec when rockchip-vpu fail This are the video resources my current kernel exposes: videoX-infos.txt Of course, HEVC support is missing because the LE patch cannot be applied. 0 Quote
iamdrq Posted November 30, 2021 Posted November 30, 2021 On 11/27/2021 at 9:44 PM, usual user said: I don't know if you run Gnome with Xwindow or Wayland backend With Xwindow backend I am running LXQt and with Wayland backend I use Plasma. This are the video resources my current kernel exposes: videoX-infos.txt Of course, HEVC support is missing because the LE patch cannot be applied. I run Gnome with Wayland backend. I have a try about Plasma, but maybe I get used to Gnome. gst-play-1.0 --videosink="glimagesink" demo320x240.mp4.149ba2bb88584b89814a1c41b5feef77.mp4 Could you try this again ? without --use-playbin3 because it isn't try another vpu when fail, and you not disable rockchip-vpu node so fail. 0 Quote
usual user Posted November 30, 2021 Posted November 30, 2021 8 hours ago, iamdrq said: Could you try this again ? This one is playing: video-pipeline.pdf 0 Quote
iamdrq Posted December 1, 2021 Posted December 1, 2021 3 hours ago, usual user said: This one is playing: video-pipeline.pdf ok. thanks. are you run it on wayland and rk3399 soc ? it's so strange why fail on my device. 0 Quote
usual user Posted December 1, 2021 Posted December 1, 2021 5 hours ago, iamdrq said: are you run it on wayland and rk3399 soc ? Yes, but I prefer the video pipeline that gst-play auto negotiates because it gives me a resizeable, semitransparent window while moveing and window decorations. 5 hours ago, iamdrq said: it's so strange why fail on my device. It is probably to blame a misconfigured display pipeline of the compositor if the video pipeline is identical. Maybe attach the video pipeline that is negotiated with this command on GNOME: GST_DEBUG_DUMP_DOT_DIR=. gst-play-1.0 demo320x240.mp4.149ba2bb88584b89814a1c41b5feef77.mp4 For visualisation convert the dot-file by: dot -Tpdf foo.dot > foo.pdf 0 Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.