Frame drops when using kmssink to mirror screen playback

[iesv]

export GST_DEBUG=2

log

When I use kmssink to mirror the screen for video playback, I experience frame drops. The logs output a large number of messages like this:

[fduncanh]

what hardware are you using? you are dropping 20 frames per second (2/3) in a stream of 30fps. so your effective video is 10fps, definitely jerky.

The tested system kmssink here also dropped frames, but about 10 frames per second, leaving a perfectly acceptable video
at 20fps kmssink dropped less frames than fbdevsink.

This was on an recent intel system with an on-board GPU doing h264 decoding..

On the desktop (using xvimagesink) no frames were dropped.

Video Frames are dropped if they can't be decoded in time to accompany the audio frame that matches them.

So I guesss your system hardware is low in cpu computing power for rendering h264 video, and does not have a gpu.

What is it?

• In any case I dont think this is a software bug.

[iesv]

Yes. I don’t have a dedicated graphics card. My configuration (cpu) is as follows:

Dell OptiPlex 3090 (0B8A)

/0 bus 02KGF7
/0/0 memory 64KiB BIOS
/0/9 memory 16GiB System Memory
/0/9/0 memory 8GiB SODIMM DDR4 Synchronous 3200 MHz (0.3 ns)
/0/9/1 memory 8GiB SODIMM DDR4 Synchronous 3200 MHz (0.3 ns)
/0/26 memory 384KiB L1 cache
/0/27 memory 1536KiB L2 cache
/0/28 memory 12MiB L3 cache
/0/29 processor Intel(R) Core(TM) i5-10500T CPU @ 2.30GHz

[fduncanh]

Your computer certainly has enough power, but you are not using its intel integrated graphics.

sudo apt install gstreamer1.0-vaapi

• from the README:

VAAPI for Intel and AMD integrated graphics, ...

With an Intel or AMD GPU, hardware decoding with the open-source VAAPI gstreamer plugin is preferable.

Installing plugins (Debian-based Linux distributions, including Ubuntu and Raspberry Pi OS) (skip if you built a complete GStreamer from source)

Next install the GStreamer plugins that are needed with sudo apt install gstreamer1.0-. Values of required are:

"plugins-base"
"libav" (for sound),
"plugins-good" (for v4l2 hardware h264 decoding)
"plugins-bad" (for h264 decoding).

Debian-based distributions split some of the plugin packages into smaller pieces: some that may also be needed include "gl" for OpenGL support (this provides the "-vs glimagesink" videosink, which can be very useful in many systems (including Raspberry Pi), and should always be used when using h264/h265 decoding by a NVIDIA GPU), "gtk3" (which provides the "-vs gtksink" videosink), and "x" for X11 support, although these may already be installed; "vaapi" is needed for hardware-accelerated h264 video decoding by Intel or AMD graphics (but not for use with NVIDIA using proprietary drivers). If sound is not working, "alsa"","pulseaudio", or "pipewire" plugins may need to be installed, depending on how your audio is set up.

Also install "gstreamer1.0-tools" to get the utility gst-inspect-1.0 for examining the GStreamer installation.

[iesv]

Your computer certainly has enough power, but you are not using its intel integrated graphics.

sudo apt install gstreamer1.0-vaapi

• from the README:

VAAPI for Intel and AMD integrated graphics, ...
With an Intel or AMD GPU, hardware decoding with the open-source VAAPI gstreamer plugin is preferable.

Installing plugins (Debian-based Linux distributions, including Ubuntu and Raspberry Pi OS) (skip if you built a complete GStreamer from source)

Next install the GStreamer plugins that are needed with sudo apt install gstreamer1.0-. Values of required are:

"plugins-base"
"libav" (for sound),
"plugins-good" (for v4l2 hardware h264 decoding)
"plugins-bad" (for h264 decoding).

Debian-based distributions split some of the plugin packages into smaller pieces: some that may also be needed include "gl" for OpenGL support (this provides the "-vs glimagesink" videosink, which can be very useful in many systems (including Raspberry Pi), and should always be used when using h264/h265 decoding by a NVIDIA GPU), "gtk3" (which provides the "-vs gtksink" videosink), and "x" for X11 support, although these may already be installed; "vaapi" is needed for hardware-accelerated h264 video decoding by Intel or AMD graphics (but not for use with NVIDIA using proprietary drivers). If sound is not working, "alsa"","pulseaudio", or "pipewire" plugins may need to be installed, depending on how your audio is set up.

Also install "gstreamer1.0-tools" to get the utility gst-inspect-1.0 for examining the GStreamer installation.

In fact, I’ve already installed all of these, but it still drops frames. I’ll try recompiling again tomorrow and see how it goes.

[iesv]

This is the installed GStreamer list, but it did not resolve the frame dropping issue with kmssink.

aa.txt

[fduncanh]

you need to make sure that vaapi264dec is used for hardware -accelerated videodecoding. You make still have some framedropping, but much less.

try to force it with "-vd vaapih264dec"

look through your output from "uxplay -d 1" to see if it is being used.

[iesv]

Yes. I have already added this option, but the video is still dropping frames. :(

[iesv]

I copied part of the log.

---

log

raop_rtp_mirror accepting client

Received unencrypted codec packet from client: payload_size 39 header 01 00 16 01 ts_client = 3439.025500
raop_rtp_mirror: unidentified extra header data 0.000000, 0.000000
begin video stream wxh = 1920x1080; source 1920x1080
raop_rtp_mirror width_source = 1920.000000 height_source = 1080.000000 width = 1920.000000 height = 1080.000000
GStreamer h264 bus message pipeline2 new-clock
GStreamer h264 bus message videoscale0 state-changed PAUSED PLAYING
GStreamer h264 bus message videoconvert1 state-changed PAUSED PLAYING
GStreamer h264 bus message capsfilter0 state-changed PAUSED PLAYING
GStreamer h264 bus message videoconvert0 state-changed PAUSED PLAYING
GStreamer h264 bus message vaapidecode_h264-0 state-changed PAUSED PLAYING
GStreamer h264 bus message h264parse0 state-changed PAUSED PLAYING
GStreamer h264 bus message queue2 state-changed PAUSED PLAYING
GStreamer h264 bus message video_source state-changed PAUSED PLAYING
video_pipeline state change from PAUSED to PLAYING

raop_rtp_mirror: SPS+PPS header size = 6
raop_rtp_mirror h264 SPS+PPS header:
01 64 00 2a ff e1

raop_rtp_mirror SPS NAL size = 20
raop_rtp_mirror h264 Sequence Parameter Set:
27 64 00 2a ac 56 80 78 02 27 e5 9b 80 80 80 83
68 20 10 96

raop_rtp_mirror PPS NAL size = 4
raop_rtp_mirror h264 Picture Parameter Set:
28 ee 37 27

remainder size = 4
remainder of SPS+PPS packet:
fd f8 f8 00

raop_rtp_mirror SEI NAL size = 30
raop_rtp_mirror h264 Supplemental Enhancement Information:
06 05 1a 47 56 4a dc 5c 4c 43 3f 94 ef c5 11 3c
d1 43 a8 01 ff cc cc ff 02 00 7a 12 00 80

Begin streaming to GStreamer video pipeline
GStreamer h264 bus message h264parse0 latency
GStreamer h264 bus message vaapidecode_h264-0 latency
GStreamer h264 bus message kmssink_h264 tag
GStreamer h264 bus message kmssink_h264 state-changed READY PAUSED
GStreamer h264 bus message pipeline2 async-done
GStreamer h264 bus message kmssink_h264 latency
GStreamer h264 bus message kmssink_h264 state-changed PAUSED PLAYING
GStreamer h264 bus message pipeline2 state-changed PAUSED PLAYING
GStreamer h264 bus message kmssink_h264 qos
GStreamer h264 bus message kmssink_h264 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message kmssink_h264 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message kmssink_h264 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos

Received video streaming performance info packet from client: payload_size 194 header 05 00 00 00 ts_raw = 0
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
new request, connection 0, socket 35 type RAOP
httpd receiving on socket 35, connection 0

httpd: current connections:
connection 0 type 1 socket 35 conn 0x7fc744000b70 RAOP ACTIVE CONNECTION

httpd request received on socket 35, connection 0, method = POST, url = /feedback, protocol = RTSP/1.0
conn_request

POST /feedback RTSP/1.0
CSeq: 8
DACP-ID: DE64B73196819602
Active-Remote: 995029158
User-Agent: AirPlay/870.14.1

Handling request POST with URL /feedback
raop_handler_feedback

RTSP/1.0 200 OK
Audio-Jack-Status: connected; type=digital
Server: AirTunes/220.68
CSeq: 8

GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos

Received video streaming performance info packet from client: payload_size 376 header 05 00 00 00 ts_raw = 0
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos

raop_ntp send time type_t=82 packetlen = 32, now = 1757129313.639816
80 d2 00 07 00 00 00 00 83 aa 8b ee b8 7a a8 a8
ec 66 28 de a3 c3 61 bf ec 66 28 e1 a3 ca fe 92

raop_ntp receive time type_t=83 packetlen = 32, now = 1757129313.640345 t1 = 3441.721199, t2 = 3441.721237
80 d3 00 07 00 00 00 00 ec 66 28 e1 a3 ca fe 92
83 aa 8b f1 b8 a0 78 69 83 aa 8b f1 b8 a2 f8 0a

raop_ntp sync correction = -65778
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos

Received video streaming performance info packet from client: payload_size 368 header 05 00 00 00 ts_raw = 0
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos
GStreamer h264 bus message vaapidecode_h264-0 qos

[fduncanh]

on the screen you should also be seeing gstreamer messages with unix times at which the frames were dropped, in seconds.

see what the interval between dropped frames is. I had about .1 sec (10 frames dropped per sec) your earlier data (without vaapih264dec) ahowed about .05 sec. (20 frames dropped per sec), without dropping you would see 30 fps

[fduncanh]

you will probably bebetter off using a desktop environment and one of the other videosinks.

[iesv]

Yes, you’re right. Having a desktop environment would improve things, but this machine is a server, so I can’t install one.

Here’s my detailed log. It might be that the server’s hardware is too low, or this issue can’t be solved—after all, even you would experience frame drops. If it can’t be fixed, then let’s just close the issue.

Anyway, thank you very much for your patient explanation!

aaaaa.txt

[fduncanh]

not enough info. do it again, twice:

export GST_DEBUG=2
uxplay -vs kmssink -d 1 2>&1 | tee out1

uxplay -vs kmssink -vd vaapih264dec -d 1 2>&1 | tee out2

[iesv]

not enough info. do it again, twice:

export GST_DEBUG=2 uxplay -vs kmssink -d 1 2>&1 | tee out1

uxplay -vs kmssink -vd vaapih264dec -d 1 2>&1 | tee out2

out1.txt
out2.txt

[fduncanh]

those results are terrible, seems to be dropping almost every video frame.

maybe a network problem?

run debug again with -d (not -d1) to see each packet arriving and being processed by uxplay before it is sent to gstreamer.