notes/gstreamer/gstreamer.md

---
title: GStreamer
---

# Internals

## RTP timestamp

- Domain for `rtptime` is `clock-time` from SDP
- 48000 in this case
- So one tick is 1/48000 of a second
- Meaning 20 milliseconds is 960

## Discussion on threading internals

<http://gstreamer-devel.966125.n4.nabble.com/gstreamer-threading-internal-td4428624.html>

## Threads and probes

[Probes](https://gstreamer.freedesktop.org/documentation/additional/design/probes.html?gi-language=c)

[Threads](https://gstreamer.freedesktop.org/documentation/application-development/advanced/threads.html?gi-language=c)

## Audio decoders

There is `max-errors` property on audio decoder elements which can be helpful in debugging.

## Autoplugging

-   [Autoplugging](https://gstreamer.freedesktop.org/documentation/application-development/advanced/autoplugging.html?gi-language=c)

-   [Playback Components](https://gstreamer.freedesktop.org/documentation/application-development/highlevel/playback-components.html?gi-language=c)

-   [Decodebin Design](https://gstreamer.freedesktop.org/documentation/additional/design/decodebin.html?gi-language=c)

## Why are queues required

-   [Demuxing mp4 and queues](http://gstreamer-devel.966125.n4.nabble.com/demuxing-mp4-and-queues-td4689905.html)

This is a classical problem, and a good illustration why you usually need queues after elements with N src pads, and before elements with N sink pads.

The demuxer is pushing to its source pads from the same thread, and if you have connected these source pads with sync=true sinks, their chain function blocks until prerolling is done, that is until each sink with sync=true has received a buffer. Adding queues decouples branches of the pipeline, as they will start their own streaming thread.

## Dynamically remove or unlink elements

- http://gstreamer-devel.966125.n4.nabble.com/Dynamically-adding-and-removing-branches-of-a-tee-td973635.html

- https://gstreamer.freedesktop.org/documentation/application-development/advanced/pipeline-manipulation.html?gi-language=c

See probes here.

## Webkit debugging

- https://eocanha.org/blog/2021/05/25/gstreamer-webkit-debugging-by-using-external-tools-2-2/

## Quark

Quarks represent a different idea Quarks are basically an internal "registry" of strings. The idea is that if you want to compare strings often, that can be inefficient, so you make a GQuark of it (essentially a hash) that you can stably compare. So if there is a string comparison you are performing often, or you want to make faster, you convert the string to a quark (hashing it to an integer), and then use that instead.

## Flushing probe

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/274

# Elements

## GstDiscoverer

- Internally uses `uridecodebin`. This element decodes data from a URI into raw media. It selects a source element that can handle the given URI scheme and connects it to a `decodebin` element. It acts like a demuxer, so it offers as many source pads as streams are found in the media.

    ``` bash
    gst-launch-1.0 uridecodebin uri=https://www.freedesktop.org/software/gstreamer-sdk/data/media/sintel_trailer-480p.webm ! videoconvert ! autovideosink
    ```

- Primary function seems to be `discoverer_collect` which gets called pipeline is pre-rolled

- The next important function is `parse_stream_topology`. It can also be called recursively

    - Checks for `next` in `GstStructure` passed as argument to this function

    - If `next` is `NULL`, then it's the end of the stream, else one has a `GstValue` holding either a structure or a list

    - For the case of a `GstStructure`, `collect_information` is then called. It reads various fields in the caps to collect information like `rate`, `channels` and `channel-mask`. Something called `TAGS` and `QUARK` also come into picture here. Quarks seem to be a 2 way association between a string and a unique identifier

    - For the case of list, along with some things, walk over the list, calling `parse_stream_topology` for each item in the list, where the item seems to a sub stream

### Decodebin

This element automatically constructs a decoding pipeline using available decoders and demuxers via auto-plugging until raw media is obtained. It is used internally by `uridecodebin` which is often more convenient to use, as it creates a suitable source element as well. It replaces the old `decodebin` element. It acts like a demuxer, so it offers as many source pads as streams are found in the media.

    ``` bash
    gst-launch-1.0 souphttpsrc location=https://www.freedesktop.org/software/gstreamer-sdk/data/media/sintel_trailer-480p.webm ! decodebin ! autovideosink
    ```

### Thumbnails

Can `GstDiscoverer` be used for thumbnails? Googling a bit, and we land on this issue [discoverer should generated video/image thumbnails](https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/issues/58). This references [Thumbnailer Spec](https://specifications.freedesktop.org/thumbnail-spec/thumbnail-spec-latest.html) and [Thumbnail Management DBus Specification](https://wiki.gnome.org/action/show/DraftSpecs/ThumbnailerSpec?action=show&redirect=ThumbnailerSpec). `GstDiscoverer` does not seem to have a thumbnailing API.

There is a totem video thumbnailer which is the video thumbnailer for the **GNOME** desktop but seems not to be available as a separate library. [ffmpegthumbnailer](https://github.com/dirkvdb/ffmpegthumbnailer) based on ffmpeg is a lightweight video thumbnailer that can be used by file managers to create thumbnails for video files. However, packaging `ffmpeg` might be a concern here and `ffmpegthumbnailer` has a *GPL-2.0* license. FFmpeg is licensed under the LGPL license, however, if a particular build of FFmpeg is linked against any GPL libraries (notably x264), then the entire binary is licensed under the GPL.

If this needs to be provided with GStreamer then `gstbasesink` has a note on `last-sample` property where it mentions that the property can be used to generate thumbnails. Also see `gst_base_sink_get_last_sample`. A related option seems to be `convertframe` which has two APIs for converting a raw video buffer into the specified output caps. See [convertframe](https://gstreamer.freedesktop.org/documentation/video/convertframe.html?gi-language=c#).

Some links which have examples.

- https://github.com/GStreamer/gst-editing-services/blob/master/examples/c/thumbnails.c
- https://stackoverflow.com/questions/15789652/how-to-create-video-thumbnails-with-python-and-gstreamer
- https://gist.github.com/dplanella/5563018

### Tracker miner

`GstDiscoverer` is used in Tracker Miner though not for everything. From [Into the Pyramid](https://samthursfield.wordpress.com/2019/12/01/into-the-pyramid/).

- Tracker has a ‘generic image extraction’ rule that tries to find metadata for any image/\* MIME type that isn’t a .bmp, .jpg, .gif, or .png. This codepath uses the GstDiscoverer API, the same as for video and audio files, in the hope that a GStreamer plugin on the system can give us useful info about the image.

- The GstDiscoverer instance is created with a timeout of 5 seconds. (This seems quite high — the gst-typefind utility that ships with GStreamer uses a timeout of 1 second).

- GStreamer’s GstDiscoverer API feeds any file where the type is unknown into an MPEG decoder, which is effectively an unwanted fuzz test and can trigger periods of high CPU and memory usage.

- 5 seconds of processing non-MPEG data with an MPEG decoder is somehow enough to cause Linux’s scheduler to lock up the entire system.

## Rank

[Modify the elements rank](https://developer.ridgerun.com/wiki/index.php?title=GStreamer_modify_the_elements_rank)

Each GStreamer element has a detail called rank that defines the priority used by the autoplugger when it wants to connect a new element but has multiple options.

# Pipeline examples

## Opus RTP encoding decoding pipeline

Note that in place of rtpjitterbuffer, rtpbin also works.

``` bash
export AUDIO_CAPS="application/x-rtp,media=(string)audio,clock-rate=(int)48000,encoding-name=(string)OPUS"
gst-launch-1.0 -v audiotestsrc ! audioconvert ! audioresample ! opusenc ! rtpopuspay ! udpsink host=localhost port=50004
gst-launch-1.0 udpsrc caps=$AUDIO_CAPS address=localhost port=50004 ! rtpjitterbuffer latency=20 do-lost=true ! rtpopusdepay ! opusdec plc=true ! pulsesink
```

## Tearing down a branch of *tee*

``` python
tee = pipe.get_by_name(RTMP_TEE)
srcPad = tee.get_static_pad("src_" + str(idx))
srcPad.add_probe(
    Gst.PadProbeType.BLOCK, rtmp_tee_srcpad_block_probe, rtmpUrl
)

def rtmp_tee_srcpad_block_probe(
    pad: Gst.Pad, info: Gst.PadProbeInfo, rtmpUrl
) -> Gst.PadProbeReturn:
    tee = pipe.get_by_name(RTMP_TEE)
    queue = pipe.get_by_name(MUX_OUTPUT_QUEUE + "-" + rtmpUrl)
    rtmpSink = pipe.get_by_name(RTMP_SINK + "-" + rtmpUrl)
    sinkPad = queue.get_static_pad("sink")
    pad.unlink(sinkPad)
    rtmpSink.set_state(Gst.State.NULL)
    queue.set_state(Gst.State.NULL)
    tee.remove_pad(pad)
    pipe.remove(queue)
    pipe.remove(rtmpSink)
    return Gst.PadProbeReturn.REMOVE
```

## Check camera capabilities

``` bash
gst-device-monitor-1.0 Video/Source
v4l2-ctl --list-formats-ext -d0
```

# Annotations

[Annotation Glossary](https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html//annotation-glossary.html)

# AAC test samples

- https://samples.mplayerhq.hu/A-codecs/AAC/

# Debugger

If `SIGSEGV` occurs, to use `gdb` and `gst-launch` to debug the crashed process, first run

``` bash
echo 0 > /proc/sys/kernel/yama/ptrace_scope
echo -1 > /proc/sys/kernel/perf_event_paranoid
```

- Arch enables the Yama LSM by default, which provides a `kernel.yama.ptrace_scope` kernel parameter. This parameter is set to 1 (restricted) by default which prevents tracers from performing a ptrace call on traces outside of a restricted scope unless the tracer is privileged or has the `CAP_SYS_PTRACE` capability.


# Terminology

- **Transmuxing**, also referred to as repackaging or packetizing, is a process in which audio and video files are repackaged into different delivery formats without changing the files' contents.

# Yocto

[Enable RTMP plugin](https://www.toradex.cn/community/questions/13303/gstreamer-rtmp-plugin.html)

# RTP

[Some FAQs about RTP](https://www.cs.columbia.edu/~hgs/rtp/faq.html)

# Basics of time and synchronization

- https://www.youtube.com/watch?v=EOHl1ktWwp4

- https://www.youtube.com/watch?v=OKTfNAmFTo4

-   `running_time` is the amount of time "spent" in playing

-   Playing => Paused: Remember running time

-   Paused => Playing: base_time = current absolutetime - running_time

# Live pipeline

## Live sources

- https://gstreamer.freedesktop.org/documentation/additional/design/live-source.html?gi-language=c

- https://gstreamer.freedesktop.org/documentation/additional/design/element-source.html?gi-language=c

-https://gstreamer.freedesktop.org/documentation/base/gstbasesrc.html?gi-language=c

## Pipeline to check base time gets updated after PAUSE -> PLAY transition

``` bash
env GST_DEBUG=basesink:6 gst-play-1.0 /usr/share/sounds/alsa/Front_Left.wav --audiosink="fakesink sync=true" -q 2>&1 | grep base_time
```

# Running test suite with gdb

``` bash
ninja -C build && env CK_FORK=no meson test --gdb -C build --suite gst-plugins-good elements_splitmuxsink
```

# Re-timestamping frames

- https://kaustav.space/2018-09-20/retimestamping-frames-in-gstreamer

# H264

- https://doc-kurento.readthedocs.io/en/latest/knowledge/h264.html

# Video mp4

- https://bitmovin.com/container-formats-fun-1
- https://docs.unified-streaming.com/documentation/live/subtitles.html#embedded-subtitles-captions
- https://stackoverflow.com/questions/24884827/possible-locations-for-sequence-picture-parameter-sets-for-h-264-stream
- https://stackoverflow.com/questions/10185474/gstreamer-how-to-get-audio-and-video-to-play-at-the-same-rate
- https://bitmovin.com/fun-with-container-formats-2/
- https://www.ramugedia.com/mp4-container
- https://gpac.github.io/mp4box.js/test/filereader.html
- https://en.wikipedia.org/wiki/Video_compression_picture_types
- https://en.wikipedia.org/wiki/Group_of_pictures
- https://observablehq.com/@benjamintoofer/iso-base-media-file-format
- https://streaminglearningcenter.com/blogs/open-and-closed-gops-all-you-need-to-know.html