presentations/rusttechx-2024/rusttechx.md

---
title:
- Multimedia using Rust and GStreamer
author:
- Sanchayan Maity
aspectratio:
- 169
fontsize:
- 14pt
papersize:
- "presentation-16-9"
---

## Who?

- Consultant Software Engineer @ [asymptotic](https://asymptotic.io/)
    - Open source consulting firm based out of Toronto, Bangalore & Hyderabad
    - Work on low level systems software centred around multimedia
    - GStreamer, PipeWire, PulseAudio
- Embedded Systems background
- C, Rust and Haskell
- Organizing Rust and Haskell meetup Bangalore since 2018

## Agenda

- Introduction to GStreamer
- Why Rust
- Rust and GStreamer

## GStreamer

- Multiplatform Pipeline based multimedia framework
- Bindings for various languages
- Supported on Linux, macOS, Android and Windows
- Allows building complex media processing workflows
- Some applications
  * GstLAL (gravitational wave data analysis)
  * PiTiVi (Video Editor)
  * amaroK, Banshee, Clementine (audio players)
  * Empathy (VOIP and video conferencing)
  * Rygel (DLNA streaming server and renderer)
  * Showtime, Clapper, Totem (Media players for desktop)

## Simple pipeline

```bash
gst-launch-1.0 videotestsrc ! autovideosink
gst-launch-1.0 audiotestsrc ! autoaudiosink
  ```

## **gst-inspect**

```bash
Factory Details:
  Rank                     none (0)
  Long-name                Video test source
  Klass                    Source/Video
  Description              Creates a test video stream
  Author                   David A. Schleef <ds@schleef.org>
  Documentation            https://gstreamer.freedesktop.org/documentation/videotestsrc/#videotestsrc-page

Plugin Details:
  Name                     videotestsrc
  Description              Creates a test video stream
  Filename                 /usr/lib/gstreamer-1.0/libgstvideotestsrc.so
  Version                  1.24.9
  License                  LGPL
  Source module            gst-plugins-base
  Documentation            https://gstreamer.freedesktop.org/documentation/videotestsrc/
  Source release date      2024-10-30
  Binary package           Arch Linux GStreamer 1.24.9-3
  Origin URL               https://www.archlinux.org/

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseSrc
                         +----GstPushSrc
                               +----GstVideoTestSrc

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw
```

## Media pipeline

![*_Simple Player_*](simple-player.png){width=80%}


## Why Rust?

- Codec implementations in pure Rust (Rust Audio, Xiph AV1, Symphonia)
- Things to care about
  * **Low cognitive overhead**
  * Immutability
  * Expressive type system
  * Memory safety and concurrency
  * Foreign Function Interface

## Why Rust?

- Bindings/abstractions over GLib/GObject and for GStreamer[^1]
- Provides a root for the object hierarchy tree filed in by the GStreamer library
- Gives basic reference counting, parenting functionality and locking.
- GObject
    - GstObject
        - GstAllocator
        - GstBufferPool
        - GstBus
        - GstClock
        - GstDevice
        - GstDeviceMonitor
        - GstDeviceProvider
        - GstElement
        - GstPad

[^1]: [GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)

## Why immutability and types matter?

```c
      let caps: gst::Caps = gst::Caps::builder("video/x-raw")
          .field("width", crop_w)
          .field("height", crop_h)
          .field("pixel-aspect-ratio", gst::Fraction::new(1, 1))
          .build();
      let s = caps.remove_structure(0);
```

## Why immutability and types matter?

```bash
warning: unused variable: `s`
   --> video-bin/src/imp.rs:152:13
    |
152 |         let s = caps.remove_structure(0);
    |             ^ help: if this is intentional, prefix it with an
                    underscore: `_s`
    |
    = note: `#[warn(unused_variables)]` on by default
error[E0596]: cannot borrow data in dereference of `gstreamer::Caps`
              as mutable
   --> video-bin/src/imp.rs:152:17
    |
152 |         let s = caps.remove_structure(0);
    |                 ^^^^^^^^^^^^^^^^^^^^^^^^ cannot borrow as mutable
    |
    = help: trait `DerefMut` is required to modify through a dereference,
            but it is not implemented for `gstreamer::Caps`
```

## Why immutability and types matter?

```c
        let mut caps: gst::Caps = gst::Caps::builder("video/x-raw")
            .field("width", crop_w)
            .field("height", crop_h)
            .field("pixel-aspect-ratio", gst::Fraction::new(1, 1))
            .build();
        let _s = caps.remove_structure(0);
```


## Why immutability and types matter?

```bash
warning: variable does not need to be mutable
   --> video-bin/src/imp.rs:147:13
    |
147 |         let mut caps: gst::Caps = gst::Caps::builder("video/x-raw")
    |             ----^^^^
    |             |
    |             help: remove this `mut`
    |
    = note: `#[warn(unused_mut)]` on by default
error[E0596]: cannot borrow data in dereference of `gstreamer::Caps`
              as mutable
   --> video-bin/src/imp.rs:152:18
    |
152 |         let _s = caps.remove_structure(0);
    |                  ^^^^^^^^^^^^^^^^^^^^^^^^ cannot borrow as mutable
    = help: trait `DerefMut` is required to modify through a dereference,
            but it is not implemented for `gstreamer::Caps`
```

## Why immutability and types matter?

```c
        let caps: gst::Caps = gst::Caps::builder("video/x-raw")
            .field("width", crop_w)
            .field("height", crop_h)
            .field("pixel-aspect-ratio", gst::Fraction::new(1, 1))
            .build();
        let caps = caps.get_mut().unwrap();
        let _s = caps.remove_structure(0);
```

## Why immutability and types matter?

```bash
error[E0596]: cannot borrow `caps` as mutable, as it is not declared
              as mutable
   --> video-bin/src/imp.rs:152:20
    |
147 |         let caps: gst::Caps = gst::Caps::builder("video/x-raw")
    |             ---- help: consider changing this to be mutable:
                      `mut caps`
...
152 |         let caps = caps.get_mut().unwrap();
    |                    ^^^^^^^^^^^^^^ cannot borrow as mutable

For more information about this error, try `rustc --explain E0596`.
```

## Why immutability and types matter?

```c
        let mut caps: gst::Caps = gst::Caps::builder("video/x-raw")
            .field("width", crop_w)
            .field("height", crop_h)
            .field("pixel-aspect-ratio", gst::Fraction::new(1, 1))
            .build();
        if let Some(caps) = caps.get_mut() {
            let _s = caps.remove_structure(0);
        }
```

## Code

```rust
    let src = gst::ElementFactory::make("filesrc")
        .property("location", "sample.ogv")
        .build()
        .unwrap();
    let demux = gst::ElementFactory::make("oggdemux").build().unwrap();

    let pipeline_weak = pipeline.downgrade();
    demux.connect("pad-added", false, move |args| {
        let pipeline = match pipeline_weak.upgrade() {
            Some(self_) => self_,
            None => return None,
        };

        let pad = args[1]
            .get::<gst::Pad>()
            .expect("Second argument to demux pad-added must be pad");
```

## Code

```rust
        if let Some(caps) = pad.current_caps() {
            let s = caps.structure(0).unwrap();

            let (decoder, sink) = if s.name().starts_with("video") {
                let decoder = gst::ElementFactory::make("theoradec").build().unwrap();
                let sink = gst::ElementFactory::make("autovideosink").build().unwrap();
                (decoder, sink)
            } else {
                let decoder = gst::ElementFactory::make("vorbisdec").build().unwrap();
                let sink = gst::ElementFactory::make("autoaudiosink").build().unwrap();
                (decoder, sink)
            };

            let queue1 = gst::ElementFactory::make("queue").build().unwrap();
            let queue2 = gst::ElementFactory::make("queue").build().unwrap();
```

## Code

```rust
            pipeline
                .add_many([&queue1, &decoder, &queue2, &sink])
                .unwrap();

            let sinkpad = queue1.static_pad("sink").unwrap();
            pad.link(&sinkpad).unwrap();

            queue1.link(&decoder).unwrap();
            decoder.link(&queue2).unwrap();
            queue2.link(&sink).unwrap();

            queue1.sync_state_with_parent().unwrap();
            decoder.sync_state_with_parent().unwrap();
            queue2.sync_state_with_parent().unwrap();
            sink.sync_state_with_parent().unwrap();
        }
```

## Code

```rust
        None
    });

    pipeline.add_many([&src, &demux]).unwrap();

    src.link(&demux).unwrap();
```

## Media pipeline

![*_OGG_Demux_*](ogg-demux-pipeline.svg)

## Some stats

- **gstreamer-rs** & **gst-plugins-rs**[^2]
    - gstreamer-rs: ~2700 commits, gst-plugins-rs: ~2600 commits
    - gstreamer-rs: ~85 contributors, gst-plugins-rs: ~110 contributors
    - gst-plugins-rs: ~ +180k SLOC / -37k SLOC
    - gst-plugins-rs: Overall 47 plugins, 149 elements
- In relation to the GStreamer monorepo
    - 1.22 cycle: ~33% commits / MRs in Rust modules
    - 1.24 cycle: ~25% commits / MRs in Rust modules

[^2]: [GStreamer & Rust: What has happened over the last 5 years](https://gstconf.ubicast.tv/videos/gstreamer-rust-what-has-happened-over-the-last-5-years_f8qxhpuzi9/)

## Resources

- [Dynamic Pipelines](https://gstreamer.freedesktop.org/documentation/tutorials/basic/dynamic-pipelines.html?gi-language=c)
- [GObject subclassing in Rust ](https://www.youtube.com/watch?v=TSf3rVyv7c8)
- [GStreamer bindings for Rust](https://gitlab.freedesktop.org/gstreamer/gstreamer-rs)
- [Rust GStreamer Plugins](https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs)
- [Using GStreamer](https://www.youtube.com/watch?v=ZphadMGufY8)
- [How to get started with GStreamer](https://www.youtube.com/watch?v=OkOsm9FyzdM&t=2s)
- [GStreamer for your backend services](https://asymptotic.io/blog/gstreamer-for-your-backend-services/)
- [OGG demultiplexing Rust sample code](https://gitlab.freedesktop.org/SanchayanMaity/ogg-demux)

## Questions?

- Rust Bangalore
  * Meetup: [https://hasgeek.com/rustbangalore](https://hasgeek.com/rustbangalore)
  * Telegram: [https://t.me/RustIndia](https://t.me/RustIndia)
- Reach out on
  * email:

        - me@sanchayanmaity.net
        - sanchayan@asymptotic.io
        - hello@asymptotic.io
  * Mastodon: [sanchayanmaity.com](https://sanchayanmaity.com/@sanchayan)
  * Blog: [sanchayanmaity.net](https://sanchayanmaity.net/)