345 lines
10 KiB
Markdown
345 lines
10 KiB
Markdown
---
|
|
title:
|
|
- Multimedia using Rust and GStreamer
|
|
author:
|
|
- Sanchayan Maity
|
|
aspectratio:
|
|
- 169
|
|
fontsize:
|
|
- 14pt
|
|
papersize:
|
|
- "presentation-16-9"
|
|
---
|
|
|
|
## Who?
|
|
|
|
- Consultant Software Engineer @ [asymptotic](https://asymptotic.io/)
|
|
- Open source consulting firm based out of Toronto, Bangalore & Hyderabad
|
|
- Work on low level systems software centred around multimedia
|
|
- GStreamer, PipeWire, PulseAudio
|
|
- Embedded Systems background
|
|
- C, Rust and Haskell
|
|
- Organizing Rust and Haskell meetup Bangalore since 2018
|
|
|
|
## Agenda
|
|
|
|
- Introduction to GStreamer
|
|
- Why Rust
|
|
- Rust and GStreamer
|
|
|
|
## GStreamer
|
|
|
|
- Multiplatform Pipeline based multimedia framework
|
|
- Bindings for various languages
|
|
- Supported on Linux, macOS, Android and Windows
|
|
- Allows building complex media processing workflows
|
|
- Some applications
|
|
* GstLAL (gravitational wave data analysis)
|
|
* PiTiVi (Video Editor)
|
|
* amaroK, Banshee, Clementine (audio players)
|
|
* Empathy (VOIP and video conferencing)
|
|
* Rygel (DLNA streaming server and renderer)
|
|
* Showtime, Clapper, Totem (Media players for desktop)
|
|
|
|
## Simple pipeline
|
|
|
|
```bash
|
|
gst-launch-1.0 videotestsrc ! autovideosink
|
|
gst-launch-1.0 audiotestsrc ! autoaudiosink
|
|
```
|
|
|
|
## **gst-inspect**
|
|
|
|
```bash
|
|
Factory Details:
|
|
Rank none (0)
|
|
Long-name Video test source
|
|
Klass Source/Video
|
|
Description Creates a test video stream
|
|
Author David A. Schleef <ds@schleef.org>
|
|
Documentation https://gstreamer.freedesktop.org/documentation/videotestsrc/#videotestsrc-page
|
|
|
|
Plugin Details:
|
|
Name videotestsrc
|
|
Description Creates a test video stream
|
|
Filename /usr/lib/gstreamer-1.0/libgstvideotestsrc.so
|
|
Version 1.24.9
|
|
License LGPL
|
|
Source module gst-plugins-base
|
|
Documentation https://gstreamer.freedesktop.org/documentation/videotestsrc/
|
|
Source release date 2024-10-30
|
|
Binary package Arch Linux GStreamer 1.24.9-3
|
|
Origin URL https://www.archlinux.org/
|
|
|
|
GObject
|
|
+----GInitiallyUnowned
|
|
+----GstObject
|
|
+----GstElement
|
|
+----GstBaseSrc
|
|
+----GstPushSrc
|
|
+----GstVideoTestSrc
|
|
|
|
Pad Templates:
|
|
SRC template: 'src'
|
|
Availability: Always
|
|
Capabilities:
|
|
video/x-raw
|
|
```
|
|
|
|
## Media pipeline
|
|
|
|
{width=80%}
|
|
|
|
|
|
## Why Rust?
|
|
|
|
- Codec implementations in pure Rust (Rust Audio, Xiph AV1, Symphonia)
|
|
- Things to care about
|
|
* **Low cognitive overhead**
|
|
* Immutability
|
|
* Expressive type system
|
|
* Memory safety and concurrency
|
|
* Foreign Function Interface
|
|
|
|
## Why Rust?
|
|
|
|
- Bindings/abstractions over GLib/GObject and for GStreamer[^1]
|
|
- Provides a root for the object hierarchy tree filed in by the GStreamer library
|
|
- Gives basic reference counting, parenting functionality and locking.
|
|
- GObject
|
|
- GstObject
|
|
- GstAllocator
|
|
- GstBufferPool
|
|
- GstBus
|
|
- GstClock
|
|
- GstDevice
|
|
- GstDeviceMonitor
|
|
- GstDeviceProvider
|
|
- GstElement
|
|
- GstPad
|
|
|
|
[^1]: [GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)
|
|
|
|
## Why immutability and types matter?
|
|
|
|
```c
|
|
let caps: gst::Caps = gst::Caps::builder("video/x-raw")
|
|
.field("width", crop_w)
|
|
.field("height", crop_h)
|
|
.field("pixel-aspect-ratio", gst::Fraction::new(1, 1))
|
|
.build();
|
|
let s = caps.remove_structure(0);
|
|
```
|
|
|
|
## Why immutability and types matter?
|
|
|
|
```bash
|
|
warning: unused variable: `s`
|
|
--> video-bin/src/imp.rs:152:13
|
|
|
|
|
152 | let s = caps.remove_structure(0);
|
|
| ^ help: if this is intentional, prefix it with an
|
|
underscore: `_s`
|
|
|
|
|
= note: `#[warn(unused_variables)]` on by default
|
|
error[E0596]: cannot borrow data in dereference of `gstreamer::Caps`
|
|
as mutable
|
|
--> video-bin/src/imp.rs:152:17
|
|
|
|
|
152 | let s = caps.remove_structure(0);
|
|
| ^^^^^^^^^^^^^^^^^^^^^^^^ cannot borrow as mutable
|
|
|
|
|
= help: trait `DerefMut` is required to modify through a dereference,
|
|
but it is not implemented for `gstreamer::Caps`
|
|
```
|
|
|
|
## Why immutability and types matter?
|
|
|
|
```c
|
|
let mut caps: gst::Caps = gst::Caps::builder("video/x-raw")
|
|
.field("width", crop_w)
|
|
.field("height", crop_h)
|
|
.field("pixel-aspect-ratio", gst::Fraction::new(1, 1))
|
|
.build();
|
|
let _s = caps.remove_structure(0);
|
|
```
|
|
|
|
|
|
## Why immutability and types matter?
|
|
|
|
```bash
|
|
warning: variable does not need to be mutable
|
|
--> video-bin/src/imp.rs:147:13
|
|
|
|
|
147 | let mut caps: gst::Caps = gst::Caps::builder("video/x-raw")
|
|
| ----^^^^
|
|
| |
|
|
| help: remove this `mut`
|
|
|
|
|
= note: `#[warn(unused_mut)]` on by default
|
|
error[E0596]: cannot borrow data in dereference of `gstreamer::Caps`
|
|
as mutable
|
|
--> video-bin/src/imp.rs:152:18
|
|
|
|
|
152 | let _s = caps.remove_structure(0);
|
|
| ^^^^^^^^^^^^^^^^^^^^^^^^ cannot borrow as mutable
|
|
= help: trait `DerefMut` is required to modify through a dereference,
|
|
but it is not implemented for `gstreamer::Caps`
|
|
```
|
|
|
|
## Why immutability and types matter?
|
|
|
|
```c
|
|
let caps: gst::Caps = gst::Caps::builder("video/x-raw")
|
|
.field("width", crop_w)
|
|
.field("height", crop_h)
|
|
.field("pixel-aspect-ratio", gst::Fraction::new(1, 1))
|
|
.build();
|
|
let caps = caps.get_mut().unwrap();
|
|
let _s = caps.remove_structure(0);
|
|
```
|
|
|
|
## Why immutability and types matter?
|
|
|
|
```bash
|
|
error[E0596]: cannot borrow `caps` as mutable, as it is not declared
|
|
as mutable
|
|
--> video-bin/src/imp.rs:152:20
|
|
|
|
|
147 | let caps: gst::Caps = gst::Caps::builder("video/x-raw")
|
|
| ---- help: consider changing this to be mutable:
|
|
`mut caps`
|
|
...
|
|
152 | let caps = caps.get_mut().unwrap();
|
|
| ^^^^^^^^^^^^^^ cannot borrow as mutable
|
|
|
|
For more information about this error, try `rustc --explain E0596`.
|
|
```
|
|
|
|
## Why immutability and types matter?
|
|
|
|
```c
|
|
let mut caps: gst::Caps = gst::Caps::builder("video/x-raw")
|
|
.field("width", crop_w)
|
|
.field("height", crop_h)
|
|
.field("pixel-aspect-ratio", gst::Fraction::new(1, 1))
|
|
.build();
|
|
if let Some(caps) = caps.get_mut() {
|
|
let _s = caps.remove_structure(0);
|
|
}
|
|
```
|
|
|
|
## Code
|
|
|
|
```rust
|
|
let src = gst::ElementFactory::make("filesrc")
|
|
.property("location", "sample.ogv")
|
|
.build()
|
|
.unwrap();
|
|
let demux = gst::ElementFactory::make("oggdemux").build().unwrap();
|
|
|
|
let pipeline_weak = pipeline.downgrade();
|
|
demux.connect("pad-added", false, move |args| {
|
|
let pipeline = match pipeline_weak.upgrade() {
|
|
Some(self_) => self_,
|
|
None => return None,
|
|
};
|
|
|
|
let pad = args[1]
|
|
.get::<gst::Pad>()
|
|
.expect("Second argument to demux pad-added must be pad");
|
|
```
|
|
|
|
## Code
|
|
|
|
```rust
|
|
if let Some(caps) = pad.current_caps() {
|
|
let s = caps.structure(0).unwrap();
|
|
|
|
let (decoder, sink) = if s.name().starts_with("video") {
|
|
let decoder = gst::ElementFactory::make("theoradec").build().unwrap();
|
|
let sink = gst::ElementFactory::make("autovideosink").build().unwrap();
|
|
(decoder, sink)
|
|
} else {
|
|
let decoder = gst::ElementFactory::make("vorbisdec").build().unwrap();
|
|
let sink = gst::ElementFactory::make("autoaudiosink").build().unwrap();
|
|
(decoder, sink)
|
|
};
|
|
|
|
let queue1 = gst::ElementFactory::make("queue").build().unwrap();
|
|
let queue2 = gst::ElementFactory::make("queue").build().unwrap();
|
|
```
|
|
|
|
## Code
|
|
|
|
```rust
|
|
pipeline
|
|
.add_many([&queue1, &decoder, &queue2, &sink])
|
|
.unwrap();
|
|
|
|
let sinkpad = queue1.static_pad("sink").unwrap();
|
|
pad.link(&sinkpad).unwrap();
|
|
|
|
queue1.link(&decoder).unwrap();
|
|
decoder.link(&queue2).unwrap();
|
|
queue2.link(&sink).unwrap();
|
|
|
|
queue1.sync_state_with_parent().unwrap();
|
|
decoder.sync_state_with_parent().unwrap();
|
|
queue2.sync_state_with_parent().unwrap();
|
|
sink.sync_state_with_parent().unwrap();
|
|
}
|
|
```
|
|
|
|
## Code
|
|
|
|
```rust
|
|
None
|
|
});
|
|
|
|
pipeline.add_many([&src, &demux]).unwrap();
|
|
|
|
src.link(&demux).unwrap();
|
|
```
|
|
|
|
## Media pipeline
|
|
|
|
|
|
|
|
## Some stats
|
|
|
|
- **gstreamer-rs** & **gst-plugins-rs**[^2]
|
|
- gstreamer-rs: ~2700 commits, gst-plugins-rs: ~2600 commits
|
|
- gstreamer-rs: ~85 contributors, gst-plugins-rs: ~110 contributors
|
|
- gst-plugins-rs: ~ +180k SLOC / -37k SLOC
|
|
- gst-plugins-rs: Overall 47 plugins, 149 elements
|
|
- In relation to the GStreamer monorepo
|
|
- 1.22 cycle: ~33% commits / MRs in Rust modules
|
|
- 1.24 cycle: ~25% commits / MRs in Rust modules
|
|
|
|
[^2]: [GStreamer & Rust: What has happened over the last 5 years](https://gstconf.ubicast.tv/videos/gstreamer-rust-what-has-happened-over-the-last-5-years_f8qxhpuzi9/)
|
|
|
|
## Resources
|
|
|
|
- [Dynamic Pipelines](https://gstreamer.freedesktop.org/documentation/tutorials/basic/dynamic-pipelines.html?gi-language=c)
|
|
- [GObject subclassing in Rust ](https://www.youtube.com/watch?v=TSf3rVyv7c8)
|
|
- [GStreamer bindings for Rust](https://gitlab.freedesktop.org/gstreamer/gstreamer-rs)
|
|
- [Rust GStreamer Plugins](https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs)
|
|
- [Using GStreamer](https://www.youtube.com/watch?v=ZphadMGufY8)
|
|
- [How to get started with GStreamer](https://www.youtube.com/watch?v=OkOsm9FyzdM&t=2s)
|
|
- [GStreamer for your backend services](https://asymptotic.io/blog/gstreamer-for-your-backend-services/)
|
|
- [OGG demultiplexing Rust sample code](https://gitlab.freedesktop.org/SanchayanMaity/ogg-demux)
|
|
|
|
## Questions?
|
|
|
|
- Rust Bangalore
|
|
* Meetup: [https://hasgeek.com/rustbangalore](https://hasgeek.com/rustbangalore)
|
|
* Telegram: [https://t.me/RustIndia](https://t.me/RustIndia)
|
|
- Reach out on
|
|
* email:
|
|
|
|
- me@sanchayanmaity.net
|
|
- sanchayan@asymptotic.io
|
|
- hello@asymptotic.io
|
|
* Mastodon: [sanchayanmaity.com](https://sanchayanmaity.com/@sanchayan)
|
|
* Blog: [sanchayanmaity.net](https://sanchayanmaity.net/)
|