sanchayanmaity/blog
1
0
Fork 0
Code Issues Pull requests Packages Projects Releases Wiki Activity

posts: Add post on PipeWire compressed offload

Browse source
This commit is contained in:
Sanchayan Maity 2024-05-07 18:23:10 +05:30
parent 00299462e9
commit 00c0a27654
Signed by: sanchayanmaity
GPG key ID: 6F6A0609C12038F3
1 changed files with 295 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

295
posts/2024-03-18-pipewire-compressed-offload.md Normal file
View file

@ -0,0 +1,295 @@
---
author: Sanchayan Maity
title: Supporting ALSA compressed offload in PipeWire
tags: linux, pipewire
---
_Editor's note: this work was completed in late 2022 but this post was
unfortunately delayed._
Modern day audio hardware these days comes integrated with Digital Signal
Processors integrated in SoCs and audio codecs. Processing compressed or encoded
data in such DSPs results in power savings in comparison to carrying out such
processing on the CPU.
```
+---------+ +---------+ +---------+
| CPU | ---> | DSP | ---> | Codec |
| | <--- | | <--- | |
+---------+ +---------+ +---------+
```
This post takes a look at how all this works.
<!-- more -->
## Audio processing
A traditional audio pipeline, might look like below. An application reads encoded
audio and then might leverage a media framework like GStreamer or library like
ffmpeg to decode the encoded audio to PCM. The decoded audio stream is then handed
off to an audio server like PulseAudio or PipeWire which eventually hands it off
to ALSA.
```
+----------------+
| Application |
+----------------+
| mp3
+----------------+
| GStreamer |
+----------------+
| pcm
+----------------+
| PipeWire |
+----------------+
| pcm
+----------------+
| ALSA |
+----------------+
```
With ALSA Compressed offload, the same audio pipeline would look like this. The
encoded audio stream would be passed through to ALSA. ALSA would then, via it's
compressed offload API, send the encoded data to the DSP. DSP does the decode
and render.
```
+----------------+
| Application |
+----------------+
| mp3
+----------------+
| GStreamer |
+----------------+
| mp3
+----------------+
| PipeWire |
+----------------+
| mp3
+----------------+
| ALSA |
+----------------+
```
Since the processing of the compressed data is handed to a specialised hardware
namely the DSP, this results in a dramatic reduction of power consumption compared
to CPU based processing.
## Challenges
- ALSA Compressed Offload API which is a different API compared to the ALSA PCM
interface, provides the control and data streaming interface for audio DSPs.
This API is provided by the [tinycompress](https://github.com/alsa-project/tinycompress)
library.
- With PCM there is the notion of `bytes ~ time`. For example, 1920 bytes,
S16LE, 2 channels, 48 KHz would correspond to 10 ms. This breaks down
for compressed streams. It's impossible to estimate reliably the duration of
audio buffers when handling most compressed data.
- While sampling rate, number of channels and bits per sample are enough to
completely specify PCM, various parameters may have to be specified to enable
the DSP to deal with multiple compressed formats.
- For some codecs, additional firmware has to be loaded by the DSP. This has to
be handled outside the context of audio server.
## Requirements
- Expose all possible compressed formats.
- Allow a client to negotiate the format.
- Stream encoded audio frames and not PCM.
## PipeWire
PipeWire has become the default sound server on Linux, handling multimedia
routing and audio pipeline processing. It offers capture and playback for
both audio and video with minimal latency and support for PulseAudio, JACK,
ALSA, and GStreamer-based applications.
## SPA
PipeWire is built on top of SPA (Simple Plugin API), a header only API for
building plugins. SPA provides a set of low-level primitives.
SPA plugins are shared libraries (.so files) that can be loaded at runtime.
Each library provides one or more `factories`, each of which may implement
several `interfaces`.
The most interesting interface is the `node`.
- A node consumes or produces buffers through ports.
- In addition to ports and other well defined interface methods, a node can have
events and callbacks.
Ports are also first class objects within the node.
- There are a set of port related interface methods on the node.
- There may be statically allocated ports in instance initialization.
- There can be dynamic ports managed with `add_port` and `remove_port` methods.
- Ports have `params` which can be queried using the `port_enum_params` method
to determine the list of formats `EnumFormat`, the currently configured format
`Format`, buffer configuration, latency information, `I/O areas` for data structures
shared by port, and other such information.
- Some params such as the selected format can be set using the `port_set_format`
method.
## Implementing compressed sink SPA node
This section covers some primary implementation details of a PipeWire SPA node which
can accept an encoded audio stream and then write it out using ALSA compressed
offload API.
```c
static const struct spa_node_methods impl_node = {
SPA_VERSION_NODE_METHODS,
.add_listener = impl_node_add_listener,
.set_callbacks = impl_node_set_callbacks,
.enum_params = impl_node_enum_params,
.set_io = impl_node_set_io,
.send_command = impl_node_send_command,
.add_port = impl_node_add_port,
.remove_port = impl_node_remove_port,
.port_enum_params = impl_node_port_enum_params,
.port_set_param = impl_node_port_set_param,
.port_use_buffers = impl_node_port_use_buffers,
.port_set_io = impl_node_port_set_io,
.process = impl_node_process,
};
```
Some key node methods defining the actual implementation are as follows.
### **`port_enum_params`**
`params` for ports are queried using this method. This is akin to finding out
the capabilities of a port on the node.
For the compressed sink SPA node, the following are present.
- `EnumFormat`
This builds up a list of the encoded formats that's handled by the node to
return as a result.
- `Format`
Returns the currently set format on the port.
- `Buffers`
Provides information on size, minimum, and maximum number of buffers to be used
when streaming data to this node.
- `IO`
The node exchanges information via `IO` areas. There are various type of `IO`
areas like buffers, clock, position. Compressed sink SPA node only advertises
`buffer` areas at the moment.
The results are returned in an [SPA POD](https://docs.pipewire.org/page_spa_pod.html).
### **`port_use_buffers`**
Tells the port to use the given buffers via the `IO` area.
### **`port_set_param`**
The various `params` on the port are set via this method.
`Format` `param` request sets the actual encoded format that's going to be streamed
to this SPA node by a pipewire client like `pw-cat` or application for sending to
the DSP.
### **`process`**
Buffers containing the encoded media are handled here. The media stream is
written to the IO buffer area which were provided in `use_buffers`. The encoded
media stream is written to the DSP by calling `compress_write`.
### **`add_port`** and **`remove_port`**
Since dynamic ports aren't supported, these methods return a `ENOTSUP`.
## `pw-cat`
`pw-cat` was modified to support negotiation of encoded formats and passing the
encoded stream as is when linked to the compressed sink node.
## Deploying on hardware
Based on discussions with upstream compress offload maintainers, we chose a
Dragonboard 845c with the Qualcomm SDM845 SoC as our test platform.
For deploying Linux on Embedded devices, the tool of choice is Yocto.
Yocto is a build automation framework and cross-compile environment used to
create custom Linux distributions/board support packages for embedded devices.
Primary dependencies are
- tinycompress
- ffmpeg
- PipeWire
- WirePlumber
The `tinycompress` library is what provides the compressed offload API. It makes
`ioctl()` calls to the underlying kernel driver/sound subsystem.
`ffmpeg` is a dependency for the example `fcplay` utility provided by
`tinycompress`. It's also used in `pw-cat` to read basic metadata of the encoded
media. This is then used to determine and negotiate the format with the compressed
sink node.
`PipeWire` is where the compressed sink node would reside and `WirePlumber` acting
as the session manager for `PipeWire`.
Going into how Yocto works is beyond the scope of what can be covered in a blog
post. Basic Yocto project concepts can be found [here](https://docs.yoctoproject.org/overview-manual/concepts.html?highlight=meta+layer#yocto-project-concepts).
In Yocto speak, a custom
[meta layer](https://github.com/asymptotic-io/meta-asymptotic) was written.
Yocto makes it quite easy to build `autoconf` based projects. A new `tinycompress`
[bitbake recipe](https://github.com/asymptotic-io/meta-asymptotic/blob/master/recipes-multimedia/tinycompress/tinycompress.bb)
was written to build the latest sources from upstream and also include the
`fcplay` and `cplay` utilities for initial testing.
The existing PipeWire and WirePlumber recipes were modified to point to custom git
sources with minor changes to default settings included as part of the build.
## Updates since the original work
Since we completed the original patches, a number of changes have happened thanks
to the community (primarily Carlos Giani). These include:
- A device plugin for autodetecting compress nodes on the system
- Replacing `tinycompress` with an internal library to make all the requisite
`ioctl()`s
- Compressed format detection (which was previously waiting on
[an upstream API addition we implemented in `tinycompress`](https://github.com/alsa-project/tinycompress/pull/16)
## Future work
- Make compressed sink node provide clocking information. While the
API provides a method to retrieve the timestamp information, the relevant
timestamp fields seem to be not populated by the `q6asm-dai` driver.
- Validate other encoded formats. So far only MP3 and FLAC have been validated.
- May be the wider community can help test this on other hardware.
- Add capability to GStreamer plugin to work with compressed sink node. This
would also help in validating pause and resume.