blog/posts/2019-08-30-tale-of-working-with-uio.md

author	title	tags
Sanchayan Maity	A tale of working with Xilinx DisplayPort & UIO	linux, drivers, kernel, uio, interrupts, xilinx

At work, we use a DisplayPort IP from Xilinx. Xilinx does not provide any driver for this. There is a TX and RX. Baremetal code support is provided, however, we needed support for Linux. Ignoring interrupts, it is easy to get this baremetal code to work on Linux. Xilinx's baremetal code at it's core uses Xil_Out32 and Xil_In32 function for writing and reading to registers. The implementations for these can be replaced with mmap for accessing the registers. For DP TX side, we do not need to handle interrupts and setting up the registers is enough. For RX, however, we need interrupts to setup RX. For example, the link training for DP is initiated once a Training Pattern 1 (TP1) interrupt is detected.

Linux being a monolithic kernel, there is clear separation between kernel and user space. Interrupts can only be handled in kernel space. However, it was easier to use the ported baremetal code in user space and so we also needed to handle interrupts in user space. Generally, one writes a driver to do all this but since we only needed the interrupt part to be handled in kernel space, this is where UIO subsystem comes in.

Using the UIO subsystem, it is possible to handle the interrupts in kernel space while the rest like reading or writing to the registers can be done in user space. There is a Userspace I/O Platform driver with generic IRQ handling code. I will take the example of Xilinx DisplayPort RX here. After my colleague who works on FPGA side generates the FPGA firmware along with device trees which has entries as per the peripherals configured on FPGA side, DP RX peripheral let's say is in the memory region 0x80004000 to 0x80006000. An interrupt will also be assinged based on how the FPGA Programmable Logic (PL) connects to the Processing System (PS). PL is the FPGA and PS is the ARM64 SoC.

The extended device tree entry looks like this. reg specifies the memory that can be mmaped in user space and accessed while the interrupts will be used by the kernel code.

&SUBBLOCK_DP_BASE_v_dp_rxss1_0 {
    compatible = "dprxss-uio";
    interrupt-parent = <&gic>;
    interrupts = <0 92 4 0 92 4>;
    reg = <0x0 0x80004000 0x0 0x2000>
    status = "okay";
}

To link the UIO platform driver to this, we add the following to the bootargs environment variable in u-boot.

uio_pdrv_genirq.of_id=dprxss-uio

We need to do this, since the compatible property for device tree is not specified in the driver. See here. So it is a module parameter.

static struct of_device_id uio_of_genirq_match[] = {
	{ /* This is filled with module_parm */ },
	{ /* Sentinel */ },
};
MODULE_DEVICE_TABLE(of, uio_of_genirq_match);
module_param_string(of_id, uio_of_genirq_match[0].compatible, 128, 0);
MODULE_PARM_DESC(of_id, "Openfirmware id of the device to be handled by uio");

Now, a combination of poll and read can be used to wait for interrupts in user space.

So all well and good, however, there are some caveats to know. Once an interrupt is handled in kernel code, it disables the interrupt.

static irqreturn_t uio_pdrv_genirq_handler(int irq, struct uio_info *dev_info)
{
	struct uio_pdrv_genirq_platdata *priv = dev_info->priv;

	/* Just disable the interrupt in the interrupt controller, and
	 * remember the state so we can allow user space to enable it later.
	 */

	spin_lock(&priv->lock);
	if (!__test_and_set_bit(UIO_IRQ_DISABLED, &priv->flags))
		disable_irq_nosync(irq);
	spin_unlock(&priv->lock);

	return IRQ_HANDLED;
}

The interrupt re-enable logic is in the below function.

static int uio_pdrv_genirq_irqcontrol(struct uio_info *dev_info, s32 irq_on)
{
	struct uio_pdrv_genirq_platdata *priv = dev_info->priv;
	unsigned long flags;

	/* Allow user space to enable and disable the interrupt
	 * in the interrupt controller, but keep track of the
	 * state to prevent per-irq depth damage.
	 *
	 * Serialize this operation to support multiple tasks and concurrency
	 * with irq handler on SMP systems.
	 */

	spin_lock_irqsave(&priv->lock, flags);
	if (irq_on) {
		if (__test_and_clear_bit(UIO_IRQ_DISABLED, &priv->flags))
			enable_irq(dev_info->irq);
	} else {
		if (!__test_and_set_bit(UIO_IRQ_DISABLED, &priv->flags))
			disable_irq_nosync(dev_info->irq);
	}
	spin_unlock_irqrestore(&priv->lock, flags);

	return 0;
}

This is called from uio_write. And if you know how file operations work, this uio_write will be called when a write system call is issued with a file descriptor received from opening the /dev/uioX node.

static const struct file_operations uio_fops = {
	.owner		= THIS_MODULE,
	.open		= uio_open,
	.release	= uio_release,
	.read		= uio_read,
	.write		= uio_write,
	.mmap		= uio_mmap,
	.poll		= uio_poll,
	.fasync		= uio_fasync,
	.llseek		= noop_llseek,
};

Now, here comes the problem. For me, after the very first interrupt I was not getting any more interrupts. So, basically my write call was not working to re-enable the interrupt. Putting print statements in uio_write, I could see a call to the write function did not result in invocation of uio_write.

I was perplexed and wasted 4-5 hours trying to figure out what's wrong. I wrote a small piece of code outside the project workspace which opened /dev/uioX and then did a write. In this case, I could see my prints from uio_write function which eventually called uio_pdrv_genirq_irqcontrol to enable the interrupt. So, something was wrong with my project.

I use neovim and ctags for code navigation. Trying jump to definition on my write call made me end up in a write.c file. The very first initial project setup was done by my FPGA engineer colleague since Xilinx SDK generates bare metal code samples based on the design done. I had not noticed this file before. It seemed to be an artifact of the code ported over from baremetal and had a write function as below.

__attribute__((weak)) sint32 write(sint32 fd, char8 *buf, sint32 nbytes)

I was aware of the weak attribute from my work in u-boot where it is used to define board specific functions over riding default ones. The GCC manual defines it as "The weak attribute causes the declaration to be emitted as a weak symbol rather than a global. This is primarily useful in defining library functions which can be overridden in user code".

There was no other write function defined in my project. Ideally it should have been picked up from glibc. However, this was not what was happening. I did not need that write implementation in write.c which was actually writing to a UART port, so after removal everything started working fine. DP link training was succeeding finally.

You can read more about UIO here.