Linux Real-Time Kernel performance requirements for AVB devices
Abstract
AVB endpoint devices are complex real-time embedded systems requiring stringent real-time deadlines to be met. Sienda usually recommends that AVB/Milan endpoint devices are implemented using a dedicated System-on-Chip (SoC) and a real-time operating system (RTOS). Modern multicore Intel and ARM SoCs however lack RTOS support, and recent improvements to PREEMPT_RT linux kernels mean that consideration must be given to the possibility of running an AVB endpoint device on Linux.
This document lists the requirements that an AVB implementation places on a Linux system, and documents how core performance indicators can be obtained from a running Linux system.
Real-Time Kernel
The first absolute requirement is a well-performing real-time linux kernel. PREEMPT_RT patched kernels are the only kernels that Sienda have had success using, as no other kernels (including low-latency and IoT kernels) have low enough interrupt latencies, wake up and context switch times.
On Intel hardware, the ‘Intel optimised real-time kernel’ from Canonical is the only kernel proven to meet these requirements. The generic Ubuntu real-time kernel, and the ClearLinux real-time kernel, while performing admirably for many use cases, do not fulfil the strict requirements for an AVB device.
Measuring interrupt latency with cyclictest
Once you have a PREEMPT_RT kernel installed and running, you can run the linux tool cyclictest to obtain a good idea of the worst case interrupt latency. This tool uses the userspace clock_nanosleep() function to make a call in the kernel (using a hr_timer) to put the thread to sleep until a specified time. When the hr_timer fires, the kernel wakes the sleeping thread and cyclictest compares the time it requested to go to sleep with the time it got to run after awakening.
Run cyclic test on all available cores with the following command:
sudo cyclictest --smp --mlockall --interval=125 --priority=99 --clock=1
This runs cyclictest at a real-time priority, with a wakeup interval of 125uS which matches the typical AVB device ‘tick’ period.
A typical output would look like this:
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 1.61 0.69 0.52 1/229 1435
T: 0 ( 1430) P:99 I:125 C: 233602 Min: 2 Act: 3 Avg: 2 Max: 31
T: 1 ( 1431) P:99 I:625 C: 46715 Min: 3 Act: 4 Avg: 3 Max: 12
T: 2 ( 1432) P:99 I:1125 C: 25949 Min: 3 Act: 4 Avg: 3 Max: 14
We are only really interested in the max times. If your cyclictest max times are showing under 10uS then you have an excellent system which could work well for AVB. If your cyclictest times are over 50uS then the system is likely not suitable, although there are several tuning options that may allow the system to still be used for AVB.
isolcpus
The linux kernel option isolcpus allows specified processor cores to be 'isolated', meaning they wont be used by the linux scheduler for running any threads. This effectively reserves the core for your real-time application. Doing so can reduce max latencies to below those reported by cyclictest. Unfortunately, cyclictest wont run on isolated cores, but we have a small test app linuxEthernetTest that will run on isolated cores, and that will also test the latency through the linux network stack and ethernet driver, which is also hugely important for AVB.
Sienda linuxEthernetTest application
This small command line application works in a similar way to cyclictest except that it can run on an isolated core and it also tests the latency of the ethernet subsystem. It also tests a mechanism for allowing for variance in the wakeup latency of your kernel by requesting to be woken up earlier than actually required, and then spinning until the actual desired wakeup time. This mechanism is not recommended as it burns CPU cycles and hammers the kernel's timer functions, but it can be used in some scenarios where the kernel is not quite as low-latency as we require.
Run the tool as such:
sudo ./linuxEthernetTest t enp0s29f1 10
The t parameter specifies that the tool is running as a 'talker' (sending packets). Replace enp0s29f1 with your own ethernet device name. The 10 specifies that the test will run for 10 seconds.
The output:
Raised to realtime priority
Locked paging
Enabled low latency operation
Talker using wakeup offset of 0 uS
Running talker for 10 seconds (80000 packets)...
using interface enp0s29f1 with MAC address 30643e54ee
Talker finished.
Late packets = 0
Max delta = 35454 ns
Here we can see that the Max delta value is 35454 (~35uS). This is the worst case wake up latency during the test run. Not great. However, we have not specified a core to pin the thread to, so this has been scheduled on one of the non-isolated cores, which will be shared with other processes, kernel thread and interrupt handlers in the system. To pin the test app to core 3 (assuming that core 3 has been isolated with isolcpus) add an extra parameter:
sudo ./linuxEthernetTest t enp0s29f1 10 3
Raised to realtime priority
Pinned thread to CPU 3
Locked paging
Enabled low latency operation
Talker using wakeup offset of 0 uS
Running talker for 10 seconds (80000 packets)...
using interface enp0s29f1 with MAC address 30643e54ee
Talker finished.
Late packets = 0
Max delta = 7323 ns
Here we can see that the thread is now pinned to core 3, and the Max delta has dropped to around 7uS. Much better. If a system can constantly achieve a max latency (Max delta) of under 10uS then it would be a good candidate for an AVB device. Run linuxEthernetTest for an extended period of time (1h, 1 day, as long as you can...) until you are satisfied that you have the true max latency figure.
If your max latency value is still too high (or you would simply like to try to minimise it further) then you can experiment with the wakeupOffset parameter of the test application. This offset causes the wakeup to happen a little earlier than required, to counter the worst case latency, and allows the app to spin for the last microseconds until the exact required run-time. Take your worst case latency figure from the previous run (rounded up to the nearest microsecond or so), and enter it as an extra parameter:
sudo ./linuxEthernetTest t enp0s29f1 10 3 8
Raised to realtime priority
Pinned thread to CPU 3
Locked paging
Enabled low latency operation
Talker using wakeup offset of 8 uS
Running talker for 10 seconds (80000 packets)...
using interface enp0s29f1 with MAC address 30643e54ee
Talker finished.
Late packets = 0
Max delta = 76 ns
Here the wakeupOffset value is set to 8 microseconds, so the thread will be woken up 8uS earlier than required, but considering the worst case latency of 8 microseconds this means that the thread should always be woken up before or at the required time. The thread will then spin until the actual required time. The Max delta is now 76ns, which is exceptional, and is simply the time it takes to call into the kernel to retrieve the time (rounded to a multiple of the timer resolution). If this system can actually dispatch AVB audio packets to the ethernet subsystem with a jitter or only 76ns then it could make an excellent AVB platform.
To determine if the system is capable of egressing ethernet packets within the requirements of 802.1Qav, we now have to capture the actual ethernet packets on the wire, using an accurate capture tap such as a Profishark. (Please note, wireshark running on another computer will most likely NOT be suitable, as the captured timestamps will not be accurate enough). Having taken a capture from a Profishark we can see the dispatch of packets from the application, 8000 per second:
The time between packets looks pretty good. There is less than a microsecond jitter between most packets. If we sort the capture by delta_time then we can check the worst (largest) delay between packets:
The 'worst' (latest) packet was 134uS after the previous packet (ideally it would be 125uS assuming the previous packet was on time). This means that it was held up in the linux network stack and/or ethernet driver for around 9uS. Whilst not ideal, this is certainly tolerable. If this platform were to be used for a real AVB device, the packets would also be subject to priority queuing and 802.1Qav traffic shaping in hardware (using the Credit-Based-Shaper), which would ensure that the packet spacing on the ethernet link would conform to 802.1Qav. The purpose of this whole exercise is to ensure that the linux platform is able to dispatch packets TO the ethernet AVB hardware within acceptable timing bounds.
Having seen that packets can egress to satisfy the requirements of a talker device it is necessary to check that listener packets can ingress to the point of processing quickly enough. The test application is now run on two separate devices which are connected together by ethernet (direct cable connection). On the listener device we run the command:
sudo ./linuxEthernetTest l enp0s29f1 10 3
and on the talker device we run the command as before:
sudo ./linuxEthernetTest t enp0s29f1 10 3 8
Raised to realtime priority
Pinned thread to CPU 3
Locked paging
Enabled low latency operation
Talker using wakeup offset of 8 uS
Running talker for 10 seconds (80000 packets)...
using interface enp0s29f1 with MAC address 30643e54ee
Talker finished.
Late packets = 0
Max delta = 73 ns
The listener will have now output a Max delta figure for the packet reception:
Raised to realtime priority
Pinned thread to CPU 3
Locked paging
Enabled low latency operation
Running listener for 80000 packets (10 seconds)...
using interface enp0s29f1 with MAC address 30643e57f8
Listener finished.
Late packets = 3
Max delta = 197201 ns
The Max delta figure on the listener side is the max time between two consecutive test packets, so the target value is 125000ns (125uS). In the example above the Max delta is 197uS, so some packets are being received and processed by the listener 72uS 'late'. This is not a great result, but is also not fatal, as a listener device can compensate for addition ingress latency by increasing the overall latency of the device. In the example above, if the AVB listener device was expected to play out the audio within a few microseconds of the packet arriving on the wire, then this 72uS ingress latency would be catastrophic. But if the listener device contains sufficient audio output buffering to delay all output by 100uS or so, then these 'late' ingress packets can be tolerated.
Ethernet coalesce parameters
Linux ethernet drivers are usually configured for an optimum balance between response and throughput with reasonable system load. For this reason they coalesce frames on both ingress and egress to minimise calls though the ethernet driver. For an AVB system the default coalesce parameters are rarely optimal, and so should be reviewed. Ideally, both rx and tx coalescing should be disabled, or where that is not possible, set to a low value. The ethtool command can be used to get and set the coalescing parameters for each network interface:
# ethtool -c enp0s29f1
Coalesce parameters for enp0s29f1:
rx-usecs: 20
rx-frames: 0
rx-usecs-irq: n/a
rx-frames-irq: n/a
tx-usecs: 0
tx-frames: 1
tx-usecs-irq: n/a
tx-frames-irq: n/a
The output shows optimal coalesce parameters for a Synopsis Designware ethernet MAC on an Intel Atom SoC. Egress (tx) frames are not coalesced at all (tx-frames == 1) meaning that egress packets are sent right away. On the ingress (rx) side, it's not possible with this driver to disable coaslescing, so the rx-usecs parameter is set to the minimal valid value of 20uS. This means that ingressing packets can be delayed in the network driver for up to 20uS before being delivered to the application. This is not ideal, but as mentioned in the previous section, is not fatal.
Coalesce parameters can be set using ethtool:
sudo ethtool -C enp0s29f1 rx-usecs 21
Energy Efficient Ethernet (EEE)
Energy Efficient Ethernet (EEE) must be disabled on all ports used for AVB. EEE is incompatible with the Precision Time Protocol (gPTP) and will cause delays to time sensitive packets.
ethtool can be used to show and set the EEE parameters for an ethernet interface:
# ethtool --show-eee enp0s29f1
EEE settings for enp0s29f1:
EEE status: disabled
Tx LPI: disabled
Supported EEE link modes: 100baseT/Full
1000baseT/Full
Advertised EEE link modes: Not reported
Link partner advertised EEE link modes: Not reported
Here we can see that EEE is disabled. If it is not already disabled, then disable it with:
ethtool --set-eee enp0s29f1 eee off
Notes and links
Ubuntu Intel IoT distribution: https://ubuntu.com/download/iot/intel-iot
Ubuntu Intel IoT real-time kernel tuning: https://ubuntu.com/blog/real-time-kernel-tuning