Blog

SysAdmin Series – What does ksoftirqd do ?

Scenario:

Operating System manages and controls I/O operations and I/O devices. Let’s have a quick overview of different ways of interacting with I/O devices, before we go through what a ksoftirqd does.

Interrupts – The CPU issues commands to the I/O module then proceeds with its normal work until interrupted by I/O device on completion of its work.

DMA – Direct Memory Access (DMA) means CPU grants I/O module, the authority to read from or write to memory without involvement.

Polling – OS simply checks from time to time if a device has completed a request.

There is DMA + Interrupts as well as DMA + polling as well.

Now let’s consider the interrupts way of interacting with the I/O. Interrupts from the hardware are known as “top-half” interrupts. Interrupts are best explained with network traffic.

When a NIC receives incoming data, it copies the data into kernel buffers using DMA. The NIC notifies the kernel of this data by raising a hard interrupt. These interrupts are processed by interrupt handlers which do minimal work, as they have already interrupted another task and cannot be interrupted themselves.

Hard interrupts can be expensive in terms of CPU usage, especially when holding kernel locks. The hard interrupt handler then leaves the majority of packet reception to a software interrupt, or SoftIRQ, a process which can be scheduled more fairly.

Hard interrupts can be seen in /proc/interrupts while soft interrupts can be seen at /proc/softirqs

The softirq system in the Linux kernel is a mechanism for executing code outside of the context of an interrupt handler implemented in a driver. This system is important because hardware interrupts may be disabled during all or part of the execution of an interrupt handler. When the interrupt handlers are processing the interrupts, they cannot be interrupted themselves. That is why we have the interrupt handlers doing the minimal work, of interruption, while the responsibility of actual receiving of network traffic is managed by software interrupts.

In some situations, IRQs come very very fast one after the other and the operating system cannot finish servicing one before another one arrives. One of the mechanisms of deferring the tasks is by implementing a queuing system, called ksoftirqd. ksoftirqd, is utilized when the machine is under heavy soft interrupt load.

If ksoftirqd is taking more than a certain percentage of CPU time, this indicates the machine is under heavy interrupt load and not eating CPU. If there is a bug in a driver or any other issue, you can see that one CPU is taking too much load while others are underutilized.

You can tweak the settings a bit, by defining which CPU picks up a certain
interrupt. You do this by changing the contents of /proc/irq/$interrupt_number/smp_affinity. You can get a list of interrupts and their meaning by doing:

cat /proc/interrupts

The number in smp_affinity is a bitmap of CPUs, represented in hex code. The rightmost bit is the least significant. For instance, if the system has 8 cores and you only wanted to use cores 0, 1 and 3, you should set the smp_affinity to

1a:

cpu_7 cpu_6 cpu_5 cpu_4 cpu_3 cpu_2 cpu_1 cpu_0
0  0  0  0  1  0  1  1 = 00001011 = 0B (in hex)

For eg: if you want to set up any of the CPU to be able to pick up interrupt 38 (eth0 in my 8-core system) with:

sudo echo ff > /proc/irq/38/smp_affinity

Category : Sever management