Interrupt storm
From Wikipedia, the free encyclopedia
In operating systems, an Interrupt Storm is an event where a processor receives an inordinate number of interrupts that consume the majority of the processor's time. Interrupt storms are typically caused by hardware devices that do not support interrupt rate limiting.
Contents |
[edit] Background
Because interrupt processing is typically a non-preemptible task in time-sharing operating systems, an interrupt storm will cause low perceived system responsiveness, or even appear to be a complete system freeze. This state is commonly known as live lock. In such a state, the system is spending so much time processing interrupts that it is not completing any other work. Therefore, it does not appear to be processing anything at all, because of a lack of output to the user, the network, or otherwise. An interrupt storm is sometimes mistaken for thrashing, since they both have similar symptoms, but different causes.
An interrupt storm can have many different causes, including misconfigured or faulty hardware devices, faulty device drivers, or flaws in the operating system. Most modern hardware implement methods for reducing or eliminating the possibility of an interrupt storm. For example, many Ethernet controllers implement interrupt 'rate limiting' which causes the controller to wait a programmable minimum amount of time between each interrupt it generates.
The most common interrupt storm is a faulty driver under an APIC (Advanced programmable interrupt controller) where a device "behind" another signals an interrupt to the APIC. The OS then asks each driver on that interrupt if it was from its hardware. Faulty drivers may always claim "yes", but then proceed no further as the hardware attached actually did not interrupt. The device which originally interrupted did not get its interrupt serviced, so interrupts again and the cycle begins anew. The system locks dead under an interrupt storm. This was (and remains) a problem on the SoundBlaster Live! series of sound cards on some motherboards and only a kernel debugger can break the storm by unloading the faulty driver.
Many OS implement a polling mode that disables interrupts for devices which generate too many interrupts. In this mode, the OS peridodically queries the hardware for pending tasks. As the number of interrupts increase and the efficiency of an interrupt mode deminishes, an OS may change the interrupting device from an interrupt mode to a polling mode. Likewise, as the polling mode becomes less efficient than the interrupt mode, the OS will switch the device back to the interrupt mode. The implementation of interrupt rate limiting in hardware almost negates the need for such polling modes.
[edit] Considerations
Interrupt rate limiting must be carefully configured for optimium results. For example, an Ethernet controller with interrupt rate limiting will buffer the packets it receives from the network in between each interrupt. If the rate is set too high, the controller's buffer will overflow, and packets will be dropped. The rate must take into account how fast the buffer may fill between interrupts, and the interrupt latency between the interrupt and the transfer of the buffer to the system .
[edit] Interrupt mitigating
There are hardware based and software based approaches to the problem. Among possible schemes is one used by NAPI
- System (driver) starts in interrupt enabled state
- Interrupt handler disables the interrupt and lets a thread/task handle the event(s) (example of event is an incoming Ethernet packet)
- Task polls the device, process some number (all) events and enables the interrupt
Another interesting approach using hardware support - device generates interrupt when event queue state changes from empty to not empty
Device
- If there is no free DMA descriptors at the RX FIFO tail drop the event
- Add event to the tail and mark the FIFO entry as occupied
- If entry (tail-1) is free (cleared) generate interrupt (level interrupt)
- Increment tail pointer
CPU (interrupt handler)
- Acknowledge the interrupt (if hardware requires acknowledge)
- Handle all (part of) valid DMA descriptors at head
- return from interrupt
