minix/kernel/watchdog.c

114 lines
2.6 KiB
C
Raw Normal View History

NMI watchdog is an awesome feature for debugging locked up kernels. There is not that much use for it on a single CPU, however, deadlock between kernel and system task can be delected. Or a runaway loop. If a kernel gets locked up the timer interrupts don't occure (as all interrupts are disabled in kernel mode). The only chance is to interrupt the kernel by a non-maskable interrupt. This patch generates NMIs using performance counters. It uses the most widely available performace counters. As the performance counters are highly model-specific this patch is not guaranteed to work on every machine. Unfortunately this is also true for KVM :-/ On the other hand adding this feature for other models is not extremely difficult and the framework makes it hopefully easy enough. Depending on the frequency of the CPU an NMI is generated at most about every 0.5s If the cpu's speed is less then 2Ghz it is generated at most every 1s. In general an NMI is generated much less often as the performance counter counts down only if the cpu is not idle. Therefore the overhead of this feature is fairly minimal even if the load is high. Uppon detecting that the kernel is locked up the kernel dumps the state of the kernel registers and panics. Local APIC must be enabled for the watchdog to work. The code is _always_ compiled in, however, it is only enabled if watchdog=<non-zero> is set in the boot monitor. One corner case is serial console debugging. As dumping a lot of stuff to the serial link may take a lot of time, the watchdog does not detect lockups during this time!!! as it would result in too many false positives. 10 nmi have to be handled before the lockup is detected. This means something between ~5s to 10s. Another corner case is that the watchdog is enabled only after the paging is enabled as it would be pure madness to try to get it right.
2010-01-16 21:53:55 +01:00
/*
* This is arch independent NMI watchdog implementaion part. It is used to
* detect kernel lockups and help debugging. each architecture must add its own
* low level code that triggers periodic checks
*/
#include "watchdog.h"
#include "arch/i386/glo.h"
#include "profile.h"
NMI watchdog is an awesome feature for debugging locked up kernels. There is not that much use for it on a single CPU, however, deadlock between kernel and system task can be delected. Or a runaway loop. If a kernel gets locked up the timer interrupts don't occure (as all interrupts are disabled in kernel mode). The only chance is to interrupt the kernel by a non-maskable interrupt. This patch generates NMIs using performance counters. It uses the most widely available performace counters. As the performance counters are highly model-specific this patch is not guaranteed to work on every machine. Unfortunately this is also true for KVM :-/ On the other hand adding this feature for other models is not extremely difficult and the framework makes it hopefully easy enough. Depending on the frequency of the CPU an NMI is generated at most about every 0.5s If the cpu's speed is less then 2Ghz it is generated at most every 1s. In general an NMI is generated much less often as the performance counter counts down only if the cpu is not idle. Therefore the overhead of this feature is fairly minimal even if the load is high. Uppon detecting that the kernel is locked up the kernel dumps the state of the kernel registers and panics. Local APIC must be enabled for the watchdog to work. The code is _always_ compiled in, however, it is only enabled if watchdog=<non-zero> is set in the boot monitor. One corner case is serial console debugging. As dumping a lot of stuff to the serial link may take a lot of time, the watchdog does not detect lockups during this time!!! as it would result in too many false positives. 10 nmi have to be handled before the lockup is detected. This means something between ~5s to 10s. Another corner case is that the watchdog is enabled only after the paging is enabled as it would be pure madness to try to get it right.
2010-01-16 21:53:55 +01:00
unsigned watchdog_local_timer_ticks = 0U;
NMI watchdog is an awesome feature for debugging locked up kernels. There is not that much use for it on a single CPU, however, deadlock between kernel and system task can be delected. Or a runaway loop. If a kernel gets locked up the timer interrupts don't occure (as all interrupts are disabled in kernel mode). The only chance is to interrupt the kernel by a non-maskable interrupt. This patch generates NMIs using performance counters. It uses the most widely available performace counters. As the performance counters are highly model-specific this patch is not guaranteed to work on every machine. Unfortunately this is also true for KVM :-/ On the other hand adding this feature for other models is not extremely difficult and the framework makes it hopefully easy enough. Depending on the frequency of the CPU an NMI is generated at most about every 0.5s If the cpu's speed is less then 2Ghz it is generated at most every 1s. In general an NMI is generated much less often as the performance counter counts down only if the cpu is not idle. Therefore the overhead of this feature is fairly minimal even if the load is high. Uppon detecting that the kernel is locked up the kernel dumps the state of the kernel registers and panics. Local APIC must be enabled for the watchdog to work. The code is _always_ compiled in, however, it is only enabled if watchdog=<non-zero> is set in the boot monitor. One corner case is serial console debugging. As dumping a lot of stuff to the serial link may take a lot of time, the watchdog does not detect lockups during this time!!! as it would result in too many false positives. 10 nmi have to be handled before the lockup is detected. This means something between ~5s to 10s. Another corner case is that the watchdog is enabled only after the paging is enabled as it would be pure madness to try to get it right.
2010-01-16 21:53:55 +01:00
struct arch_watchdog *watchdog;
int watchdog_enabled;
PRIVATE void lockup_check(struct nmi_frame * frame)
NMI watchdog is an awesome feature for debugging locked up kernels. There is not that much use for it on a single CPU, however, deadlock between kernel and system task can be delected. Or a runaway loop. If a kernel gets locked up the timer interrupts don't occure (as all interrupts are disabled in kernel mode). The only chance is to interrupt the kernel by a non-maskable interrupt. This patch generates NMIs using performance counters. It uses the most widely available performace counters. As the performance counters are highly model-specific this patch is not guaranteed to work on every machine. Unfortunately this is also true for KVM :-/ On the other hand adding this feature for other models is not extremely difficult and the framework makes it hopefully easy enough. Depending on the frequency of the CPU an NMI is generated at most about every 0.5s If the cpu's speed is less then 2Ghz it is generated at most every 1s. In general an NMI is generated much less often as the performance counter counts down only if the cpu is not idle. Therefore the overhead of this feature is fairly minimal even if the load is high. Uppon detecting that the kernel is locked up the kernel dumps the state of the kernel registers and panics. Local APIC must be enabled for the watchdog to work. The code is _always_ compiled in, however, it is only enabled if watchdog=<non-zero> is set in the boot monitor. One corner case is serial console debugging. As dumping a lot of stuff to the serial link may take a lot of time, the watchdog does not detect lockups during this time!!! as it would result in too many false positives. 10 nmi have to be handled before the lockup is detected. This means something between ~5s to 10s. Another corner case is that the watchdog is enabled only after the paging is enabled as it would be pure madness to try to get it right.
2010-01-16 21:53:55 +01:00
{
/* FIXME this should be CPU local */
static unsigned no_ticks;
static unsigned last_tick_count = (unsigned) -1;
/*
* when debugging on serial console, printing takes a lot of time some
* times while the kernel is certainly not locked up. We don't want to
* report a lockup in such situation
*/
if (serial_debug_active)
return;
NMI watchdog is an awesome feature for debugging locked up kernels. There is not that much use for it on a single CPU, however, deadlock between kernel and system task can be delected. Or a runaway loop. If a kernel gets locked up the timer interrupts don't occure (as all interrupts are disabled in kernel mode). The only chance is to interrupt the kernel by a non-maskable interrupt. This patch generates NMIs using performance counters. It uses the most widely available performace counters. As the performance counters are highly model-specific this patch is not guaranteed to work on every machine. Unfortunately this is also true for KVM :-/ On the other hand adding this feature for other models is not extremely difficult and the framework makes it hopefully easy enough. Depending on the frequency of the CPU an NMI is generated at most about every 0.5s If the cpu's speed is less then 2Ghz it is generated at most every 1s. In general an NMI is generated much less often as the performance counter counts down only if the cpu is not idle. Therefore the overhead of this feature is fairly minimal even if the load is high. Uppon detecting that the kernel is locked up the kernel dumps the state of the kernel registers and panics. Local APIC must be enabled for the watchdog to work. The code is _always_ compiled in, however, it is only enabled if watchdog=<non-zero> is set in the boot monitor. One corner case is serial console debugging. As dumping a lot of stuff to the serial link may take a lot of time, the watchdog does not detect lockups during this time!!! as it would result in too many false positives. 10 nmi have to be handled before the lockup is detected. This means something between ~5s to 10s. Another corner case is that the watchdog is enabled only after the paging is enabled as it would be pure madness to try to get it right.
2010-01-16 21:53:55 +01:00
if (last_tick_count != watchdog_local_timer_ticks) {
if (no_ticks == 1) {
printf("watchdog : kernel unlocked\n");
NMI watchdog is an awesome feature for debugging locked up kernels. There is not that much use for it on a single CPU, however, deadlock between kernel and system task can be delected. Or a runaway loop. If a kernel gets locked up the timer interrupts don't occure (as all interrupts are disabled in kernel mode). The only chance is to interrupt the kernel by a non-maskable interrupt. This patch generates NMIs using performance counters. It uses the most widely available performace counters. As the performance counters are highly model-specific this patch is not guaranteed to work on every machine. Unfortunately this is also true for KVM :-/ On the other hand adding this feature for other models is not extremely difficult and the framework makes it hopefully easy enough. Depending on the frequency of the CPU an NMI is generated at most about every 0.5s If the cpu's speed is less then 2Ghz it is generated at most every 1s. In general an NMI is generated much less often as the performance counter counts down only if the cpu is not idle. Therefore the overhead of this feature is fairly minimal even if the load is high. Uppon detecting that the kernel is locked up the kernel dumps the state of the kernel registers and panics. Local APIC must be enabled for the watchdog to work. The code is _always_ compiled in, however, it is only enabled if watchdog=<non-zero> is set in the boot monitor. One corner case is serial console debugging. As dumping a lot of stuff to the serial link may take a lot of time, the watchdog does not detect lockups during this time!!! as it would result in too many false positives. 10 nmi have to be handled before the lockup is detected. This means something between ~5s to 10s. Another corner case is that the watchdog is enabled only after the paging is enabled as it would be pure madness to try to get it right.
2010-01-16 21:53:55 +01:00
no_ticks = 0;
}
/* we are still ticking, everything seems good */
last_tick_count = watchdog_local_timer_ticks;
return;
NMI watchdog is an awesome feature for debugging locked up kernels. There is not that much use for it on a single CPU, however, deadlock between kernel and system task can be delected. Or a runaway loop. If a kernel gets locked up the timer interrupts don't occure (as all interrupts are disabled in kernel mode). The only chance is to interrupt the kernel by a non-maskable interrupt. This patch generates NMIs using performance counters. It uses the most widely available performace counters. As the performance counters are highly model-specific this patch is not guaranteed to work on every machine. Unfortunately this is also true for KVM :-/ On the other hand adding this feature for other models is not extremely difficult and the framework makes it hopefully easy enough. Depending on the frequency of the CPU an NMI is generated at most about every 0.5s If the cpu's speed is less then 2Ghz it is generated at most every 1s. In general an NMI is generated much less often as the performance counter counts down only if the cpu is not idle. Therefore the overhead of this feature is fairly minimal even if the load is high. Uppon detecting that the kernel is locked up the kernel dumps the state of the kernel registers and panics. Local APIC must be enabled for the watchdog to work. The code is _always_ compiled in, however, it is only enabled if watchdog=<non-zero> is set in the boot monitor. One corner case is serial console debugging. As dumping a lot of stuff to the serial link may take a lot of time, the watchdog does not detect lockups during this time!!! as it would result in too many false positives. 10 nmi have to be handled before the lockup is detected. This means something between ~5s to 10s. Another corner case is that the watchdog is enabled only after the paging is enabled as it would be pure madness to try to get it right.
2010-01-16 21:53:55 +01:00
}
/*
* if watchdog_local_timer_ticks didn't changed since last time, give it
* some more time and only if it still dead, trigger the watchdog alarm
*/
if (++no_ticks < 10) {
if (no_ticks == 1)
printf("WARNING watchdog : possible kernel lockup\n");
return;
NMI watchdog is an awesome feature for debugging locked up kernels. There is not that much use for it on a single CPU, however, deadlock between kernel and system task can be delected. Or a runaway loop. If a kernel gets locked up the timer interrupts don't occure (as all interrupts are disabled in kernel mode). The only chance is to interrupt the kernel by a non-maskable interrupt. This patch generates NMIs using performance counters. It uses the most widely available performace counters. As the performance counters are highly model-specific this patch is not guaranteed to work on every machine. Unfortunately this is also true for KVM :-/ On the other hand adding this feature for other models is not extremely difficult and the framework makes it hopefully easy enough. Depending on the frequency of the CPU an NMI is generated at most about every 0.5s If the cpu's speed is less then 2Ghz it is generated at most every 1s. In general an NMI is generated much less often as the performance counter counts down only if the cpu is not idle. Therefore the overhead of this feature is fairly minimal even if the load is high. Uppon detecting that the kernel is locked up the kernel dumps the state of the kernel registers and panics. Local APIC must be enabled for the watchdog to work. The code is _always_ compiled in, however, it is only enabled if watchdog=<non-zero> is set in the boot monitor. One corner case is serial console debugging. As dumping a lot of stuff to the serial link may take a lot of time, the watchdog does not detect lockups during this time!!! as it would result in too many false positives. 10 nmi have to be handled before the lockup is detected. This means something between ~5s to 10s. Another corner case is that the watchdog is enabled only after the paging is enabled as it would be pure madness to try to get it right.
2010-01-16 21:53:55 +01:00
}
/* if we get this far, the kernel is locked up */
NMI watchdog is an awesome feature for debugging locked up kernels. There is not that much use for it on a single CPU, however, deadlock between kernel and system task can be delected. Or a runaway loop. If a kernel gets locked up the timer interrupts don't occure (as all interrupts are disabled in kernel mode). The only chance is to interrupt the kernel by a non-maskable interrupt. This patch generates NMIs using performance counters. It uses the most widely available performace counters. As the performance counters are highly model-specific this patch is not guaranteed to work on every machine. Unfortunately this is also true for KVM :-/ On the other hand adding this feature for other models is not extremely difficult and the framework makes it hopefully easy enough. Depending on the frequency of the CPU an NMI is generated at most about every 0.5s If the cpu's speed is less then 2Ghz it is generated at most every 1s. In general an NMI is generated much less often as the performance counter counts down only if the cpu is not idle. Therefore the overhead of this feature is fairly minimal even if the load is high. Uppon detecting that the kernel is locked up the kernel dumps the state of the kernel registers and panics. Local APIC must be enabled for the watchdog to work. The code is _always_ compiled in, however, it is only enabled if watchdog=<non-zero> is set in the boot monitor. One corner case is serial console debugging. As dumping a lot of stuff to the serial link may take a lot of time, the watchdog does not detect lockups during this time!!! as it would result in too many false positives. 10 nmi have to be handled before the lockup is detected. This means something between ~5s to 10s. Another corner case is that the watchdog is enabled only after the paging is enabled as it would be pure madness to try to get it right.
2010-01-16 21:53:55 +01:00
arch_watchdog_lockup(frame);
}
NMI watchdog is an awesome feature for debugging locked up kernels. There is not that much use for it on a single CPU, however, deadlock between kernel and system task can be delected. Or a runaway loop. If a kernel gets locked up the timer interrupts don't occure (as all interrupts are disabled in kernel mode). The only chance is to interrupt the kernel by a non-maskable interrupt. This patch generates NMIs using performance counters. It uses the most widely available performace counters. As the performance counters are highly model-specific this patch is not guaranteed to work on every machine. Unfortunately this is also true for KVM :-/ On the other hand adding this feature for other models is not extremely difficult and the framework makes it hopefully easy enough. Depending on the frequency of the CPU an NMI is generated at most about every 0.5s If the cpu's speed is less then 2Ghz it is generated at most every 1s. In general an NMI is generated much less often as the performance counter counts down only if the cpu is not idle. Therefore the overhead of this feature is fairly minimal even if the load is high. Uppon detecting that the kernel is locked up the kernel dumps the state of the kernel registers and panics. Local APIC must be enabled for the watchdog to work. The code is _always_ compiled in, however, it is only enabled if watchdog=<non-zero> is set in the boot monitor. One corner case is serial console debugging. As dumping a lot of stuff to the serial link may take a lot of time, the watchdog does not detect lockups during this time!!! as it would result in too many false positives. 10 nmi have to be handled before the lockup is detected. This means something between ~5s to 10s. Another corner case is that the watchdog is enabled only after the paging is enabled as it would be pure madness to try to get it right.
2010-01-16 21:53:55 +01:00
PUBLIC void nmi_watchdog_handler(struct nmi_frame * frame)
{
#if SPROFILE
/*
* Do not check for lockups while profiling, it is extremely likely that
* a false positive is detected if the frequency is high
*/
if (watchdog_enabled && !sprofiling)
lockup_check(frame);
if (sprofiling)
nmi_sprofile_handler(frame);
if ((watchdog_enabled || sprofiling) && watchdog->reinit)
NMI watchdog is an awesome feature for debugging locked up kernels. There is not that much use for it on a single CPU, however, deadlock between kernel and system task can be delected. Or a runaway loop. If a kernel gets locked up the timer interrupts don't occure (as all interrupts are disabled in kernel mode). The only chance is to interrupt the kernel by a non-maskable interrupt. This patch generates NMIs using performance counters. It uses the most widely available performace counters. As the performance counters are highly model-specific this patch is not guaranteed to work on every machine. Unfortunately this is also true for KVM :-/ On the other hand adding this feature for other models is not extremely difficult and the framework makes it hopefully easy enough. Depending on the frequency of the CPU an NMI is generated at most about every 0.5s If the cpu's speed is less then 2Ghz it is generated at most every 1s. In general an NMI is generated much less often as the performance counter counts down only if the cpu is not idle. Therefore the overhead of this feature is fairly minimal even if the load is high. Uppon detecting that the kernel is locked up the kernel dumps the state of the kernel registers and panics. Local APIC must be enabled for the watchdog to work. The code is _always_ compiled in, however, it is only enabled if watchdog=<non-zero> is set in the boot monitor. One corner case is serial console debugging. As dumping a lot of stuff to the serial link may take a lot of time, the watchdog does not detect lockups during this time!!! as it would result in too many false positives. 10 nmi have to be handled before the lockup is detected. This means something between ~5s to 10s. Another corner case is that the watchdog is enabled only after the paging is enabled as it would be pure madness to try to get it right.
2010-01-16 21:53:55 +01:00
watchdog->reinit(cpuid);
#else
if (watchdog_enabled) {
lockup_check(frame);
if (watchdog->reinit)
watchdog->reinit(cpuid);
}
#endif
NMI watchdog is an awesome feature for debugging locked up kernels. There is not that much use for it on a single CPU, however, deadlock between kernel and system task can be delected. Or a runaway loop. If a kernel gets locked up the timer interrupts don't occure (as all interrupts are disabled in kernel mode). The only chance is to interrupt the kernel by a non-maskable interrupt. This patch generates NMIs using performance counters. It uses the most widely available performace counters. As the performance counters are highly model-specific this patch is not guaranteed to work on every machine. Unfortunately this is also true for KVM :-/ On the other hand adding this feature for other models is not extremely difficult and the framework makes it hopefully easy enough. Depending on the frequency of the CPU an NMI is generated at most about every 0.5s If the cpu's speed is less then 2Ghz it is generated at most every 1s. In general an NMI is generated much less often as the performance counter counts down only if the cpu is not idle. Therefore the overhead of this feature is fairly minimal even if the load is high. Uppon detecting that the kernel is locked up the kernel dumps the state of the kernel registers and panics. Local APIC must be enabled for the watchdog to work. The code is _always_ compiled in, however, it is only enabled if watchdog=<non-zero> is set in the boot monitor. One corner case is serial console debugging. As dumping a lot of stuff to the serial link may take a lot of time, the watchdog does not detect lockups during this time!!! as it would result in too many false positives. 10 nmi have to be handled before the lockup is detected. This means something between ~5s to 10s. Another corner case is that the watchdog is enabled only after the paging is enabled as it would be pure madness to try to get it right.
2010-01-16 21:53:55 +01:00
}
int nmi_watchdog_start_profiling(const unsigned freq)
{
int err;
/* if watchdog hasn't been enabled, we must enable it now */
if (!watchdog_enabled) {
if (arch_watchdog_init())
return ENODEV;
}
if (!watchdog->profile_init) {
printf("WARNING NMI watchdog profiling not supported\n");
nmi_watchdog_stop_profiling();
return ENODEV;
}
err = watchdog->profile_init(freq);
if (err != OK)
return err;
watchdog->resetval = watchdog->profile_resetval;
return OK;
}
void nmi_watchdog_stop_profiling(void)
{
/*
* if we do not rearm the NMI source, we are done, if we want to keep
* the watchdog running, we reset is to its normal value
*/
if (watchdog)
watchdog->resetval = watchdog->watchdog_resetval;
if (!watchdog_enabled)
arch_watchdog_stop();
}