Commit 9544f9e6 authored by Li RongQing's avatar Li RongQing Committed by Andrew Morton
Browse files

hung_task: panic when there are more than N hung tasks at the same time 

The hung_task_panic sysctl is currently a blunt instrument: it's all or
nothing.

Panicking on a single hung task can be an overreaction to a transient
glitch.  A more reliable indicator of a systemic problem is when
multiple tasks hang simultaneously.

Extend hung_task_panic to accept an integer threshold, allowing the
kernel to panic only when N hung tasks are detected in a single scan. 
This provides finer control to distinguish between isolated incidents
and system-wide failures.

The accepted values are:
- 0: Don't panic (unchanged)
- 1: Panic on the first hung task (unchanged)
- N > 1: Panic after N hung tasks are detected in a single scan

The original behavior is preserved for values 0 and 1, maintaining full
backward compatibility.

[lance.yang@linux.dev: new changelog]
Link: https://lkml.kernel.org/r/20251015063615.2632-1-lirongqing@baidu.com


Signed-off-by: default avatarLi RongQing <lirongqing@baidu.com>
Reviewed-by: default avatarMasami Hiramatsu (Google) <mhiramat@kernel.org>
Reviewed-by: default avatarLance Yang <lance.yang@linux.dev>
Tested-by: default avatarLance Yang <lance.yang@linux.dev>
Acked-by: Andrew Jeffery <andrew@codeconstruct.com.au> [aspeed_g5_defconfig]
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: David Hildenbrand <david@redhat.com>
Cc: Florian Wesphal <fw@strlen.de>
Cc: Jakub Kacinski <kuba@kernel.org>
Cc: Jason A. Donenfeld <jason@zx2c4.com>
Cc: Joel Granados <joel.granados@kernel.org>
Cc: Joel Stanley <joel@jms.id.au>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Kees Cook <kees@kernel.org>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: "Paul E . McKenney" <paulmck@kernel.org>
Cc: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Cc: Petr Mladek <pmladek@suse.com>
Cc: Phil Auld <pauld@redhat.com>
Cc: Randy Dunlap <rdunlap@infradead.org>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Simon Horman <horms@kernel.org>
Cc: Stanislav Fomichev <sdf@fomichev.me>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
parent 05d6f1cc

Documentation/admin-guide/kernel-parameters.txt

+13 −7
Original line number Diff line number Diff line
@@ -2010,14 +2010,20 @@ 
			the added memory block itself do not be affected.



	hung_task_panic=

			[KNL] Should the hung task detector generate panics.

			Format: 0 | 1

			[KNL] Number of hung tasks to trigger kernel panic.

			Format: <int>



			When set to a non-zero value, a kernel panic will be triggered if

			the number of detected hung tasks reaches this value.



			0: don't panic

			1: panic immediately on first hung task

			N: panic after N hung tasks are detected in a single scan



			A value of 1 instructs the kernel to panic when a

			hung task is detected. The default value is controlled

			by the CONFIG_BOOTPARAM_HUNG_TASK_PANIC build-time

			option. The value selected by this boot parameter can

			be changed later by the kernel.hung_task_panic sysctl.

			The default value is controlled by the

			CONFIG_BOOTPARAM_HUNG_TASK_PANIC build-time option. The value

			selected by this boot parameter can be changed later by the

			kernel.hung_task_panic sysctl.



	hvc_iucv=	[S390]	Number of z/VM IUCV hypervisor console (HVC)

				terminal devices. Valid values: 0..8

Documentation/admin-guide/sysctl/kernel.rst

+5 −4
Original line number Diff line number Diff line
@@ -397,13 +397,14 @@ a hung task is detected. 
hung_task_panic

===============



Controls the kernel's behavior when a hung task is detected.

When set to a non-zero value, a kernel panic will be triggered if the

number of hung tasks found during a single scan reaches this value.

This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled.



= =================================================

= =======================================================

0 Continue operation. This is the default behavior.

1 Panic immediately.

= =================================================

N Panic when N hung tasks are found during a single scan.

= =======================================================





hung_task_check_count

arch/arm/configs/aspeed_g5_defconfig

+1 −1
Original line number Diff line number Diff line
@@ -308,7 +308,7 @@ CONFIG_PANIC_ON_OOPS=y 
CONFIG_PANIC_TIMEOUT=-1

CONFIG_SOFTLOCKUP_DETECTOR=y

CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y

CONFIG_BOOTPARAM_HUNG_TASK_PANIC=y

CONFIG_BOOTPARAM_HUNG_TASK_PANIC=1

CONFIG_WQ_WATCHDOG=y

# CONFIG_SCHED_DEBUG is not set

CONFIG_FUNCTION_TRACER=y

kernel/configs/debug.config

+1 −1
Original line number Diff line number Diff line
@@ -83,7 +83,7 @@ CONFIG_SLUB_DEBUG_ON=y 
#

# Debug Oops, Lockups and Hangs

#

# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set

CONFIG_BOOTPARAM_HUNG_TASK_PANIC=0

# CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set

CONFIG_DEBUG_ATOMIC_SLEEP=y

CONFIG_DETECT_HUNG_TASK=y

kernel/hung_task.c

+10 −5
Original line number Diff line number Diff line
@@ -81,7 +81,7 @@ static unsigned int __read_mostly sysctl_hung_task_all_cpu_backtrace; 
 * hung task is detected:

 */

static unsigned int __read_mostly sysctl_hung_task_panic =

	IS_ENABLED(CONFIG_BOOTPARAM_HUNG_TASK_PANIC);

	CONFIG_BOOTPARAM_HUNG_TASK_PANIC;



static int

hung_task_panic(struct notifier_block *this, unsigned long event, void *ptr)
@@ -218,8 +218,11 @@ static inline void debug_show_blocker(struct task_struct *task, unsigned long ti 
}

#endif



static void check_hung_task(struct task_struct *t, unsigned long timeout)

static void check_hung_task(struct task_struct *t, unsigned long timeout,

		unsigned long prev_detect_count)

{

	unsigned long total_hung_task;



	if (!task_is_hung(t, timeout))

		return;
@@ -229,9 +232,10 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout) 
	 */

	sysctl_hung_task_detect_count++;



	total_hung_task = sysctl_hung_task_detect_count - prev_detect_count;

	trace_sched_process_hang(t);



	if (sysctl_hung_task_panic) {

	if (sysctl_hung_task_panic && total_hung_task >= sysctl_hung_task_panic) {

		console_verbose();

		hung_task_show_lock = true;

		hung_task_call_panic = true;
@@ -300,6 +304,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) 
	int max_count = sysctl_hung_task_check_count;

	unsigned long last_break = jiffies;

	struct task_struct *g, *t;

	unsigned long prev_detect_count = sysctl_hung_task_detect_count;



	/*

	 * If the system crashed already then all bets are off,
@@ -320,7 +325,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) 
			last_break = jiffies;

		}



		check_hung_task(t, timeout);

		check_hung_task(t, timeout, prev_detect_count);

	}

 unlock:

	rcu_read_unlock();
@@ -389,7 +394,7 @@ static const struct ctl_table hung_task_sysctls[] = { 
		.mode		= 0644,

		.proc_handler	= proc_dointvec_minmax,

		.extra1		= SYSCTL_ZERO,

		.extra2		= SYSCTL_ONE,

		.extra2		= SYSCTL_INT_MAX,

	},

	{

		.procname	= "hung_task_check_count",