Commit 0bb11a37 authored by Paul E. McKenney's avatar Paul E. McKenney Committed by Boqun Feng
Browse files

rcu-tasks: Maintain real-time response in rcu_tasks_postscan() 

The current code will scan the entirety of each per-CPU list of exiting
tasks in ->rtp_exit_list with interrupts disabled.  This is normally just
fine, because each CPU typically won't have very many tasks in this state.
However, if a large number of tasks block late in do_exit(), these lists
could be arbitrarily long.  Low probability, perhaps, but it really
could happen.

This commit therefore occasionally re-enables interrupts while traversing
these lists, inserting a dummy element to hold the current place in the
list.  In kernels built with CONFIG_PREEMPT_RT=y, this re-enabling happens
after each list element is processed, otherwise every one-to-two jiffies.

[ paulmck: Apply Frederic Weisbecker feedback. ]

Link: https://lore.kernel.org/all/ZdeI_-RfdLR8jlsm@localhost.localdomain/



Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Sebastian Siewior <bigeasy@linutronix.de>
Cc: Anna-Maria Behnsen <anna-maria@linutronix.de>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: default avatarBoqun Feng <boqun.feng@gmail.com>
parent 1612160b

kernel/rcu/tasks.h

+21 −1
Original line number Diff line number Diff line
@@ -968,13 +968,33 @@ static void rcu_tasks_postscan(struct list_head *hop) 
	 */



	for_each_possible_cpu(cpu) {

		unsigned long j = jiffies + 1;

		struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu);

		struct task_struct *t;

		struct task_struct *t1;

		struct list_head tmp;



		raw_spin_lock_irq_rcu_node(rtpcp);

		list_for_each_entry(t, &rtpcp->rtp_exit_list, rcu_tasks_exit_list)

		list_for_each_entry_safe(t, t1, &rtpcp->rtp_exit_list, rcu_tasks_exit_list) {

			if (list_empty(&t->rcu_tasks_holdout_list))

				rcu_tasks_pertask(t, hop);



			// RT kernels need frequent pauses, otherwise

			// pause at least once per pair of jiffies.

			if (!IS_ENABLED(CONFIG_PREEMPT_RT) && time_before(jiffies, j))

				continue;



			// Keep our place in the list while pausing.

			// Nothing else traverses this list, so adding a

			// bare list_head is OK.

			list_add(&tmp, &t->rcu_tasks_exit_list);

			raw_spin_unlock_irq_rcu_node(rtpcp);

			cond_resched(); // For CONFIG_PREEMPT=n kernels

			raw_spin_lock_irq_rcu_node(rtpcp);

			t1 = list_entry(tmp.next, struct task_struct, rcu_tasks_exit_list);

			list_del(&tmp);

			j = jiffies + 1;

		}

		raw_spin_unlock_irq_rcu_node(rtpcp);

	}