Commit 9650edd9 authored by Paul E. McKenney's avatar Paul E. McKenney Committed by Frederic Weisbecker
Browse files

rcu: Finer-grained grace-period-end checks in rcu_dump_cpu_stacks() 



This commit pushes the grace-period-end checks further down into
rcu_dump_cpu_stacks(), and also uses lockless checks coupled with
finer-grained locking.

The result is that the current leaf rcu_node structure's ->lock is
acquired only if a stack backtrace might be needed from the current CPU,
and is held across only that CPU's backtrace.  As a result, if there are
no stalled CPUs associated with a given rcu_node structure, then its
->lock will not be acquired at all.  On large systems, it is usually
(though not always) the case that a small number of CPUs are stalling
the current grace period, which means that the ->lock need be acquired
only for a small fraction of the rcu_node structures.

[ paulmck: Apply Dan Carpenter feedback. ]

Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
Reviewed-by: default avatarJoel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: default avatarFrederic Weisbecker <frederic@kernel.org>
parent cbe644aa

kernel/rcu/tree_stall.h

+11 −7
Original line number Diff line number Diff line
@@ -342,13 +342,16 @@ static void rcu_dump_cpu_stacks(unsigned long gp_seq) 
	struct rcu_node *rnp;



	rcu_for_each_leaf_node(rnp) {

		printk_deferred_enter();

		for_each_leaf_node_possible_cpu(rnp, cpu) {

			if (gp_seq != data_race(rcu_state.gp_seq)) {

				printk_deferred_exit();

				pr_err("INFO: Stall ended during stack backtracing.\n");

				return;

			}

		printk_deferred_enter();

			if (!(data_race(rnp->qsmask) & leaf_node_cpu_bit(rnp, cpu)))

				continue;

			raw_spin_lock_irqsave_rcu_node(rnp, flags);

		for_each_leaf_node_possible_cpu(rnp, cpu)

			if (rnp->qsmask & leaf_node_cpu_bit(rnp, cpu)) {

				if (cpu_is_offline(cpu))

					pr_err("Offline CPU %d blocking current GP.\n", cpu);
@@ -356,6 +359,7 @@ static void rcu_dump_cpu_stacks(unsigned long gp_seq) 
					dump_cpu_task(cpu);

			}

			raw_spin_unlock_irqrestore_rcu_node(rnp, flags);

		}

		printk_deferred_exit();

	}

}