Commit 48fda62d authored Mar 24, 2026 by John Stultz Committed by Peter Zijlstra Apr 03, 2026

sched: Add logic to zap balance callbacks if we pick again



With proxy-exec, a task is selected to run via pick_next_task(),
and then if it is a mutex blocked task, we call find_proxy_task()
to find a runnable owner. If the runnable owner is on another
cpu, we will need to migrate the selected donor task away, after
which we will pick_again can call pick_next_task() to choose
something else.

However, in the first call to pick_next_task(), we may have
had a balance_callback setup by the class scheduler. After we
pick again, its possible pick_next_task_fair() will be called
which calls sched_balance_newidle() and sched_balance_rq().

This will throw a warning:
[    8.796467] rq->balance_callback && rq->balance_callback != &balance_push_callback
[    8.796467] WARNING: CPU: 32 PID: 458 at kernel/sched/sched.h:1750 sched_balance_rq+0xe92/0x1250
...
[    8.796467] Call Trace:
[    8.796467]  <TASK>
[    8.796467]  ? __warn.cold+0xb2/0x14e
[    8.796467]  ? sched_balance_rq+0xe92/0x1250
[    8.796467]  ? report_bug+0x107/0x1a0
[    8.796467]  ? handle_bug+0x54/0x90
[    8.796467]  ? exc_invalid_op+0x17/0x70
[    8.796467]  ? asm_exc_invalid_op+0x1a/0x20
[    8.796467]  ? sched_balance_rq+0xe92/0x1250
[    8.796467]  sched_balance_newidle+0x295/0x820
[    8.796467]  pick_next_task_fair+0x51/0x3f0
[    8.796467]  __schedule+0x23a/0x14b0
[    8.796467]  ? lock_release+0x16d/0x2e0
[    8.796467]  schedule+0x3d/0x150
[    8.796467]  worker_thread+0xb5/0x350
[    8.796467]  ? __pfx_worker_thread+0x10/0x10
[    8.796467]  kthread+0xee/0x120
[    8.796467]  ? __pfx_kthread+0x10/0x10
[    8.796467]  ret_from_fork+0x31/0x50
[    8.796467]  ? __pfx_kthread+0x10/0x10
[    8.796467]  ret_from_fork_asm+0x1a/0x30
[    8.796467]  </TASK>

This is because if a RT task was originally picked, it will
setup the rq->balance_callback with push_rt_tasks() via
set_next_task_rt().

Once the task is migrated away and we pick again, we haven't
processed any balance callbacks, so rq->balance_callback is not
in the same state as it was the first time pick_next_task was
called.

To handle this, add a zap_balance_callbacks() helper function
which cleans up the balance callbacks without running them. This
should be ok, as we are effectively undoing the state set in
the first call to pick_next_task(), and when we pick again,
the new callback can be configured for the donor task actually
selected.

Signed-off-by: John Stultz <jstultz@google.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Link: https://patch.msgid.link/20260324191337.1841376-9-jstultz@google.com

parent f9530b31

kernel/sched/core.c

+34 −2

Original line number	Diff line number	Diff line
		@@ -4917,6 +4917,34 @@ static inline void finish_task(struct task_struct *prev)
		smp_store_release(&prev->on_cpu, 0);
		}

		/*
		* Only called from __schedule context
		*
		* There are some cases where we are going to re-do the action
		* that added the balance callbacks. We may not be in a state
		* where we can run them, so just zap them so they can be
		* properly re-added on the next time around. This is similar
		* handling to running the callbacks, except we just don't call
		* them.
		*/
		static void zap_balance_callbacks(struct rq *rq)
		{
		struct balance_callback next, head;
		bool found = false;

		lockdep_assert_rq_held(rq);

		head = rq->balance_callback;
		while (head) {
		if (head == &balance_push_callback)
		found = true;
		next = head->next;
		head->next = NULL;
		head = next;
		}
		rq->balance_callback = found ? &balance_push_callback : NULL;
		}

		static void do_balance_callbacks(struct rq rq, struct balance_callback head)
		{
		void (func)(struct rq rq);
		@@ -6862,11 +6890,15 @@ static void __sched notrace __schedule(int sched_mode)
		rq_set_donor(rq, next);
		if (unlikely(next->blocked_on)) {
		next = find_proxy_task(rq, next, &rf);
		if (!next)
		if (!next) {
		zap_balance_callbacks(rq);
		goto pick_again;
		if (next == rq->idle)
		}
		if (next == rq->idle) {
		zap_balance_callbacks(rq);
		goto keep_resched;
		}
		}
		if (rq->donor == prev_donor && prev != next) {
		struct task_struct *donor = rq->donor;
		/*