Commit 33f13714 authored by Paul E. McKenney's avatar Paul E. McKenney Committed by Steven Rostedt (Google)
Browse files

ftrace: Use asynchronous grace period for register_ftrace_direct() 

When running heavy test workloads with KASAN enabled, RCU Tasks grace
periods can extend for many tens of seconds, significantly slowing
trace registration.  Therefore, make the registration-side RCU Tasks
grace period be asynchronous via call_rcu_tasks().

Link: https://lore.kernel.org/linux-trace-kernel/ac05be77-2972-475b-9b57-56bef15aa00a@paulmck-laptop



Reported-by: default avatarJakub Kicinski <kuba@kernel.org>
Reported-by: default avatarAlexei Starovoitov <ast@kernel.org>
Reported-by: default avatarChris Mason <clm@fb.com>
Reviewed-by: default avatarMasami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: default avatarPaul E. McKenney <paulmck@kernel.org>
Signed-off-by: default avatarSteven Rostedt (Google) <rostedt@goodmis.org>
parent c5963a09

kernel/trace/ftrace.c

+9 −4
Original line number Diff line number Diff line
@@ -5366,6 +5366,13 @@ static void remove_direct_functions_hash(struct ftrace_hash *hash, unsigned long 
	}

}



static void register_ftrace_direct_cb(struct rcu_head *rhp)

{

	struct ftrace_hash *fhp = container_of(rhp, struct ftrace_hash, rcu);



	free_ftrace_hash(fhp);

}



/**

 * register_ftrace_direct - Call a custom trampoline directly

 * for multiple functions registered in @ops
@@ -5464,10 +5471,8 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) 
 out_unlock:

	mutex_unlock(&direct_mutex);



	if (free_hash && free_hash != EMPTY_HASH) {

		synchronize_rcu_tasks();

		free_ftrace_hash(free_hash);

	}

	if (free_hash && free_hash != EMPTY_HASH)

		call_rcu_tasks(&free_hash->rcu, register_ftrace_direct_cb);



	if (new_hash)

		free_ftrace_hash(new_hash);