Commit 17666e2d authored by Jens Axboe's avatar Jens Axboe
Browse files

io_uring/tw: serialize ctx->retry_llist with ->uring_lock 



The DEFER_TASKRUN local task work paths all run under ctx->uring_lock,
which serializes them with each other and with the rest of the ring's
hot paths. io_move_task_work_from_local() is the exception - it's called
from io_ring_exit_work() on a kworker without holding the lock and from
the iopoll cancelation side right after dropping it.

->work_llist is fine with this, as it's only ever updated via the
expected paths. But the ->retry_llist is updated while runing, and hence
it could potentially race between normal task_work running and the
task-has-exited shutdown path.

Simply grab ->uring_lock while moving the local work to the fallback
list for exit purposes, which nicely serializes it across both the
normal additions and the exit prune path.

Cc: stable@vger.kernel.org
Fixes: f46b9cdb ("io_uring: limit local tw done")
Reported-by: default avatarRobert Femmer <robert.femmer@x41-dsec.de>
Reported-by: default avatarChristian Reitter <invd@inhq.net>
Reported-by: default avatarMichael Rodler <michael.rodler@x41-dsec.de>
Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
parent df8599ee

io_uring/tw.c

+11 −1
Original line number Diff line number Diff line
@@ -273,8 +273,18 @@ void io_req_task_work_add_remote(struct io_kiocb *req, unsigned flags) 


void __cold io_move_task_work_from_local(struct io_ring_ctx *ctx)

{

	struct llist_node *node = llist_del_all(&ctx->work_llist);

	struct llist_node *node;



	/*

	 * Running the work items may utilize ->retry_llist as a means

	 * for capping the number of task_work entries run at the same

	 * time. But that list can potentially race with moving the work

	 * from here, if the task is exiting. As any normal task_work

	 * running holds ->uring_lock already, just guard this slow path

	 * with ->uring_lock to avoid racing on ->retry_llist.

	 */

	guard(mutex)(&ctx->uring_lock);

	node = llist_del_all(&ctx->work_llist);

	__io_fallback_tw(node, false);

	node = llist_del_all(&ctx->retry_llist);

	__io_fallback_tw(node, false);