Commit 3afcb3b2 authored by Pavel Begunkov's avatar Pavel Begunkov Committed by Jens Axboe
Browse files

io_uring: defer iowq cqe overflow via task_work 



Don't handle CQE overflows in io_req_complete_post() and defer it to
flush_completions. It cuts some duplication, and I also want to limit
the number of places directly overflowing completions.

Signed-off-by: default avatarPavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/9046410ac27e18f2baa6f7cdb363ec921cbc3b79.1742829388.git.asml.silence@gmail.com


Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
parent 3f0cb8de

io_uring/io_uring.c

+7 −4
Original line number Diff line number Diff line
@@ -892,6 +892,7 @@ bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags) 
static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags)

{

	struct io_ring_ctx *ctx = req->ctx;

	bool completed = true;



	/*

	 * All execution paths but io-wq use the deferred completions by
@@ -905,18 +906,20 @@ static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) 
	 * the submitter task context, IOPOLL protects with uring_lock.

	 */

	if (ctx->lockless_cq || (req->flags & REQ_F_REISSUE)) {

defer_complete:

		req->io_task_work.func = io_req_task_complete;

		io_req_task_work_add(req);

		return;

	}



	io_cq_lock(ctx);

	if (!(req->flags & REQ_F_CQE_SKIP)) {

		if (!io_fill_cqe_req(ctx, req))

			io_req_cqe_overflow(req);

	}

	if (!(req->flags & REQ_F_CQE_SKIP))

		completed = io_fill_cqe_req(ctx, req);

	io_cq_unlock_post(ctx);



	if (!completed)

		goto defer_complete;



	/*

	 * We don't free the request here because we know it's called from

	 * io-wq only, which holds a reference, so it cannot be the last put.