Commit 6d885421 authored by Linus Torvalds's avatar Linus Torvalds
Browse files

Merge tag 'block-6.16-20250606' of git://git.kernel.dk/linux 

Pull more block updates from Jens Axboe:

 - NVMe pull request via Christoph:
      - TCP error handling fix (Shin'ichiro Kawasaki)
      - TCP I/O stall handling fixes (Hannes Reinecke)
      - fix command limits status code (Keith Busch)
      - support vectored buffers also for passthrough (Pavel Begunkov)
      - spelling fixes (Yi Zhang)

 - MD pull request via Yu:
      - fix REQ_RAHEAD and REQ_NOWAIT IO err handling for raid1/10
      - fix max_write_behind setting for dm-raid
      - some minor cleanups

 - Integrity data direction fix and cleanup

 - bcache NULL pointer fix

 - Fix for loop missing write start/end handling

 - Decouple hardware queues and IO threads in ublk

 - Slew of ublk selftests additions and updates

* tag 'block-6.16-20250606' of git://git.kernel.dk/linux: (29 commits)
  nvme: spelling fixes
  nvme-tcp: fix I/O stalls on congested sockets
  nvme-tcp: sanitize request list handling
  nvme-tcp: remove tag set when second admin queue config fails
  nvme: enable vectored registered bufs for passthrough cmds
  nvme: fix implicit bool to flags conversion
  nvme: fix command limits status code
  selftests: ublk: kublk: improve behavior on init failure
  block: flip iter directions in blk_rq_integrity_map_user()
  block: drop direction param from bio_integrity_copy_user()
  selftests: ublk: cover PER_IO_DAEMON in more stress tests
  Documentation: ublk: document UBLK_F_PER_IO_DAEMON
  selftests: ublk: add stress test for per io daemons
  selftests: ublk: add functional test for per io daemons
  selftests: ublk: kublk: decouple ublk_queues from ublk server threads
  selftests: ublk: kublk: move per-thread data out of ublk_queue
  selftests: ublk: kublk: lift queue initialization out of thread
  selftests: ublk: kublk: tie sqe allocation to io instead of queue
  selftests: ublk: kublk: plumb q_id in io_uring user_data
  ublk: have a per-io daemon instead of a per-queue daemon
  ...
parents 794a5492 6f65947a

Documentation/block/ublk.rst

+24 −11
Original line number Diff line number Diff line
@@ -115,15 +115,15 @@ managing and controlling ublk devices with help of several control commands: 


- ``UBLK_CMD_START_DEV``



  After the server prepares userspace resources (such as creating per-queue

  pthread & io_uring for handling ublk IO), this command is sent to the

  After the server prepares userspace resources (such as creating I/O handler

  threads & io_uring for handling ublk IO), this command is sent to the

  driver for allocating & exposing ``/dev/ublkb*``. Parameters set via

  ``UBLK_CMD_SET_PARAMS`` are applied for creating the device.



- ``UBLK_CMD_STOP_DEV``



  Halt IO on ``/dev/ublkb*`` and remove the device. When this command returns,

  ublk server will release resources (such as destroying per-queue pthread &

  ublk server will release resources (such as destroying I/O handler threads &

  io_uring).



- ``UBLK_CMD_DEL_DEV``
@@ -208,15 +208,15 @@ managing and controlling ublk devices with help of several control commands: 
  modify how I/O is handled while the ublk server is dying/dead (this is called

  the ``nosrv`` case in the driver code).



  With just ``UBLK_F_USER_RECOVERY`` set, after one ubq_daemon(ublk server's io

  handler) is dying, ublk does not delete ``/dev/ublkb*`` during the whole

  With just ``UBLK_F_USER_RECOVERY`` set, after the ublk server exits,

  ublk does not delete ``/dev/ublkb*`` during the whole

  recovery stage and ublk device ID is kept. It is ublk server's

  responsibility to recover the device context by its own knowledge.

  Requests which have not been issued to userspace are requeued. Requests

  which have been issued to userspace are aborted.



  With ``UBLK_F_USER_RECOVERY_REISSUE`` additionally set, after one ubq_daemon

  (ublk server's io handler) is dying, contrary to ``UBLK_F_USER_RECOVERY``,

  With ``UBLK_F_USER_RECOVERY_REISSUE`` additionally set, after the ublk server

  exits, contrary to ``UBLK_F_USER_RECOVERY``,

  requests which have been issued to userspace are requeued and will be

  re-issued to the new process after handling ``UBLK_CMD_END_USER_RECOVERY``.

  ``UBLK_F_USER_RECOVERY_REISSUE`` is designed for backends who tolerate
@@ -241,10 +241,11 @@ can be controlled/accessed just inside this container. 
Data plane

----------



ublk server needs to create per-queue IO pthread & io_uring for handling IO

commands via io_uring passthrough. The per-queue IO pthread

focuses on IO handling and shouldn't handle any control & management

tasks.

The ublk server should create dedicated threads for handling I/O. Each

thread should have its own io_uring through which it is notified of new

I/O, and through which it can complete I/O. These dedicated threads

should focus on IO handling and shouldn't handle any control &

management tasks.



The's IO is assigned by a unique tag, which is 1:1 mapping with IO

request of ``/dev/ublkb*``.
@@ -265,6 +266,18 @@ with specified IO tag in the command data: 
  destined to ``/dev/ublkb*``. This command is sent only once from the server

  IO pthread for ublk driver to setup IO forward environment.



  Once a thread issues this command against a given (qid,tag) pair, the thread

  registers itself as that I/O's daemon. In the future, only that I/O's daemon

  is allowed to issue commands against the I/O. If any other thread attempts

  to issue a command against a (qid,tag) pair for which the thread is not the

  daemon, the command will fail. Daemons can be reset only be going through

  recovery.



  The ability for every (qid,tag) pair to have its own independent daemon task

  is indicated by the ``UBLK_F_PER_IO_DAEMON`` feature. If this feature is not

  supported by the driver, daemons must be per-queue instead - i.e. all I/Os

  associated to a single qid must be handled by the same task.



- ``UBLK_IO_COMMIT_AND_FETCH_REQ``



  When an IO request is destined to ``/dev/ublkb*``, the driver stores

block/bio-integrity.c

+5 −12
Original line number Diff line number Diff line
@@ -154,10 +154,9 @@ int bio_integrity_add_page(struct bio *bio, struct page *page, 
EXPORT_SYMBOL(bio_integrity_add_page);



static int bio_integrity_copy_user(struct bio *bio, struct bio_vec *bvec,

				   int nr_vecs, unsigned int len,

				   unsigned int direction)

				   int nr_vecs, unsigned int len)

{

	bool write = direction == ITER_SOURCE;

	bool write = op_is_write(bio_op(bio));

	struct bio_integrity_payload *bip;

	struct iov_iter iter;

	void *buf;
@@ -168,7 +167,7 @@ static int bio_integrity_copy_user(struct bio *bio, struct bio_vec *bvec, 
		return -ENOMEM;



	if (write) {

		iov_iter_bvec(&iter, direction, bvec, nr_vecs, len);

		iov_iter_bvec(&iter, ITER_SOURCE, bvec, nr_vecs, len);

		if (!copy_from_iter_full(buf, len, &iter)) {

			ret = -EFAULT;

			goto free_buf;
@@ -264,7 +263,7 @@ int bio_integrity_map_user(struct bio *bio, struct iov_iter *iter) 
	struct page *stack_pages[UIO_FASTIOV], **pages = stack_pages;

	struct bio_vec stack_vec[UIO_FASTIOV], *bvec = stack_vec;

	size_t offset, bytes = iter->count;

	unsigned int direction, nr_bvecs;

	unsigned int nr_bvecs;

	int ret, nr_vecs;

	bool copy;
@@ -273,11 +272,6 @@ int bio_integrity_map_user(struct bio *bio, struct iov_iter *iter) 
	if (bytes >> SECTOR_SHIFT > queue_max_hw_sectors(q))

		return -E2BIG;



	if (bio_data_dir(bio) == READ)

		direction = ITER_DEST;

	else

		direction = ITER_SOURCE;



	nr_vecs = iov_iter_npages(iter, BIO_MAX_VECS + 1);

	if (nr_vecs > BIO_MAX_VECS)

		return -E2BIG;
@@ -300,8 +294,7 @@ int bio_integrity_map_user(struct bio *bio, struct iov_iter *iter) 
		copy = true;



	if (copy)

		ret = bio_integrity_copy_user(bio, bvec, nr_bvecs, bytes,

					      direction);

		ret = bio_integrity_copy_user(bio, bvec, nr_bvecs, bytes);

	else

		ret = bio_integrity_init_user(bio, bvec, nr_bvecs, bytes);

	if (ret)

block/blk-integrity.c

+1 −6
Original line number Diff line number Diff line
@@ -117,13 +117,8 @@ int blk_rq_integrity_map_user(struct request *rq, void __user *ubuf, 
{

	int ret;

	struct iov_iter iter;

	unsigned int direction;



	if (op_is_write(req_op(rq)))

		direction = ITER_DEST;

	else

		direction = ITER_SOURCE;

	iov_iter_ubuf(&iter, direction, ubuf, bytes);

	iov_iter_ubuf(&iter, rq_data_dir(rq), ubuf, bytes);

	ret = bio_integrity_map_user(rq->bio, &iter);

	if (ret)

		return ret;

drivers/block/loop.c

+6 −2
Original line number Diff line number Diff line
@@ -308,11 +308,14 @@ static void lo_complete_rq(struct request *rq) 
static void lo_rw_aio_do_completion(struct loop_cmd *cmd)

{

	struct request *rq = blk_mq_rq_from_pdu(cmd);

	struct loop_device *lo = rq->q->queuedata;



	if (!atomic_dec_and_test(&cmd->ref))

		return;

	kfree(cmd->bvec);

	cmd->bvec = NULL;

	if (req_op(rq) == REQ_OP_WRITE)

		file_end_write(lo->lo_backing_file);

	if (likely(!blk_should_fake_timeout(rq->q)))

		blk_mq_complete_request(rq);

}
@@ -387,9 +390,10 @@ static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd, 
		cmd->iocb.ki_flags = 0;

	}



	if (rw == ITER_SOURCE)

	if (rw == ITER_SOURCE) {

		file_start_write(lo->lo_backing_file);

		ret = file->f_op->write_iter(&cmd->iocb, &iter);

	else

	} else

		ret = file->f_op->read_iter(&cmd->iocb, &iter);



	lo_rw_aio_do_completion(cmd);

drivers/block/ublk_drv.c

+56 −55
Original line number Diff line number Diff line
@@ -69,7 +69,8 @@ 
		| UBLK_F_USER_RECOVERY_FAIL_IO \

		| UBLK_F_UPDATE_SIZE \

		| UBLK_F_AUTO_BUF_REG \

		| UBLK_F_QUIESCE)

		| UBLK_F_QUIESCE \

		| UBLK_F_PER_IO_DAEMON)



#define UBLK_F_ALL_RECOVERY_FLAGS (UBLK_F_USER_RECOVERY \

		| UBLK_F_USER_RECOVERY_REISSUE \
@@ -166,6 +167,8 @@ struct ublk_io { 
		/* valid if UBLK_IO_FLAG_OWNED_BY_SRV is set */

		struct request *req;

	};



	struct task_struct *task;

};



struct ublk_queue {
@@ -173,11 +176,9 @@ struct ublk_queue { 
	int q_depth;



	unsigned long flags;

	struct task_struct	*ubq_daemon;

	struct ublksrv_io_desc *io_cmd_buf;



	bool force_abort;

	bool timeout;

	bool canceling;

	bool fail_io; /* copy of dev->state == UBLK_S_DEV_FAIL_IO */

	unsigned short nr_io_ready;	/* how many ios setup */
@@ -1099,11 +1100,6 @@ static inline struct ublk_uring_cmd_pdu *ublk_get_uring_cmd_pdu( 
	return io_uring_cmd_to_pdu(ioucmd, struct ublk_uring_cmd_pdu);

}



static inline bool ubq_daemon_is_dying(struct ublk_queue *ubq)

{

	return !ubq->ubq_daemon || ubq->ubq_daemon->flags & PF_EXITING;

}



/* todo: handle partial completion */

static inline void __ublk_complete_rq(struct request *req)

{
@@ -1275,13 +1271,13 @@ static void ublk_dispatch_req(struct ublk_queue *ubq, 
	/*

	 * Task is exiting if either:

	 *

	 * (1) current != ubq_daemon.

	 * (1) current != io->task.

	 * io_uring_cmd_complete_in_task() tries to run task_work

	 * in a workqueue if ubq_daemon(cmd's task) is PF_EXITING.

	 * in a workqueue if cmd's task is PF_EXITING.

	 *

	 * (2) current->flags & PF_EXITING.

	 */

	if (unlikely(current != ubq->ubq_daemon || current->flags & PF_EXITING)) {

	if (unlikely(current != io->task || current->flags & PF_EXITING)) {

		__ublk_abort_rq(ubq, req);

		return;

	}
@@ -1330,24 +1326,22 @@ static void ublk_cmd_list_tw_cb(struct io_uring_cmd *cmd, 
{

	struct ublk_uring_cmd_pdu *pdu = ublk_get_uring_cmd_pdu(cmd);

	struct request *rq = pdu->req_list;

	struct ublk_queue *ubq = pdu->ubq;

	struct request *next;



	do {

		next = rq->rq_next;

		rq->rq_next = NULL;

		ublk_dispatch_req(ubq, rq, issue_flags);

		ublk_dispatch_req(rq->mq_hctx->driver_data, rq, issue_flags);

		rq = next;

	} while (rq);

}



static void ublk_queue_cmd_list(struct ublk_queue *ubq, struct rq_list *l)

static void ublk_queue_cmd_list(struct ublk_io *io, struct rq_list *l)

{

	struct request *rq = rq_list_peek(l);

	struct io_uring_cmd *cmd = ubq->ios[rq->tag].cmd;

	struct io_uring_cmd *cmd = io->cmd;

	struct ublk_uring_cmd_pdu *pdu = ublk_get_uring_cmd_pdu(cmd);



	pdu->req_list = rq;

	pdu->req_list = rq_list_peek(l);

	rq_list_init(l);

	io_uring_cmd_complete_in_task(cmd, ublk_cmd_list_tw_cb);

}
@@ -1355,13 +1349,10 @@ static void ublk_queue_cmd_list(struct ublk_queue *ubq, struct rq_list *l) 
static enum blk_eh_timer_return ublk_timeout(struct request *rq)

{

	struct ublk_queue *ubq = rq->mq_hctx->driver_data;

	struct ublk_io *io = &ubq->ios[rq->tag];



	if (ubq->flags & UBLK_F_UNPRIVILEGED_DEV) {

		if (!ubq->timeout) {

			send_sig(SIGKILL, ubq->ubq_daemon, 0);

			ubq->timeout = true;

		}



		send_sig(SIGKILL, io->task, 0);

		return BLK_EH_DONE;

	}
@@ -1429,24 +1420,25 @@ static void ublk_queue_rqs(struct rq_list *rqlist) 
{

	struct rq_list requeue_list = { };

	struct rq_list submit_list = { };

	struct ublk_queue *ubq = NULL;

	struct ublk_io *io = NULL;

	struct request *req;



	while ((req = rq_list_pop(rqlist))) {

		struct ublk_queue *this_q = req->mq_hctx->driver_data;

		struct ublk_io *this_io = &this_q->ios[req->tag];



		if (ubq && ubq != this_q && !rq_list_empty(&submit_list))

			ublk_queue_cmd_list(ubq, &submit_list);

		ubq = this_q;

		if (io && io->task != this_io->task && !rq_list_empty(&submit_list))

			ublk_queue_cmd_list(io, &submit_list);

		io = this_io;



		if (ublk_prep_req(ubq, req, true) == BLK_STS_OK)

		if (ublk_prep_req(this_q, req, true) == BLK_STS_OK)

			rq_list_add_tail(&submit_list, req);

		else

			rq_list_add_tail(&requeue_list, req);

	}



	if (ubq && !rq_list_empty(&submit_list))

		ublk_queue_cmd_list(ubq, &submit_list);

	if (!rq_list_empty(&submit_list))

		ublk_queue_cmd_list(io, &submit_list);

	*rqlist = requeue_list;

}
@@ -1474,17 +1466,6 @@ static void ublk_queue_reinit(struct ublk_device *ub, struct ublk_queue *ubq) 
	/* All old ioucmds have to be completed */

	ubq->nr_io_ready = 0;



	/*

	 * old daemon is PF_EXITING, put it now

	 *

	 * It could be NULL in case of closing one quisced device.

	 */

	if (ubq->ubq_daemon)

		put_task_struct(ubq->ubq_daemon);

	/* We have to reset it to NULL, otherwise ub won't accept new FETCH_REQ */

	ubq->ubq_daemon = NULL;

	ubq->timeout = false;



	for (i = 0; i < ubq->q_depth; i++) {

		struct ublk_io *io = &ubq->ios[i];
@@ -1495,6 +1476,17 @@ static void ublk_queue_reinit(struct ublk_device *ub, struct ublk_queue *ubq) 
		io->flags &= UBLK_IO_FLAG_CANCELED;

		io->cmd = NULL;

		io->addr = 0;



		/*

		 * old task is PF_EXITING, put it now

		 *

		 * It could be NULL in case of closing one quiesced

		 * device.

		 */

		if (io->task) {

			put_task_struct(io->task);

			io->task = NULL;

		}

	}

}
@@ -1516,7 +1508,7 @@ static void ublk_reset_ch_dev(struct ublk_device *ub) 
	for (i = 0; i < ub->dev_info.nr_hw_queues; i++)

		ublk_queue_reinit(ub, ublk_get_queue(ub, i));



	/* set to NULL, otherwise new ubq_daemon cannot mmap the io_cmd_buf */

	/* set to NULL, otherwise new tasks cannot mmap io_cmd_buf */

	ub->mm = NULL;

	ub->nr_queues_ready = 0;

	ub->nr_privileged_daemon = 0;
@@ -1783,6 +1775,7 @@ static void ublk_uring_cmd_cancel_fn(struct io_uring_cmd *cmd, 
	struct ublk_uring_cmd_pdu *pdu = ublk_get_uring_cmd_pdu(cmd);

	struct ublk_queue *ubq = pdu->ubq;

	struct task_struct *task;

	struct ublk_io *io;



	if (WARN_ON_ONCE(!ubq))

		return;
@@ -1791,13 +1784,14 @@ static void ublk_uring_cmd_cancel_fn(struct io_uring_cmd *cmd, 
		return;



	task = io_uring_cmd_get_task(cmd);

	if (WARN_ON_ONCE(task && task != ubq->ubq_daemon))

	io = &ubq->ios[pdu->tag];

	if (WARN_ON_ONCE(task && task != io->task))

		return;



	if (!ubq->canceling)

		ublk_start_cancel(ubq);



	WARN_ON_ONCE(ubq->ios[pdu->tag].cmd != cmd);

	WARN_ON_ONCE(io->cmd != cmd);

	ublk_cancel_cmd(ubq, pdu->tag, issue_flags);

}
@@ -1930,8 +1924,6 @@ static void ublk_mark_io_ready(struct ublk_device *ub, struct ublk_queue *ubq) 
{

	ubq->nr_io_ready++;

	if (ublk_queue_ready(ubq)) {

		ubq->ubq_daemon = current;

		get_task_struct(ubq->ubq_daemon);

		ub->nr_queues_ready++;



		if (capable(CAP_SYS_ADMIN))
@@ -2084,6 +2076,7 @@ static int ublk_fetch(struct io_uring_cmd *cmd, struct ublk_queue *ubq, 
	}



	ublk_fill_io_cmd(io, cmd, buf_addr);

	WRITE_ONCE(io->task, get_task_struct(current));

	ublk_mark_io_ready(ub, ubq);

out:

	mutex_unlock(&ub->mutex);
@@ -2179,6 +2172,7 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd *cmd, 
			       const struct ublksrv_io_cmd *ub_cmd)

{

	struct ublk_device *ub = cmd->file->private_data;

	struct task_struct *task;

	struct ublk_queue *ubq;

	struct ublk_io *io;

	u32 cmd_op = cmd->cmd_op;
@@ -2193,13 +2187,14 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd *cmd, 
		goto out;



	ubq = ublk_get_queue(ub, ub_cmd->q_id);

	if (ubq->ubq_daemon && ubq->ubq_daemon != current)

		goto out;



	if (tag >= ubq->q_depth)

		goto out;



	io = &ubq->ios[tag];

	task = READ_ONCE(io->task);

	if (task && task != current)

		goto out;



	/* there is pending io cmd, something must be wrong */

	if (io->flags & UBLK_IO_FLAG_ACTIVE) {
@@ -2449,9 +2444,14 @@ static void ublk_deinit_queue(struct ublk_device *ub, int q_id) 
{

	int size = ublk_queue_cmd_buf_size(ub, q_id);

	struct ublk_queue *ubq = ublk_get_queue(ub, q_id);

	int i;



	for (i = 0; i < ubq->q_depth; i++) {

		struct ublk_io *io = &ubq->ios[i];

		if (io->task)

			put_task_struct(io->task);

	}



	if (ubq->ubq_daemon)

		put_task_struct(ubq->ubq_daemon);

	if (ubq->io_cmd_buf)

		free_pages((unsigned long)ubq->io_cmd_buf, get_order(size));

}
@@ -2923,7 +2923,8 @@ static int ublk_ctrl_add_dev(const struct ublksrv_ctrl_cmd *header) 
	ub->dev_info.flags &= UBLK_F_ALL;



	ub->dev_info.flags |= UBLK_F_CMD_IOCTL_ENCODE |

		UBLK_F_URING_CMD_COMP_IN_TASK;

		UBLK_F_URING_CMD_COMP_IN_TASK |

		UBLK_F_PER_IO_DAEMON;



	/* GET_DATA isn't needed any more with USER_COPY or ZERO COPY */

	if (ub->dev_info.flags & (UBLK_F_USER_COPY | UBLK_F_SUPPORT_ZERO_COPY |
@@ -3188,14 +3189,14 @@ static int ublk_ctrl_end_recovery(struct ublk_device *ub, 
	int ublksrv_pid = (int)header->data[0];

	int ret = -EINVAL;



	pr_devel("%s: Waiting for new ubq_daemons(nr: %d) are ready, dev id %d...\n",

			__func__, ub->dev_info.nr_hw_queues, header->dev_id);

	/* wait until new ubq_daemon sending all FETCH_REQ */

	pr_devel("%s: Waiting for all FETCH_REQs, dev id %d...\n", __func__,

		 header->dev_id);



	if (wait_for_completion_interruptible(&ub->completion))

		return -EINTR;



	pr_devel("%s: All new ubq_daemons(nr: %d) are ready, dev id %d\n",

			__func__, ub->dev_info.nr_hw_queues, header->dev_id);

	pr_devel("%s: All FETCH_REQs received, dev id %d\n", __func__,

		 header->dev_id);



	mutex_lock(&ub->mutex);

	if (ublk_nosrv_should_stop_dev(ub))