From: Jan Kara <jack@suse.cz>

Date: Tue, 7 Mar 2023 13:42:09 +0100

Subject: [PATCH] block: do not reverse request order when flushing plug list

References: bsc#1208081 bsc#1208588 bsc#1208076

Patch-mainline: v6.3-rc3

Git-commit: 34e0a279a993debaff03158fc2fbf6a00c093643

Commit 26fed4ac4eab ("block: flush plug based on hardware and software

queue order") changed flushing of plug list to submit requests one

device at a time. However while doing that it also started using

list_add_tail() instead of list_add() used previously thus effectively

submitting requests in reverse order. Also when forming a rq_list with

remaining requests (in case two or more devices are used), we

effectively reverse the ordering of the plug list for each device we

process. Submitting requests in reverse order has negative impact on

performance for rotational disks (when BFQ is not in use). We observe

10-25% regression in random 4k write throughput on rotational storage on

btrfs filesystem.

Fix the problem by preserving ordering of the plug list when inserting

requests into the queuelist as well as by appending to requeue_list

instead of prepending to it.

Fixes: 26fed4ac4eab ("block: flush plug based on hardware and software queue order")

Signed-off-by: Jan Kara <jack@suse.cz>

Reviewed-by: Christoph Hellwig <hch@lst.de>

Link: https://lore.kernel.org/r/20230313093002.11756-1-jack@suse.cz

Signed-off-by: Jens Axboe <axboe@kernel.dk>

---

 block/blk-mq.c         | 5 +++--

 include/linux/blk-mq.h | 6 ++++++

 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c

index d0cb2ef18fe2..cf1a39adf9a5 100644

--- a/block/blk-mq.c

+++ b/block/blk-mq.c

@@ -2725,6 +2725,7 @@ static void blk_mq_dispatch_plug_list(struct blk_plug *plug, bool from_sched)

 	struct blk_mq_hw_ctx *this_hctx = NULL;

 	struct blk_mq_ctx *this_ctx = NULL;

 	struct request *requeue_list = NULL;

+	struct request **requeue_lastp = &requeue_list;

 	unsigned int depth = 0;

 	LIST_HEAD(list);

@@ -2735,10 +2736,10 @@ static void blk_mq_dispatch_plug_list(struct blk_plug *plug, bool from_sched)

 			this_hctx = rq->mq_hctx;

 			this_ctx = rq->mq_ctx;

 		} else if (this_hctx != rq->mq_hctx || this_ctx != rq->mq_ctx) {

-			rq_list_add(&requeue_list, rq);

+			rq_list_add_tail(&requeue_lastp, rq);

 			continue;

 		}

-		list_add_tail(&rq->queuelist, &list);

+		list_add(&rq->queuelist, &list);

 		depth++;

 	} while (!rq_list_empty(plug->mq_list));

diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h

index dd5ce1137f04..de0b0c3e7395 100644

--- a/include/linux/blk-mq.h

+++ b/include/linux/blk-mq.h

@@ -228,6 +228,12 @@ static inline unsigned short req_get_ioprio(struct request *req)

 	*(listptr) = rq;				\

 } while (0)

+#define rq_list_add_tail(lastpptr, rq)	do {		\

+	(rq)->rq_next = NULL;				\

+	**(lastpptr) = rq;				\

+	*(lastpptr) = &rq->rq_next;			\

+} while (0)

+

 #define rq_list_pop(listptr)				\

 ({							\

 	struct request *__req = NULL;			\

-- 

2.35.3