From: Waiman Long <longman@redhat.com>

Date: Wed, 25 Jan 2023 19:36:25 -0500

Subject: [PATCH] locking/rwsem: Prevent non-first waiter from spinning in

 down_write() slowpath

References: bsc#1012628

Patch-mainline: 6.2.3

Git-commit: b613c7f31476c44316bfac1af7cac714b7d6bef9

commit b613c7f31476c44316bfac1af7cac714b7d6bef9 upstream.

A non-first waiter can potentially spin in the for loop of

rwsem_down_write_slowpath() without sleeping but fail to acquire the

lock even if the rwsem is free if the following sequence happens:

  Non-first RT waiter    First waiter      Lock holder

  -------------------    ------------      -----------

  Acquire wait_lock

  rwsem_try_write_lock():

    Set handoff bit if RT or

      wait too long

    Set waiter->handoff_set

  Release wait_lock

                         Acquire wait_lock

                         Inherit waiter->handoff_set

                         Release wait_lock

					   Clear owner

                                           Release lock

  if (waiter.handoff_set) {

    rwsem_spin_on_owner(();

    if (OWNER_NULL)

      goto trylock_again;

  }

  trylock_again:

  Acquire wait_lock

  rwsem_try_write_lock():

     if (first->handoff_set && (waiter != first))

	return false;

  Release wait_lock

A non-first waiter cannot really acquire the rwsem even if it mistakenly

believes that it can spin on OWNER_NULL value. If that waiter happens

to be an RT task running on the same CPU as the first waiter, it can

block the first waiter from acquiring the rwsem leading to live lock.

Fix this problem by making sure that a non-first waiter cannot spin in

the slowpath loop without sleeping.

Fixes: d257cc8cb8d5 ("locking/rwsem: Make handoff bit handling more consistent")

Signed-off-by: Waiman Long <longman@redhat.com>

Signed-off-by: Ingo Molnar <mingo@kernel.org>

Tested-by: Mukesh Ojha <quic_mojha@quicinc.com>

Reviewed-by: Mukesh Ojha <quic_mojha@quicinc.com>

Cc: stable@vger.kernel.org

Link: https://lore.kernel.org/r/20230126003628.365092-2-longman@redhat.com

Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

Signed-off-by: Jiri Slaby <jslaby@suse.cz>

---

 kernel/locking/rwsem.c | 19 +++++++++----------

 1 file changed, 9 insertions(+), 10 deletions(-)

diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c

index 324fc370..84d5b649 100644

--- a/kernel/locking/rwsem.c

+++ b/kernel/locking/rwsem.c

@@ -624,18 +624,16 @@ static inline bool rwsem_try_write_lock(struct rw_semaphore *sem,

 			 */

 			if (first->handoff_set && (waiter != first))

 				return false;

-

-			/*

-			 * First waiter can inherit a previously set handoff

-			 * bit and spin on rwsem if lock acquisition fails.

-			 */

-			if (waiter == first)

-				waiter->handoff_set = true;

 		}

 		new = count;

 		if (count & RWSEM_LOCK_MASK) {

+			/*

+			 * A waiter (first or not) can set the handoff bit

+			 * if it is an RT task or wait in the wait queue

+			 * for too long.

+			 */

 			if (has_handoff || (!rt_task(waiter->task) &&

 					    !time_after(jiffies, waiter->timeout)))

 				return false;

@@ -651,11 +649,12 @@ static inline bool rwsem_try_write_lock(struct rw_semaphore *sem,

 	} while (!atomic_long_try_cmpxchg_acquire(&sem->count, &count, new));

 	/*

-	 * We have either acquired the lock with handoff bit cleared or

-	 * set the handoff bit.

+	 * We have either acquired the lock with handoff bit cleared or set

+	 * the handoff bit. Only the first waiter can have its handoff_set

+	 * set here to enable optimistic spinning in slowpath loop.

 	 */

 	if (new & RWSEM_FLAG_HANDOFF) {

-		waiter->handoff_set = true;

+		first->handoff_set = true;

 		lockevent_inc(rwsem_wlock_handoff);

 		return false;

 	}

-- 

2.35.3