From: Harald Freudenberger <freude@linux.ibm.com>
Subject: s390/ap: Fix hanging ioctl caused by wrong msg counter

Patch-mainline: v5.13-rc7
Git-commit: e73a99f3287a740a07d6618e9470f4d6cb217da8
References: bsc#1188982 LTC#193818

Description:   kernel: Fix hanging ioctl caused by wrong msg counter
Symptom:       Application hang after offline/online switch of a
               crypto queue.
Problem:       When a AP queue is switched to soft offline, all
               pending requests are purged out of the pending requests
               list and 'received' by the upper layer like zcrypt
               device drivers. This is also done for requests which
               are already enqueued into the firmware queue. A request
               in a firmware queue may eventually produce an response
               message, but there is no waiting process any
               more. However, the response was counted with the
               queue_counter and as this counter was reset to 0 with
               the offline switch, the pending response caused the
               queue_counter to get negative. The next request
               increased this counter to 0 (instead of 1) which caused
               the ap code to assume there is nothing to receive and
               so the response for this valid request was never tried
               to fetch from the firmware queue.
               This all caused a queue to not work properly after a
               switch offline/online and in the end processes to hang
               forever when trying to send a crypto request after an
               queue offline/online switch cicle.
               Fixed by a) making sure the counter does not drop below
               0 and b) on a successful enqueue of a message has at
               least a value of 1.
               Additionally a warning is emitted, when a reply can't
               get assigned to a waiting process. This may be normal
               operation (process had timeout or has been killed) but
               may give a hint that something unexpected happened
               (like this odd behavior described above).
Solution:      Fixed by fixing the code.
Reproduction:  Switch all but one queue offline. Run your application
               against this only queue and while doing so switch the
               queue offline. Application will stop with ENODEV. Now
               switch back this only queue to online and restart the
               application. Without the fix it will hang unable to
               receive an reply for crypto requests. With the fix it
               will run as expected without any hangs.

Upstream-Description:

              s390/ap: Fix hanging ioctl caused by wrong msg counter

              When a AP queue is switched to soft offline, all pending
              requests are purged out of the pending requests list and
              'received' by the upper layer like zcrypt device drivers.
              This is also done for requests which are already enqueued
              into the firmware queue. A request in a firmware queue
              may eventually produce an response message, but there is
              no waiting process any more. However, the response was
              counted with the queue_counter and as this counter was
              reset to 0 with the offline switch, the pending response
              caused the queue_counter to get negative. The next request
              increased this counter to 0 (instead of 1) which caused
              the ap code to assume there is nothing to receive and so
              the response for this valid request was never tried to
              fetch from the firmware queue.

              This all caused a queue to not work properly after a
              switch offline/online and in the end processes to hang
              forever when trying to send a crypto request after an
              queue offline/online switch cicle.

              Fixed by a) making sure the counter does not drop below 0
              and b) on a successful enqueue of a message has at least
              a value of 1.

              Additionally a warning is emitted, when a reply can't get
              assigned to a waiting process. This may be normal operation
              (process had timeout or has been killed) but may give a
              hint that something unexpected happened (like this odd
              behavior described above).

              Signed-off-by: Harald Freudenberger <freude@linux.ibm.com>
              Cc: stable@vger.kernel.org
              Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>


Signed-off-by: Harald Freudenberger <freude@linux.ibm.com>
Acked-by: Michal Suchanek <msuchanek@suse.de>
---
 drivers/s390/crypto/ap_queue.c |   11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

--- a/drivers/s390/crypto/ap_queue.c
+++ b/drivers/s390/crypto/ap_queue.c
@@ -13,6 +13,7 @@
 #include <asm/facility.h>
 
 #include "ap_bus.h"
+#include "ap_debug.h"
 
 /**
  * ap_queue_enable_interruption(): Enable interruption on an AP queue.
@@ -131,12 +131,13 @@ static struct ap_queue_status ap_sm_recv
 {
 	struct ap_queue_status status;
 	struct ap_message *ap_msg;
+	bool found = false;
 
 	status = ap_dqap(aq->qid, &aq->reply->psmid,
 			 aq->reply->message, aq->reply->length);
 	switch (status.response_code) {
 	case AP_RESPONSE_NORMAL:
-		aq->queue_count--;
+		aq->queue_count = max_t(int, 0, aq->queue_count - 1);
 		if (aq->queue_count > 0)
 			mod_timer(&aq->timeout,
 				  jiffies + aq->request_timeout);
@@ -146,8 +147,14 @@ static struct ap_queue_status ap_sm_recv
 			list_del_init(&ap_msg->list);
 			aq->pendingq_count--;
 			ap_msg->receive(aq, ap_msg, aq->reply);
+			found = true;
 			break;
 		}
+		if (!found) {
+			AP_DBF(DBF_WARN, "%s unassociated reply psmid=0x%016llx on 0x%02x.%04x\n",
+				    __func__, aq->reply->psmid,
+				    AP_QID_CARD(aq->qid), AP_QID_QUEUE(aq->qid));
+		}
 	case AP_RESPONSE_NO_PENDING_REPLY:
 		if (!status.queue_empty || aq->queue_count <= 0)
 			break;
@@ -239,7 +246,7 @@ static enum ap_wait ap_sm_write(struct a
 			   ap_msg->message, ap_msg->length, ap_msg->special);
 	switch (status.response_code) {
 	case AP_RESPONSE_NORMAL:
-		aq->queue_count++;
+		aq->queue_count = max_t(int, 1, aq->queue_count + 1);
 		if (aq->queue_count == 1)
 			mod_timer(&aq->timeout, jiffies + aq->request_timeout);
 		list_move_tail(&ap_msg->list, &aq->pendingq);
