[RFC PATCH v1 0/3] fix header length on skb merging

Hello,

this patchset fixes skbuff merging during rx path. Problem fires when
we are trying to append data to skbuff which is processed in dequeue
callback at the same time. Dequeue callback calls 'skb_pull()' which
changes 'skb->len'. While appending data, this dynamic 'skb->len' will
be used to update length in header of last skbuff. This is wrong, because
length in header is used to update credit parameters ('rx_bytes' and
'fwd_cnt') and must be constant. To set valid length in header of last
skbuff after appending new data to it, we need to sum header values from
both last and new skbuff.

This bug was introduced by:
Fixes: 077706165717 ("virtio/vsock: don't use skbuff state to account credit")

I really forgot about this branch in rx path when implemented patch
above.

This patchset contains 3 patches:
1) Fix itself.
2) Patch with WARN() as kernel part of reproducer. I've added error
   return from dequeue callback if this bug fires, otherwise you'll
   get busyloop in kernel: callback always returns 0, but rx loop in
   af_vsock.c sees that rx_bytes is non-zero thus trying to call
   dequeue callback again and again.
3) Patch with reproducer in vsock_test.c. It looks like new test, but
   i'm not sure how to test this branch (appending data to last skbuff)
   of virtio transport. So only way to detect problem is WARN() in 2).

May be, it will be good practice to add some WARN() checks like in 2)
to different parts of virtio/vsock, because such bugs are difficult to
detect.

Arseniy Krasnov (3):
  virtio/vsock: fix header length on skb merging
  virtio/vsock: add WARN() for invalid state of socket
  test/vsock: skbuff merging test

 net/vmw_vsock/virtio_transport_common.c |  9 ++-
 tools/testing/vsock/vsock_test.c        | 81 +++++++++++++++++++++++++
 2 files changed, 89 insertions(+), 1 deletion(-)

-- 
2.25.1

This fixes header length calculation of skbuff during data appending to
it. When such skbuff is processed in dequeue callbacks, e.g. 'skb_pull()'
is called on it, 'skb->len' is dynamic value, so it is impossible to use
it in header, because value from header must be permanent for valid
credit calculation ('rx_bytes'/'fwd_cnt').

Fixes: 077706165717 ("virtio/vsock: don't use skbuff state to account credit")
Signed-off-by: Arseniy Krasnov <redacted>
---
 net/vmw_vsock/virtio_transport_common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
index 6d15cd4d090a..3c75986e16c2 100644
--- a/net/vmw_vsock/virtio_transport_common.c
+++ b/net/vmw_vsock/virtio_transport_common.c

@@ -1091,7 +1091,7 @@ virtio_transport_recv_enqueue(struct vsock_sock *vsk,
 			memcpy(skb_put(last_skb, skb->len), skb->data, skb->len);
 			free_pkt = true;
 			last_hdr->flags |= hdr->flags;
-			last_hdr->len = cpu_to_le32(last_skb->len);
+			le32_add_cpu(&last_hdr->len, len);
 			goto out;
 		}
 	}

-- 
2.25.1

This prints WARN() and returns from stream dequeue callback when socket's
queue is empty, but 'rx_bytes' still non-zero.

Signed-off-by: Arseniy Krasnov <redacted>
---
 net/vmw_vsock/virtio_transport_common.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
index 3c75986e16c2..c35b03adad8d 100644
--- a/net/vmw_vsock/virtio_transport_common.c
+++ b/net/vmw_vsock/virtio_transport_common.c

@@ -388,6 +388,13 @@ virtio_transport_stream_do_dequeue(struct vsock_sock *vsk,
 	u32 free_space;
 
 	spin_lock_bh(&vvs->rx_lock);
+
+	if (skb_queue_empty(&vvs->rx_queue) && vvs->rx_bytes) {
+		WARN(1, "No skbuffs with non-zero 'rx_bytes'\n");
+		spin_unlock_bh(&vvs->rx_lock);
+		return err;
+	}
+
 	while (total < len && !skb_queue_empty(&vvs->rx_queue)) {
 		skb = skb_peek(&vvs->rx_queue);
 

-- 
2.25.1

This adds test which checks case when data of newly received skbuff is
appended to the last skbuff in the socket's queue.

This test is actual only for virtio transport.

Signed-off-by: Arseniy Krasnov <redacted>
---
 tools/testing/vsock/vsock_test.c | 81 ++++++++++++++++++++++++++++++++
 1 file changed, 81 insertions(+)

diff --git a/tools/testing/vsock/vsock_test.c b/tools/testing/vsock/vsock_test.c
index 3de10dbb50f5..00216c52d8b6 100644
--- a/tools/testing/vsock/vsock_test.c
+++ b/tools/testing/vsock/vsock_test.c

@@ -968,6 +968,82 @@ static void test_seqpacket_inv_buf_server(const struct test_opts *opts)
 	test_inv_buf_server(opts, false);
 }
 
+static void test_stream_virtio_skb_merge_client(const struct test_opts *opts)
+{
+	ssize_t res;
+	int fd;
+
+	fd = vsock_stream_connect(opts->peer_cid, 1234);
+	if (fd < 0) {
+		perror("connect");
+		exit(EXIT_FAILURE);
+	}
+
+	res = send(fd, "HELLO", strlen("HELLO"), 0);
+	if (res != strlen("HELLO")) {
+		fprintf(stderr, "unexpected send(2) result %zi\n", res);
+		exit(EXIT_FAILURE);
+	}
+
+	control_writeln("SEND0");
+	/* Peer reads part of first packet. */
+	control_expectln("REPLY0");
+
+	/* Send second skbuff, it will be merged. */
+	res = send(fd, "WORLD", strlen("WORLD"), 0);
+	if (res != strlen("WORLD")) {
+		fprintf(stderr, "unexpected send(2) result %zi\n", res);
+		exit(EXIT_FAILURE);
+	}
+
+	control_writeln("SEND1");
+	/* Peer reads merged skbuff packet. */
+	control_expectln("REPLY1");
+
+	close(fd);
+}
+
+static void test_stream_virtio_skb_merge_server(const struct test_opts *opts)
+{
+	unsigned char buf[64];
+	ssize_t res;
+	int fd;
+
+	fd = vsock_stream_accept(VMADDR_CID_ANY, 1234, NULL);
+	if (fd < 0) {
+		perror("accept");
+		exit(EXIT_FAILURE);
+	}
+
+	control_expectln("SEND0");
+
+	/* Read skbuff partially. */
+	res = recv(fd, buf, 2, 0);
+	if (res != 2) {
+		fprintf(stderr, "expected recv(2) failure, got %zi\n", res);
+		exit(EXIT_FAILURE);
+	}
+
+	control_writeln("REPLY0");
+	control_expectln("SEND1");
+
+	res = recv(fd, buf, sizeof(buf), 0);
+	if (res != 8) {
+		fprintf(stderr, "expected recv(2) failure, got %zi\n", res);
+		exit(EXIT_FAILURE);
+	}
+
+	res = recv(fd, buf, sizeof(buf), MSG_DONTWAIT);
+	if (res != -1) {
+		fprintf(stderr, "expected recv(2) success, got %zi\n", res);
+		exit(EXIT_FAILURE);
+	}
+
+	control_writeln("REPLY1");
+
+	close(fd);
+}
+
 static struct test_case test_cases[] = {
 	{
 		.name = "SOCK_STREAM connection reset",

@@ -1038,6 +1114,11 @@ static struct test_case test_cases[] = {
 		.run_client = test_seqpacket_inv_buf_client,
 		.run_server = test_seqpacket_inv_buf_server,
 	},
+	{
+		.name = "SOCK_STREAM virtio skb merge",
+		.run_client = test_stream_virtio_skb_merge_client,
+		.run_server = test_stream_virtio_skb_merge_server,
+	},
 	{},
 };
 

-- 
2.25.1

On Sun, Mar 19, 2023 at 09:51:06PM +0300, Arseniy Krasnov wrote:

This fixes header length calculation of skbuff during data appending to
it. When such skbuff is processed in dequeue callbacks, e.g. 'skb_pull()'
is called on it, 'skb->len' is dynamic value, so it is impossible to use
it in header, because value from header must be permanent for valid
credit calculation ('rx_bytes'/'fwd_cnt').

Fixes: 077706165717 ("virtio/vsock: don't use skbuff state to account credit")

I don't understand how this commit introduced this problem, can you
explain it better?

Is it related more to the credit than to the size in the header itself?

Anyway, the patch LGTM, but we should explain better the issue.

Thanks,
Stefano

On Sun, Mar 19, 2023 at 09:52:19PM +0300, Arseniy Krasnov wrote:

I would use WARN_ONCE, since we can't recover so we will flood the log.

And you can put the condition in the first argument, I mean something
like this:
         if (WARN_ONCE(skb_queue_empty(&vvs->rx_queue) && vvs->rx_bytes,
                       "rx_queue is empty, but rx_bytes is non-zero\n")) {

Thanks,
Stefano

+		spin_unlock_bh(&vvs->rx_lock);
+		return err;
+	}
+
	while (total < len && !skb_queue_empty(&vvs->rx_queue)) {
		skb = skb_peek(&vvs->rx_queue);

-- 
2.25.1

On Sun, Mar 19, 2023 at 09:53:54PM +0300, Arseniy Krasnov wrote:

Please use a macro for "HELLO" or a variabile, e.g.

         char *buf;
         ...

         buf = "HELLO";
         res = send(fd, buf, strlen(buf), 0);
         ...

+	res = send(fd, "HELLO", strlen("HELLO"), 0);
+	if (res != strlen("HELLO")) {
+		fprintf(stderr, "unexpected send(2) result %zi\n", res);
+		exit(EXIT_FAILURE);
+	}
+
+	control_writeln("SEND0");
+	/* Peer reads part of first packet. */
+	control_expectln("REPLY0");
+
+	/* Send second skbuff, it will be merged. */
+	res = send(fd, "WORLD", strlen("WORLD"), 0);

Ditto.

+	if (res != strlen("WORLD")) {
+		fprintf(stderr, "unexpected send(2) result %zi\n", res);
+		exit(EXIT_FAILURE);
+	}
+
+	control_writeln("SEND1");
+	/* Peer reads merged skbuff packet. */
+	control_expectln("REPLY1");
+
+	close(fd);
+}
+
+static void test_stream_virtio_skb_merge_server(const struct test_opts *opts)
+{
+	unsigned char buf[64];
+	ssize_t res;
+	int fd;
+
+	fd = vsock_stream_accept(VMADDR_CID_ANY, 1234, NULL);
+	if (fd < 0) {
+		perror("accept");
+		exit(EXIT_FAILURE);
+	}
+
+	control_expectln("SEND0");
+
+	/* Read skbuff partially. */
+	res = recv(fd, buf, 2, 0);
+	if (res != 2) {
+		fprintf(stderr, "expected recv(2) failure, got %zi\n", res);

We don't expect a failure, so please update the error message and make
it easy to figure out which recv() is failing. For example by saying
how many bytes you expected and how many you received.

+		exit(EXIT_FAILURE);
+	}
+
+	control_writeln("REPLY0");
+	control_expectln("SEND1");
+
+
+	res = recv(fd, buf, sizeof(buf), 0);

Perhaps a comment here to explain why we expect only 8 bytes.

+	if (res != 8) {
+		fprintf(stderr, "expected recv(2) failure, got %zi\n", res);

Ditto.

+		exit(EXIT_FAILURE);
+	}
+
+	res = recv(fd, buf, sizeof(buf), MSG_DONTWAIT);
+	if (res != -1) {
+		fprintf(stderr, "expected recv(2) success, got %zi\n", res);

It's the other way around, isn't it?
Here you expect it to fail instead it is not failing.

+		exit(EXIT_FAILURE);
+	}

Moving the pointer correctly, I would also check that there is
HELLOWORLD in the buffer.

Thanks for adding tests in this suite!
Stefano

On 20.03.2023 18:31, Stefano Garzarella wrote:

On Sun, Mar 19, 2023 at 09:53:54PM +0300, Arseniy Krasnov wrote:

quoted

This adds test which checks case when data of newly received skbuff is
appended to the last skbuff in the socket's queue.

This test is actual only for virtio transport.

Signed-off-by: Arseniy Krasnov <redacted>
---
tools/testing/vsock/vsock_test.c | 81 ++++++++++++++++++++++++++++++++
1 file changed, 81 insertions(+)

diff --git a/tools/testing/vsock/vsock_test.c b/tools/testing/vsock/vsock_test.c
index 3de10dbb50f5..00216c52d8b6 100644
--- a/tools/testing/vsock/vsock_test.c
+++ b/tools/testing/vsock/vsock_test.c

@@ -968,6 +968,82 @@ static void test_seqpacket_inv_buf_server(const struct test_opts *opts)

    test_inv_buf_server(opts, false);
}

+static void test_stream_virtio_skb_merge_client(const struct test_opts *opts)
+{
+    ssize_t res;
+    int fd;
+
+    fd = vsock_stream_connect(opts->peer_cid, 1234);
+    if (fd < 0) {
+        perror("connect");
+        exit(EXIT_FAILURE);
+    }
+

Please use a macro for "HELLO" or a variabile, e.g.

        char *buf;
        ...

        buf = "HELLO";
        res = send(fd, buf, strlen(buf), 0);
        ...

quoted

+    res = send(fd, "HELLO", strlen("HELLO"), 0);
+    if (res != strlen("HELLO")) {
+        fprintf(stderr, "unexpected send(2) result %zi\n", res);
+        exit(EXIT_FAILURE);
+    }
+
+    control_writeln("SEND0");
+    /* Peer reads part of first packet. */
+    control_expectln("REPLY0");
+
+    /* Send second skbuff, it will be merged. */
+    res = send(fd, "WORLD", strlen("WORLD"), 0);

Ditto.

quoted

+    if (res != strlen("WORLD")) {
+        fprintf(stderr, "unexpected send(2) result %zi\n", res);
+        exit(EXIT_FAILURE);
+    }
+
+    control_writeln("SEND1");
+    /* Peer reads merged skbuff packet. */
+    control_expectln("REPLY1");
+
+    close(fd);
+}
+
+static void test_stream_virtio_skb_merge_server(const struct test_opts *opts)
+{
+    unsigned char buf[64];
+    ssize_t res;
+    int fd;
+
+    fd = vsock_stream_accept(VMADDR_CID_ANY, 1234, NULL);
+    if (fd < 0) {
+        perror("accept");
+        exit(EXIT_FAILURE);
+    }
+
+    control_expectln("SEND0");
+
+    /* Read skbuff partially. */
+    res = recv(fd, buf, 2, 0);
+    if (res != 2) {
+        fprintf(stderr, "expected recv(2) failure, got %zi\n", res);

We don't expect a failure, so please update the error message and make
it easy to figure out which recv() is failing. For example by saying
how many bytes you expected and how many you received.

quoted

+        exit(EXIT_FAILURE);
+    }
+
+    control_writeln("REPLY0");
+    control_expectln("SEND1");
+
+
+    res = recv(fd, buf, sizeof(buf), 0);

Perhaps a comment here to explain why we expect only 8 bytes.

quoted

+    if (res != 8) {
+        fprintf(stderr, "expected recv(2) failure, got %zi\n", res);

Ditto.

quoted

+        exit(EXIT_FAILURE);
+    }
+
+    res = recv(fd, buf, sizeof(buf), MSG_DONTWAIT);
+    if (res != -1) {
+        fprintf(stderr, "expected recv(2) success, got %zi\n", res);

It's the other way around, isn't it?
Here you expect it to fail instead it is not failing.

quoted

+        exit(EXIT_FAILURE);
+    }

Moving the pointer correctly, I would also check that there is
HELLOWORLD in the buffer.

Thanks for adding tests in this suite!
Stefano

Thanks for review, i didn't pay any attention on this test, because it is
just bug reproducer. But if we are going to add it, of course i'll clean
it's code.

Thanks, Arseniy

quoted

+
+    control_writeln("REPLY1");
+
+    close(fd);
+}
+
static struct test_case test_cases[] = {
    {
        .name = "SOCK_STREAM connection reset",

@@ -1038,6 +1114,11 @@ static struct test_case test_cases[] = {

        .run_client = test_seqpacket_inv_buf_client,
        .run_server = test_seqpacket_inv_buf_server,
    },
+    {
+        .name = "SOCK_STREAM virtio skb merge",
+        .run_client = test_stream_virtio_skb_merge_client,
+        .run_server = test_stream_virtio_skb_merge_server,
+    },
    {},
};

-- 
2.25.1

On 20.03.2023 18:07, Stefano Garzarella wrote:

On Sun, Mar 19, 2023 at 09:52:19PM +0300, Arseniy Krasnov wrote:

quoted

This prints WARN() and returns from stream dequeue callback when socket's
queue is empty, but 'rx_bytes' still non-zero.

Signed-off-by: Arseniy Krasnov <redacted>
---
net/vmw_vsock/virtio_transport_common.c | 7 +++++++
1 file changed, 7 insertions(+)

diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
index 3c75986e16c2..c35b03adad8d 100644
--- a/net/vmw_vsock/virtio_transport_common.c
+++ b/net/vmw_vsock/virtio_transport_common.c

@@ -388,6 +388,13 @@ virtio_transport_stream_do_dequeue(struct vsock_sock *vsk,

    u32 free_space;

    spin_lock_bh(&vvs->rx_lock);
+
+    if (skb_queue_empty(&vvs->rx_queue) && vvs->rx_bytes) {
+        WARN(1, "No skbuffs with non-zero 'rx_bytes'\n");

I would use WARN_ONCE, since we can't recover so we will flood the log.

And you can put the condition in the first argument, I mean something
like this:
        if (WARN_ONCE(skb_queue_empty(&vvs->rx_queue) && vvs->rx_bytes,
                      "rx_queue is empty, but rx_bytes is non-zero\n")) {

I see, ok.

Thanks,
Stefano

quoted

+        spin_unlock_bh(&vvs->rx_lock);
+        return err;
+    }
+
    while (total < len && !skb_queue_empty(&vvs->rx_queue)) {
        skb = skb_peek(&vvs->rx_queue);

-- 
2.25.1

On 20.03.2023 17:57, Stefano Garzarella wrote:

On Sun, Mar 19, 2023 at 09:51:06PM +0300, Arseniy Krasnov wrote:

quoted

This fixes header length calculation of skbuff during data appending to
it. When such skbuff is processed in dequeue callbacks, e.g. 'skb_pull()'
is called on it, 'skb->len' is dynamic value, so it is impossible to use
it in header, because value from header must be permanent for valid
credit calculation ('rx_bytes'/'fwd_cnt').

Fixes: 077706165717 ("virtio/vsock: don't use skbuff state to account credit")

I don't understand how this commit introduced this problem, can you
explain it better?

Sorry, seems i said it wrong a little bit. Before 0777, implementation was buggy, but
exactly this problem was not actual - it didn't triggered somehow. I checked it with
reproducer from this patch. But in 0777 as value from header was used to 'rx_bytes'
calculation, bug become actual. Yes, may be it is not "Fixes:" for 0777, but critical
addition. I'm not sure.

Is it related more to the credit than to the size in the header itself?

It is related to size in header more.

Anyway, the patch LGTM, but we should explain better the issue.

 
Ok, I'll write it more clear in the commit message.

Thanks, Arseniy

Thanks,
Stefano

quoted

Signed-off-by: Arseniy Krasnov <redacted>
---
net/vmw_vsock/virtio_transport_common.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
index 6d15cd4d090a..3c75986e16c2 100644
--- a/net/vmw_vsock/virtio_transport_common.c
+++ b/net/vmw_vsock/virtio_transport_common.c

@@ -1091,7 +1091,7 @@ virtio_transport_recv_enqueue(struct vsock_sock *vsk,

            memcpy(skb_put(last_skb, skb->len), skb->data, skb->len);
            free_pkt = true;
            last_hdr->flags |= hdr->flags;
-            last_hdr->len = cpu_to_le32(last_skb->len);
+            le32_add_cpu(&last_hdr->len, len);
            goto out;
        }
    }
-- 
2.25.1

On Mon, Mar 20, 2023 at 09:10:13PM +0300, Arseniy Krasnov wrote:

On 20.03.2023 17:57, Stefano Garzarella wrote:

quoted

On Sun, Mar 19, 2023 at 09:51:06PM +0300, Arseniy Krasnov wrote:

quoted

This fixes header length calculation of skbuff during data appending to
it. When such skbuff is processed in dequeue callbacks, e.g. 'skb_pull()'
is called on it, 'skb->len' is dynamic value, so it is impossible to use
it in header, because value from header must be permanent for valid
credit calculation ('rx_bytes'/'fwd_cnt').

Fixes: 077706165717 ("virtio/vsock: don't use skbuff state to account credit")

I don't understand how this commit introduced this problem, can you
explain it better?

Sorry, seems i said it wrong a little bit. Before 0777, implementation was buggy, but
exactly this problem was not actual - it didn't triggered somehow. I checked it with
reproducer from this patch. But in 0777 as value from header was used to 'rx_bytes'
calculation, bug become actual. Yes, may be it is not "Fixes:" for 0777, but critical
addition. I'm not sure.

quoted

Is it related more to the credit than to the size in the header itself?

It is related to size in header more.

quoted

Anyway, the patch LGTM, but we should explain better the issue.

Ok, I'll write it more clear in the commit message.

Okay, if 077706165717 triggered the problem, even if it was there from
before, then IMHO it is okay to use that commit in Fixes.

Please, explain it better in the message, so it's clear for everyone ;-)

Thanks,
Stefano