This patchset implements support of SOCK_SEQPACKET for virtio
transport.
As SOCK_SEQPACKET guarantees to save record boundaries, so to
do it, two new packet operations were added: first for start of record
and second to mark end of record(SEQ_BEGIN and SEQ_END later). Also,
both operations carries metadata - to maintain boundaries and payload
integrity. Metadata is introduced by adding special header with two
fields - message id and message length:
struct virtio_vsock_seq_hdr {
__le32 msg_id;
__le32 msg_len;
} __attribute__((packed));
This header is transmitted as payload of SEQ_BEGIN and SEQ_END
packets(buffer of second virtio descriptor in chain) in the same way as
data transmitted in RW packets. Payload was chosen as buffer for this
header to avoid touching first virtio buffer which carries header of
packet, because someone could check that size of this buffer is equal
to size of packet header. To send record, packet with start marker is
sent first(it's header carries length of record and id),then all data
is sent as usual 'RW' packets and finally SEQ_END is sent(it carries
id of message, which is equal to id of SEQ_BEGIN), also after sending
SEQ_END id is incremented. On receiver's side,size of record is known
from packet with start record marker. To check that no packets were
dropped by transport, 'msg_id's of two sequential SEQ_BEGIN and SEQ_END
are checked to be equal and length of data between two markers is
compared to then length in SEQ_BEGIN header.
Now as packets of one socket are not reordered neither on
vsock nor on vhost transport layers, such markers allows to restore
original record on receiver's side. If user's buffer is smaller that
record length, when all out of size data is dropped.
Maximum length of datagram is not limited as in stream socket,
because same credit logic is used. Difference with stream socket is
that user is not woken up until whole record is received or error
occurred. Implementation also supports 'MSG_EOR' and 'MSG_TRUNC' flags.
Tests also implemented.
Thanks to stsp2@yandex.ru for encouragements and initial design
recommendations.
Arseny Krasnov (22):
af_vsock: update functions for connectible socket
af_vsock: separate wait data loop
af_vsock: separate receive data loop
af_vsock: implement SEQPACKET receive loop
af_vsock: separate wait space loop
af_vsock: implement send logic for SEQPACKET
af_vsock: rest of SEQPACKET support
af_vsock: update comments for stream sockets
virtio/vsock: set packet's type in virtio_transport_send_pkt_info()
virtio/vsock: simplify credit update function API
virtio/vsock: dequeue callback for SOCK_SEQPACKET
virtio/vsock: fetch length for SEQPACKET record
virtio/vsock: add SEQPACKET receive logic
virtio/vsock: rest of SOCK_SEQPACKET support
virtio/vsock: SEQPACKET feature bit
vhost/vsock: SEQPACKET feature bit support
virtio/vsock: SEQPACKET feature bit support
virtio/vsock: setup SEQPACKET ops for transport
vhost/vsock: setup SEQPACKET ops for transport
vsock/loopback: setup SEQPACKET ops for transport
vsock_test: add SOCK_SEQPACKET tests
virtio/vsock: update trace event for SEQPACKET
drivers/vhost/vsock.c | 22 +-
include/linux/virtio_vsock.h | 22 +
include/net/af_vsock.h | 10 +
.../events/vsock_virtio_transport_common.h | 48 +-
include/uapi/linux/virtio_vsock.h | 19 +
net/vmw_vsock/af_vsock.c | 589 +++++++++++------
net/vmw_vsock/virtio_transport.c | 18 +
net/vmw_vsock/virtio_transport_common.c | 364 ++++++++--
net/vmw_vsock/vsock_loopback.c | 13 +
tools/testing/vsock/util.c | 32 +-
tools/testing/vsock/util.h | 3 +
tools/testing/vsock/vsock_test.c | 126 ++++
12 files changed, 1013 insertions(+), 253 deletions(-)
v5 -> v6:
General changelog:
- virtio transport specific callbacks which send SEQ_BEGIN or
SEQ_END now hidden inside virtio transport. Only enqueue,
dequeue and record length callbacks are provided by transport.
- virtio feature bit for SEQPACKET socket support introduced:
VIRTIO_VSOCK_F_SEQPACKET.
- 'msg_cnt' field in 'struct virtio_vsock_seq_hdr' renamed to
'msg_id' and used as id.
Per patch changelog:
- 'af_vsock: separate wait data loop':
1) Commit message updated.
2) 'prepare_to_wait()' moved inside while loop(thanks to
Jorgen Hansen).
Marked 'Reviewed-by' with 1), but as 2) I removed R-b.
- 'af_vsock: separate receive data loop': commit message
updated.
Marked 'Reviewed-by' with that fix.
- 'af_vsock: implement SEQPACKET receive loop': style fixes.
- 'af_vsock: rest of SEQPACKET support':
1) 'module_put()' added when transport callback check failed.
2) Now only 'seqpacket_allow()' callback called to check
support of SEQPACKET by transport.
- 'af_vsock: update comments for stream sockets': commit message
updated.
Marked 'Reviewed-by' with that fix.
- 'virtio/vsock: set packet's type in send':
1) Commit message updated.
2) Parameter 'type' from 'virtio_transport_send_credit_update()'
also removed in this patch instead of in next.
- 'virtio/vsock: dequeue callback for SOCK_SEQPACKET': SEQPACKET
related state wrapped to special struct.
- 'virtio/vsock: update trace event for SEQPACKET': format strings
now not broken by new lines.
v4 -> v5:
- patches reorganized:
1) Setting of packet's type in 'virtio_transport_send_pkt_info()'
is moved to separate patch.
2) Simplifying of 'virtio_transport_send_credit_update()' is
moved to separate patch and before main virtio/vsock patches.
- style problem fixed
- in 'af_vsock: separate receive data loop' extra 'release_sock()'
removed
- added trace event fields for SEQPACKET
- in 'af_vsock: separate wait data loop':
1) 'vsock_wait_data()' removed 'goto out;'
2) Comment for invalid data amount is changed.
- in 'af_vsock: rest of SEQPACKET support', 'new_transport' pointer
check is moved after 'try_module_get()'
- in 'af_vsock: update comments for stream sockets', 'connect-oriented'
replaced with 'connection-oriented'
- in 'loopback/vsock: setup SEQPACKET ops for transport',
'loopback/vsock' replaced with 'vsock/loopback'
v3 -> v4:
- SEQPACKET specific metadata moved from packet header to payload
and called 'virtio_vsock_seq_hdr'
- record integrity check:
1) SEQ_END operation was added, which marks end of record.
2) Both SEQ_BEGIN and SEQ_END carries counter which is incremented
on every marker send.
- af_vsock.c: socket operations for STREAM and SEQPACKET call same
functions instead of having own "gates" differs only by names:
'vsock_seqpacket/stream_getsockopt()' now replaced with
'vsock_connectible_getsockopt()'.
- af_vsock.c: 'seqpacket_dequeue' callback returns error and flag that
record ready. There is no need to return number of copied bytes,
because case when record received successfully is checked at virtio
transport layer, when SEQ_END is processed. Also user doesn't need
number of copied bytes, because 'recv()' from SEQPACKET could return
error, length of users's buffer or length of whole record(both are
known in af_vsock.c).
- af_vsock.c: both wait loops in af_vsock.c(for data and space) moved
to separate functions because now both called from several places.
- af_vsock.c: 'vsock_assign_transport()' checks that 'new_transport'
pointer is not NULL and returns 'ESOCKTNOSUPPORT' instead of 'ENODEV'
if failed to use transport.
- tools/testing/vsock/vsock_test.c: rename tests
v2 -> v3:
- patches reorganized: split for prepare and implementation patches
- local variables are declared in "Reverse Christmas tree" manner
- virtio_transport_common.c: valid leXX_to_cpu() for vsock header
fields access
- af_vsock.c: 'vsock_connectible_*sockopt()' added as shared code
between stream and seqpacket sockets.
- af_vsock.c: loops in '__vsock_*_recvmsg()' refactored.
- af_vsock.c: 'vsock_wait_data()' refactored.
v1 -> v2:
- patches reordered: af_vsock.c related changes now before virtio vsock
- patches reorganized: more small patches, where +/- are not mixed
- tests for SOCK_SEQPACKET added
- all commit messages updated
- af_vsock.c: 'vsock_pre_recv_check()' inlined to
'vsock_connectible_recvmsg()'
- af_vsock.c: 'vsock_assign_transport()' returns ENODEV if transport
was not found
- virtio_transport_common.c: transport callback for seqpacket dequeue
- virtio_transport_common.c: simplified
'virtio_transport_recv_connected()'
- virtio_transport_common.c: send reset on socket and packet type
mismatch.
Signed-off-by: Arseny Krasnov <redacted>
--
2.25.1
This prepares af_vsock.c for SEQPACKET support: some functions such
as setsockopt(), getsockopt(), connect(), recvmsg(), sendmsg() are
shared between both types of sockets, so rename them in general
manner.
Signed-off-by: Arseny Krasnov <redacted>
---
net/vmw_vsock/af_vsock.c | 64 +++++++++++++++++++++-------------------
1 file changed, 34 insertions(+), 30 deletions(-)
This moves wait loop for data to dedicated function, because later it
will be used by SEQPACKET data receive loop. While moving the code
around, let's update an old comment.
Signed-off-by: Arseny Krasnov <redacted>
---
net/vmw_vsock/af_vsock.c | 156 +++++++++++++++++++++------------------
1 file changed, 84 insertions(+), 72 deletions(-)
@@ -1832,6 +1832,69 @@ static int vsock_connectible_sendmsg(struct socket *sock, struct msghdr *msg,returnerr;}+staticintvsock_wait_data(structsock*sk,structwait_queue_entry*wait,+longtimeout,+structvsock_transport_recv_notify_data*recv_data,+size_ttarget)+{+conststructvsock_transport*transport;+structvsock_sock*vsk;+s64data;+interr;++vsk=vsock_sk(sk);+err=0;+transport=vsk->transport;++while((data=vsock_stream_has_data(vsk))==0){+prepare_to_wait(sk_sleep(sk),wait,TASK_INTERRUPTIBLE);++if(sk->sk_err!=0||+(sk->sk_shutdown&RCV_SHUTDOWN)||+(vsk->peer_shutdown&SEND_SHUTDOWN)){+break;+}++/* Don't wait for non-blocking sockets. */+if(timeout==0){+err=-EAGAIN;+break;+}++if(recv_data){+err=transport->notify_recv_pre_block(vsk,target,recv_data);+if(err<0)+break;+}++release_sock(sk);+timeout=schedule_timeout(timeout);+lock_sock(sk);++if(signal_pending(current)){+err=sock_intr_errno(timeout);+break;+}elseif(timeout==0){+err=-EAGAIN;+break;+}+}++finish_wait(sk_sleep(sk),wait);++if(err)+returnerr;++/* Internal transport error when checking for available+*data.XXXThisshouldbechangedtoaconnection+*resetinalaterchange.+*/+if(data<0)+return-ENOMEM;++returndata;+}+staticintvsock_connectible_recvmsg(structsocket*sock,structmsghdr*msg,size_tlen,intflags)
This adds receive loop for SEQPACKET. It looks like receive loop for
STREAM, but there is a little bit difference:
1) It doesn't call notify callbacks.
2) It doesn't care about 'SO_SNDLOWAT' and 'SO_RCVLOWAT' values, because
there is no sense for these values in SEQPACKET case.
3) It waits until whole record is received or error is found during
receiving.
4) It processes and sets 'MSG_TRUNC' flag.
So to avoid extra conditions for two types of socket inside one loop, two
independent functions were created.
Signed-off-by: Arseny Krasnov <redacted>
---
include/net/af_vsock.h | 5 +++
net/vmw_vsock/af_vsock.c | 95 +++++++++++++++++++++++++++++++++++++++-
2 files changed, 99 insertions(+), 1 deletion(-)
@@ -1973,6 +1973,96 @@ static int __vsock_stream_recvmsg(struct sock *sk, struct msghdr *msg,returnerr;}+staticint__vsock_seqpacket_recvmsg(structsock*sk,structmsghdr*msg,+size_tlen,intflags)+{+conststructvsock_transport*transport;+conststructiovec*orig_iov;+unsignedlongorig_nr_segs;+boolmsg_ready;+structvsock_sock*vsk;+size_trecord_len;+longtimeout;+interr=0;+DEFINE_WAIT(wait);++vsk=vsock_sk(sk);+transport=vsk->transport;++timeout=sock_rcvtimeo(sk,flags&MSG_DONTWAIT);+orig_nr_segs=msg->msg_iter.nr_segs;+orig_iov=msg->msg_iter.iov;+msg_ready=false;+record_len=0;++while(1){+err=vsock_wait_data(sk,&wait,timeout,NULL,0);++if(err<=0){+/* In case of any loop break(timeout, signal+*interruptorshutdown),wereportuserthat+*nothingwascopied.+*/+err=0;+break;+}++if(record_len==0){+record_len=+transport->seqpacket_seq_get_len(vsk);++if(record_len==0)+continue;+}++err=transport->seqpacket_dequeue(vsk,msg,flags,&msg_ready);+if(err<0){+if(err==-EAGAIN){+iov_iter_init(&msg->msg_iter,READ,+orig_iov,orig_nr_segs,+len);+/* Clear 'MSG_EOR' here, because dequeue+*callbackabovesetitagainifitwas+*setbysender.This'MSG_EOR'isfrom+*droppedrecord.+*/+msg->msg_flags&=~MSG_EOR;+record_len=0;+continue;+}++err=-ENOMEM;+break;+}++if(msg_ready)+break;+}++if(sk->sk_err)+err=-sk->sk_err;+elseif(sk->sk_shutdown&RCV_SHUTDOWN)+err=0;++if(msg_ready){+/* User sets MSG_TRUNC, so return real length of+*packet.+*/+if(flags&MSG_TRUNC)+err=record_len;+else+err=len-msg->msg_iter.count;++/* Always set MSG_TRUNC if real length of packet is+*biggerthanuser'sbuffer.+*/+if(record_len>len)+msg->msg_flags|=MSG_TRUNC;+}++returnerr;+}+staticintvsock_connectible_recvmsg(structsocket*sock,structmsghdr*msg,size_tlen,intflags)
Move STREAM specific data receive logic to '__vsock_stream_recvmsg()'
dedicated function, while checks, that will be same for both STREAM
and SEQPACKET sockets, stays in 'vsock_connectible_recvmsg()' shared
functions.
Signed-off-by: Arseny Krasnov <redacted>
Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
---
net/vmw_vsock/af_vsock.c | 116 ++++++++++++++++++++++-----------------
1 file changed, 67 insertions(+), 49 deletions(-)
@@ -1895,65 +1895,22 @@ static int vsock_wait_data(struct sock *sk, struct wait_queue_entry *wait,returndata;}-staticint-vsock_connectible_recvmsg(structsocket*sock,structmsghdr*msg,size_tlen,-intflags)+staticint__vsock_stream_recvmsg(structsock*sk,structmsghdr*msg,+size_tlen,intflags){-structsock*sk;-structvsock_sock*vsk;+structvsock_transport_recv_notify_datarecv_data;conststructvsock_transport*transport;-interr;-size_ttarget;+structvsock_sock*vsk;ssize_tcopied;+size_ttarget;longtimeout;-structvsock_transport_recv_notify_datarecv_data;+interr;DEFINE_WAIT(wait);-sk=sock->sk;vsk=vsock_sk(sk);-err=0;--lock_sock(sk);-transport=vsk->transport;-if(!transport||sk->sk_state!=TCP_ESTABLISHED){-/* Recvmsg is supposed to return 0 if a peer performs an-*orderlyshutdown.Differentiatebetweenthatcaseandwhena-*peerhasnotconnectedoralocalshutdownoccuredwiththe-*SOCK_DONEflag.-*/-if(sock_flag(sk,SOCK_DONE))-err=0;-else-err=-ENOTCONN;--gotoout;-}--if(flags&MSG_OOB){-err=-EOPNOTSUPP;-gotoout;-}--/* We don't check peer_shutdown flag here since peer may actually shut-*down,buttherecanbedatainthequeuethatalocalsocketcan-*receive.-*/-if(sk->sk_shutdown&RCV_SHUTDOWN){-err=0;-gotoout;-}--/* It is valid on Linux to pass in a zero-length receive buffer. This-*isnotanerror.Wemayaswellbailoutnow.-*/-if(!len){-err=0;-gotoout;-}-/* We must not copy less than target bytes into the user's buffer*beforereturningsuccessfully,sowewaitfortheconsumequeueto*havethatmuchdatatoconsumebeforedequeueing.Notethatthis
@@ -2012,6 +1969,67 @@ vsock_connectible_recvmsg(struct socket *sock, struct msghdr *msg, size_t len,if(copied>0)err=copied;+out:+returnerr;+}++staticint+vsock_connectible_recvmsg(structsocket*sock,structmsghdr*msg,size_tlen,+intflags)+{+structsock*sk;+structvsock_sock*vsk;+conststructvsock_transport*transport;+interr;++DEFINE_WAIT(wait);++sk=sock->sk;+vsk=vsock_sk(sk);+err=0;++lock_sock(sk);++transport=vsk->transport;++if(!transport||sk->sk_state!=TCP_ESTABLISHED){+/* Recvmsg is supposed to return 0 if a peer performs an+*orderlyshutdown.Differentiatebetweenthatcaseandwhena+*peerhasnotconnectedoralocalshutdownoccurredwiththe+*SOCK_DONEflag.+*/+if(sock_flag(sk,SOCK_DONE))+err=0;+else+err=-ENOTCONN;++gotoout;+}++if(flags&MSG_OOB){+err=-EOPNOTSUPP;+gotoout;+}++/* We don't check peer_shutdown flag here since peer may actually shut+*down,buttherecanbedatainthequeuethatalocalsocketcan+*receive.+*/+if(sk->sk_shutdown&RCV_SHUTDOWN){+err=0;+gotoout;+}++/* It is valid on Linux to pass in a zero-length receive buffer. This+*isnotanerror.Wemayaswellbailoutnow.+*/+if(!len){+err=0;+gotoout;+}++err=__vsock_stream_recvmsg(sk,msg,len,flags);+out:release_sock(sk);returnerr;
This moves loop that waits for space on send to separate function,
because it will be used for SEQ_BEGIN/SEQ_END sending before and
after data transmission. Waiting for SEQ_BEGIN/SEQ_END is needed
because such packets carries SEQPACKET header that couldn't be
fragmented by credit mechanism, so to avoid it, sender waits until
enough space will be ready.
Signed-off-by: Arseny Krasnov <redacted>
Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
---
include/net/af_vsock.h | 2 +
net/vmw_vsock/af_vsock.c | 99 +++++++++++++++++++++++++---------------
2 files changed, 63 insertions(+), 38 deletions(-)
@@ -1740,9 +1797,6 @@ static int vsock_connectible_sendmsg(struct socket *sock, struct msghdr *msg,gotoout;}-/* Wait for room in the produce queue to enqueue our user's data. */-timeout=sock_sndtimeo(sk,msg->msg_flags&MSG_DONTWAIT);-err=transport->notify_send_init(vsk,&send_data);if(err<0)gotoout;
@@ -1750,39 +1804,8 @@ static int vsock_connectible_sendmsg(struct socket *sock, struct msghdr *msg,while(total_written<len){ssize_twritten;-add_wait_queue(sk_sleep(sk),&wait);-while(vsock_stream_has_space(vsk)==0&&-sk->sk_err==0&&-!(sk->sk_shutdown&SEND_SHUTDOWN)&&-!(vsk->peer_shutdown&RCV_SHUTDOWN)){--/* Don't wait for non-blocking sockets. */-if(timeout==0){-err=-EAGAIN;-remove_wait_queue(sk_sleep(sk),&wait);-gotoout_err;-}--err=transport->notify_send_pre_block(vsk,&send_data);-if(err<0){-remove_wait_queue(sk_sleep(sk),&wait);-gotoout_err;-}--release_sock(sk);-timeout=wait_woken(&wait,TASK_INTERRUPTIBLE,timeout);-lock_sock(sk);-if(signal_pending(current)){-err=sock_intr_errno(timeout);-remove_wait_queue(sk_sleep(sk),&wait);-gotoout_err;-}elseif(timeout==0){-err=-EAGAIN;-remove_wait_queue(sk_sleep(sk),&wait);-gotoout_err;-}-}-remove_wait_queue(sk_sleep(sk),&wait);+if(vsock_wait_space(sk,1,msg->msg_flags,&send_data))+gotoout_err;/* These checks occur both as part of and after the loop*conditionalsinceweneedtocheckbeforeandafter
This adds some logic to current stream enqueue function for SEQPACKET
support:
1) Use transport's seqpacket enqueue callback.
2) Return value from enqueue function is whole record length or error
for SOCK_SEQPACKET.
Signed-off-by: Arseny Krasnov <redacted>
---
include/net/af_vsock.h | 2 ++
net/vmw_vsock/af_vsock.c | 22 ++++++++++++++++------
2 files changed, 18 insertions(+), 6 deletions(-)
@@ -1844,12 +1849,17 @@ static int vsock_connectible_sendmsg(struct socket *sock, struct msghdr *msg,vsk,written,&send_data);if(err<0)gotoout_err;-}out_err:-if(total_written>0)-err=total_written;+if(total_written>0){+/* Return number of written bytes only if:+*1)SOCK_STREAMsocket.+*2)SOCK_SEQPACKETsocketwhenwholebufferissent.+*/+if(sk->sk_type==SOCK_STREAM||total_written==len)+err=total_written;+}out:release_sock(sk);returnerr;
This moves passing type of packet from 'info' structure to 'virtio_
transport_send_pkt_info()' function. There is no need to set type of
packet which differs from type of socket. Since at current time only
stream type is supported, set it directly in 'virtio_transport_send_
pkt_info()', so callers don't need to set it.
Signed-off-by: Arseny Krasnov <redacted>
---
net/vmw_vsock/virtio_transport_common.c | 19 +++++--------------
1 file changed, 5 insertions(+), 14 deletions(-)
@@ -624,7 +620,6 @@ int virtio_transport_connect(struct vsock_sock *vsk){structvirtio_vsock_pkt_infoinfo={.op=VIRTIO_VSOCK_OP_REQUEST,-.type=VIRTIO_VSOCK_TYPE_STREAM,.vsk=vsk,};
@@ -636,7 +631,6 @@ int virtio_transport_shutdown(struct vsock_sock *vsk, int mode){structvirtio_vsock_pkt_infoinfo={.op=VIRTIO_VSOCK_OP_SHUTDOWN,-.type=VIRTIO_VSOCK_TYPE_STREAM,.flags=(mode&RCV_SHUTDOWN?VIRTIO_VSOCK_SHUTDOWN_RCV:0)|(mode&SEND_SHUTDOWN?
This replaces 'stream' to 'connection oriented' in comments as
SEQPACKET is also connection oriented.
Signed-off-by: Arseny Krasnov <redacted>
---
net/vmw_vsock/af_vsock.c | 31 +++++++++++++++++--------------
1 file changed, 17 insertions(+), 14 deletions(-)
@@ -415,8 +415,8 @@ static void vsock_deassign_transport(struct vsock_sock *vsk)/* Assign a transport to a socket and call the .init transport callback.*-*Note:forstreamsocketthismustbecalledwhenvsk->remote_addrisset-*(e.g.duringtheconnect()orwhenaconnectionrequestonalistener+*Note:forconnectionorientedsocketthismustbecalledwhenvsk->remote_addr+*isset(e.g.duringtheconnect()orwhenaconnectionrequestonalistener*socketisreceived).*Thevsk->remote_addrisusedtodecidewhichtransporttouse:*-remoteCID==VMADDR_CID_LOCALorg2h->local_cidorVMADDR_CID_HOSTif
@@ -470,10 +470,10 @@ int vsock_assign_transport(struct vsock_sock *vsk, struct vsock_sock *psk)return0;/* transport->release() must be called with sock lock acquired.-*Thispathcanonlybetakenduringvsock_stream_connect(),-*wherewehavealreadyheldthesocklock.-*Intheothercases,thisfunctioniscalledonanewsocket-*whichisnotassignedtoanytransport.+*Thispathcanonlybetakenduringvsock_connect(),wherewe+*havealreadyheldthesocklock.Intheothercases,this+*functioniscalledonanewsocketwhichisnotassignedto+*anytransport.*/vsk->transport->release(vsk);vsock_deassign_transport(vsk);
@@ -658,9 +658,10 @@ static int __vsock_bind_connectible(struct vsock_sock *vsk,vsock_addr_init(&vsk->local_addr,new_addr.svm_cid,new_addr.svm_port);-/* Remove stream sockets from the unbound list and add them to the hash-*tableforeasylookupbyitsaddress.Theunboundlistissimplyan-*extraentryattheendofthehashtable,atrickusedbyAF_UNIX.+/* Remove connection oriented sockets from the unbound list and add them+*tothehashtableforeasylookupbyitsaddress.Theunboundlist+*issimplyanextraentryattheendofthehashtable,atrickused+*byAF_UNIX.*/__vsock_remove_bound(vsk);__vsock_insert_bound(vsock_bound_sockets(&vsk->local_addr),vsk);
@@ -951,10 +952,10 @@ static int vsock_shutdown(struct socket *sock, int mode)if((mode&~SHUTDOWN_MASK)||!mode)return-EINVAL;-/* If this is a STREAM socket and it is not connected then bail out-*immediately.IfitisaDGRAMsocketthenwemustfirstkickthe-*socketsothatitwakesupfromanysleepingcalls,forexample-*recv(),andthenafterwardsreturntheerror.+/* If this is a connection oriented socket and it is not connected then+*bailoutimmediately.IfitisaDGRAMsocketthenwemustfirst+*kickthesocketsothatitwakesupfromanysleepingcalls,for+*examplerecv(),andthenafterwardsreturntheerror.*/sk=sock->sk;
@@ -1783,7 +1784,9 @@ static int vsock_connectible_sendmsg(struct socket *sock, struct msghdr *msg,transport=vsk->transport;-/* Callers should not provide a destination with stream sockets. */+/* Callers should not provide a destination with connection oriented+*sockets.+*/if(msg->msg_namelen){err=sk->sk_state==TCP_ESTABLISHED?-EISCONN:-EOPNOTSUPP;gotoout;
This adds transport callback and it's logic for SEQPACKET dequeue.
Callback fetches RW packets from rx queue of socket until whole record
is copied(if user's buffer is full, user is not woken up). This is done
to not stall sender, because if we wake up user and it leaves syscall,
nobody will send credit update for rest of record, and sender will wait
for next enter of read syscall at receiver's side. So if user buffer is
full, we just send credit update and drop data. If during copy SEQ_BEGIN
was found(and not all data was copied), copying is restarted by reset
user's iov iterator(previous unfinished data is dropped).
Signed-off-by: Arseny Krasnov <redacted>
---
include/linux/virtio_vsock.h | 13 +++
include/uapi/linux/virtio_vsock.h | 16 ++++
net/vmw_vsock/virtio_transport_common.c | 116 ++++++++++++++++++++++++
3 files changed, 145 insertions(+)
@@ -83,6 +89,11 @@ enum virtio_vsock_op {VIRTIO_VSOCK_OP_CREDIT_UPDATE=6,/* Request the peer to send the credit info to us */VIRTIO_VSOCK_OP_CREDIT_REQUEST=7,++/* Record begin for SOCK_SEQPACKET */+VIRTIO_VSOCK_OP_SEQ_BEGIN=8,+/* Record end for SOCK_SEQPACKET */+VIRTIO_VSOCK_OP_SEQ_END=9,};/* VIRTIO_VSOCK_OP_SHUTDOWN flags values */
@@ -393,6 +393,110 @@ virtio_transport_stream_do_dequeue(struct vsock_sock *vsk,returnerr;}+staticinlinevoidvirtio_transport_remove_pkt(structvirtio_vsock_pkt*pkt)+{+list_del(&pkt->list);+virtio_transport_free_pkt(pkt);+}++staticintvirtio_transport_seqpacket_do_dequeue(structvsock_sock*vsk,+structmsghdr*msg,+bool*msg_ready)+{+structvirtio_vsock_sock*vvs=vsk->trans;+structvirtio_vsock_pkt*pkt;+interr=0;+size_tuser_buf_len=msg->msg_iter.count;++*msg_ready=false;+spin_lock_bh(&vvs->rx_lock);++while(!*msg_ready&&!list_empty(&vvs->rx_queue)&&!err){+pkt=list_first_entry(&vvs->rx_queue,structvirtio_vsock_pkt,list);++switch(le16_to_cpu(pkt->hdr.op)){+caseVIRTIO_VSOCK_OP_SEQ_BEGIN:{+/* Unexpected 'SEQ_BEGIN' during record copy:+*Leavereceiveloop,'EAGAIN'willrestartitfrom+*outerreceiveloop,packetisstillinqueueand+*countersarecleared.Soinnextloopenter,+*'SEQ_BEGIN'willbedequeuedfirst.User'siov+*iteratorwillberesetinouterloop.Also+*sendcreditupdate,becausesomebytescouldbe+*copied.Userwillneverseeunfinishedrecord.+*/+err=-EAGAIN;+break;+}+caseVIRTIO_VSOCK_OP_SEQ_END:{+structvirtio_vsock_seq_hdr*seq_hdr;++seq_hdr=(structvirtio_vsock_seq_hdr*)pkt->buf;+/* First check that whole record is received. */++if(vvs->seqpacket_state.user_read_copied!=+vvs->seqpacket_state.user_read_seq_len||+le32_to_cpu(seq_hdr->msg_id)!=+vvs->seqpacket_state.curr_rx_msg_id){+/* Tail of current record and head of next missed,+*sothisEORisfromnextrecord.Restartreceive.+*Currentrecordwillbedropped,nextheadlesswill+*bedroppedonnextattempttogetrecordlength.+*/+err=-EAGAIN;+}else{+/* Success. */+*msg_ready=true;+}++break;+}+caseVIRTIO_VSOCK_OP_RW:{+size_tbytes_to_copy;+size_tpkt_len;++pkt_len=(size_t)le32_to_cpu(pkt->hdr.len);+bytes_to_copy=min(user_buf_len,pkt_len);++/* sk_lock is held by caller so no one else can dequeue.+*Unlockrx_locksincememcpy_to_msg()maysleep.+*/+spin_unlock_bh(&vvs->rx_lock);++if(memcpy_to_msg(msg,pkt->buf,bytes_to_copy)){+spin_lock_bh(&vvs->rx_lock);+err=-EINVAL;+break;+}++spin_lock_bh(&vvs->rx_lock);+user_buf_len-=bytes_to_copy;+vvs->seqpacket_state.user_read_copied+=pkt_len;++if(le32_to_cpu(pkt->hdr.flags)&VIRTIO_VSOCK_RW_EOR)+msg->msg_flags|=MSG_EOR;+break;+}+default:+;+}++/* For unexpected 'SEQ_BEGIN', keep such packet in queue,+*butdropanyothertypeofpacket.+*/+if(le16_to_cpu(pkt->hdr.op)!=VIRTIO_VSOCK_OP_SEQ_BEGIN){+virtio_transport_dec_rx_pkt(vvs,pkt);+virtio_transport_remove_pkt(pkt);+}+}++spin_unlock_bh(&vvs->rx_lock);++virtio_transport_send_credit_update(vsk);++returnerr;+}+ssize_tvirtio_transport_stream_dequeue(structvsock_sock*vsk,structmsghdr*msg,
This adds transport callback which tries to fetch record begin marker
from socket's rx queue. It is called from af_vsock.c before reading data
packets of record.
Signed-off-by: Arseny Krasnov <redacted>
---
include/linux/virtio_vsock.h | 1 +
net/vmw_vsock/virtio_transport_common.c | 53 +++++++++++++++++++++++++
2 files changed, 54 insertions(+)
This modifies current receive logic for SEQPACKET support:
1) Inserts 'SEQ_BEGIN' packet to socket's rx queue.
2) Inserts 'RW' packet to socket's rx queue, but without merging with
buffer of last packet in queue.
3) Performs check for packet and socket types on receive(if mismatch,
then reset connection).
Signed-off-by: Arseny Krasnov <redacted>
---
net/vmw_vsock/virtio_transport_common.c | 63 +++++++++++++++++--------
1 file changed, 44 insertions(+), 19 deletions(-)
@@ -165,6 +165,14 @@ void virtio_transport_deliver_tap_pkt(struct virtio_vsock_pkt *pkt)}EXPORT_SYMBOL_GPL(virtio_transport_deliver_tap_pkt);+staticu16virtio_transport_get_type(structsock*sk)+{+if(sk->sk_type==SOCK_STREAM)+returnVIRTIO_VSOCK_TYPE_STREAM;+else+returnVIRTIO_VSOCK_TYPE_SEQPACKET;+}+/* This function can only be used on connecting/connected sockets,*sinceasocketassignedtoatransportisrequired.*
@@ -1062,25 +1070,27 @@ virtio_transport_recv_enqueue(struct vsock_sock *vsk,gotoout;}-/* Try to copy small packets into the buffer of last packet queued,-*toavoidwastingmemoryqueueingtheentirebufferwithasmall-*payload.-*/-if(pkt->len<=GOOD_COPY_LEN&&!list_empty(&vvs->rx_queue)){-structvirtio_vsock_pkt*last_pkt;+if(le16_to_cpu(pkt->hdr.type)==VIRTIO_VSOCK_TYPE_STREAM){+/* Try to copy small packets into the buffer of last packet queued,+*toavoidwastingmemoryqueueingtheentirebufferwithasmall+*payload.+*/+if(pkt->len<=GOOD_COPY_LEN&&!list_empty(&vvs->rx_queue)){+structvirtio_vsock_pkt*last_pkt;-last_pkt=list_last_entry(&vvs->rx_queue,-structvirtio_vsock_pkt,list);+last_pkt=list_last_entry(&vvs->rx_queue,+structvirtio_vsock_pkt,list);-/* If there is space in the last packet queued, we copy the-*newpacketinitsbuffer.-*/-if(pkt->len<=last_pkt->buf_len-last_pkt->len){-memcpy(last_pkt->buf+last_pkt->len,pkt->buf,-pkt->len);-last_pkt->len+=pkt->len;-free_pkt=true;-gotoout;+/* If there is space in the last packet queued, we copy the+*newpacketinitsbuffer.+*/+if(pkt->len<=last_pkt->buf_len-last_pkt->len){+memcpy(last_pkt->buf+last_pkt->len,pkt->buf,+pkt->len);+last_pkt->len+=pkt->len;+free_pkt=true;+gotoout;+}}}
@@ -1245,6 +1259,12 @@ virtio_transport_recv_listen(struct sock *sk, struct virtio_vsock_pkt *pkt,return0;}+staticboolvirtio_transport_valid_type(u16type)+{+return(type==VIRTIO_VSOCK_TYPE_STREAM)||+(type==VIRTIO_VSOCK_TYPE_SEQPACKET);+}+/* We are under the virtio-vsock's vsock->rx_lock or vhost-vsock's vq->mutex*lock.*/
This adds rest of logic for SEQPACKET:
1) SEQPACKET specific functions which send SEQ_BEGIN/SEQ_END.
Note that both functions may sleep to wait enough space for
SEQPACKET header.
2) SEQ_BEGIN/SEQ_END in TAP packet capture.
3) Send SHUTDOWN on socket close for SEQPACKET type.
4) Set SEQPACKET packet type during send.
5) Set MSG_EOR in flags for SEQPACKET during send.
6) 'seqpacket_allow' flag to virtio transport.
Signed-off-by: Arseny Krasnov <redacted>
---
include/linux/virtio_vsock.h | 8 +++
net/vmw_vsock/virtio_transport_common.c | 87 ++++++++++++++++++++++++-
2 files changed, 93 insertions(+), 2 deletions(-)
@@ -187,7 +189,12 @@ static int virtio_transport_send_pkt_info(struct vsock_sock *vsk,structvirtio_vsock_pkt*pkt;u32pkt_len=info->pkt_len;-info->type=VIRTIO_VSOCK_TYPE_STREAM;+info->type=virtio_transport_get_type(sk_vsock(vsk));++if(info->type==VIRTIO_VSOCK_TYPE_SEQPACKET&&+info->msg&&+info->msg->msg_flags&MSG_EOR)+info->flags|=VIRTIO_VSOCK_RW_EOR;t_ops=virtio_transport_get_ops(vsk);if(unlikely(!t_ops))
@@ -401,6 +408,43 @@ virtio_transport_stream_do_dequeue(struct vsock_sock *vsk,returnerr;}+staticintvirtio_transport_seqpacket_send_ctrl(structvsock_sock*vsk,+inttype,+size_tlen,+intflags)+{+structvirtio_vsock_sock*vvs=vsk->trans;+structvirtio_vsock_pkt_infoinfo={+.op=type,+.vsk=vsk,+.pkt_len=sizeof(structvirtio_vsock_seq_hdr)+};++structvirtio_vsock_seq_hdrseq_hdr={+.msg_id=cpu_to_le32(vvs->seqpacket_state.next_tx_msg_id),+.msg_len=cpu_to_le32(len)+};++structkvecseq_hdr_kiov={+.iov_base=(void*)&seq_hdr,+.iov_len=sizeof(structvirtio_vsock_seq_hdr)+};++structmsghdrmsg={0};++//XXX: do we need 'vsock_transport_send_notify_data' pointer?+if(vsock_wait_space(sk_vsock(vsk),+sizeof(structvirtio_vsock_seq_hdr),+flags,NULL))+return-1;++iov_iter_kvec(&msg.msg_iter,WRITE,&seq_hdr_kiov,1,sizeof(seq_hdr));++info.msg=&msg;++returnvirtio_transport_send_pkt_info(vsk,&info);+}+staticinlinevoidvirtio_transport_remove_pkt(structvirtio_vsock_pkt*pkt){list_del(&pkt->list);
This adds new virtio vsock specific feature bit which means
SOCK_SEQPACKET support. Guest negotiates this bit with vhost,
thus checking that vhost side supports SEQPACKET.
Signed-off-by: Arseny Krasnov <redacted>
---
include/uapi/linux/virtio_vsock.h | 3 +++
1 file changed, 3 insertions(+)
This adds handling of SEQPACKET bit: if guest sets features with
this bit cleared, then SOCK_SEQPACKET support will be disabled.
Signed-off-by: Arseny Krasnov <redacted>
---
drivers/vhost/vsock.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
This also removes ignore of non-stream type of packets and adds
'seqpacket_allow()' callback.
Signed-off-by: Arseny Krasnov <redacted>
---
drivers/vhost/vsock.c | 16 ++++++++++++++--
1 file changed, 14 insertions(+), 2 deletions(-)
This adds two tests of SOCK_SEQPACKET socket: both transfer data and
then test MSG_EOR and MSG_TRUNC flags. Cases for connect(), bind(),
etc. are not tested, because it is same as for stream socket.
Signed-off-by: Arseny Krasnov <redacted>
---
tools/testing/vsock/util.c | 32 ++++++--
tools/testing/vsock/util.h | 3 +
tools/testing/vsock/vsock_test.c | 126 +++++++++++++++++++++++++++++++
3 files changed, 156 insertions(+), 5 deletions(-)
@@ -84,7 +84,7 @@ void vsock_wait_remote_close(int fd)}/* Connect to <cid, port> and return the file descriptor. */-intvsock_stream_connect(unsignedintcid,unsignedintport)+staticintvsock_connect(unsignedintcid,unsignedintport,inttype){union{structsockaddrsa;
@@ -101,7 +101,7 @@ int vsock_stream_connect(unsigned int cid, unsigned int port)control_expectln("LISTENING");-fd=socket(AF_VSOCK,SOCK_STREAM,0);+fd=socket(AF_VSOCK,type,0);timeout_begin(TIMEOUT);do{
@@ -120,11 +120,21 @@ int vsock_stream_connect(unsigned int cid, unsigned int port)returnfd;}+intvsock_stream_connect(unsignedintcid,unsignedintport)+{+returnvsock_connect(cid,port,SOCK_STREAM);+}++intvsock_seqpacket_connect(unsignedintcid,unsignedintport)+{+returnvsock_connect(cid,port,SOCK_SEQPACKET);+}+/* Listen on <cid, port> and return the first incoming connection. The remote*addressisstoredtoclientaddrp.clientaddrpmaybeNULL.*/-intvsock_stream_accept(unsignedintcid,unsignedintport,-structsockaddr_vm*clientaddrp)+staticintvsock_accept(unsignedintcid,unsignedintport,+structsockaddr_vm*clientaddrp,inttype){union{structsockaddrsa;
@@ -145,7 +155,7 @@ int vsock_stream_accept(unsigned int cid, unsigned int port,intclient_fd;intold_errno;-fd=socket(AF_VSOCK,SOCK_STREAM,0);+fd=socket(AF_VSOCK,type,0);if(bind(fd,&addr.sa,sizeof(addr.svm))<0){perror("bind");
@@ -189,6 +199,18 @@ int vsock_stream_accept(unsigned int cid, unsigned int port,returnclient_fd;}+intvsock_stream_accept(unsignedintcid,unsignedintport,+structsockaddr_vm*clientaddrp)+{+returnvsock_accept(cid,port,clientaddrp,SOCK_STREAM);+}++intvsock_seqpacket_accept(unsignedintcid,unsignedintport,+structsockaddr_vm*clientaddrp)+{+returnvsock_accept(cid,port,clientaddrp,SOCK_SEQPACKET);+}+/* Transmit one byte and check the return value.**expected_ret:
Hi Arseny,
thanks for this new version.
It's a busy week for me, but I hope to review this series by the end of
this week :-)
Thanks,
Stefano
On Sun, Mar 07, 2021 at 08:57:19PM +0300, Arseny Krasnov wrote:
This patchset implements support of SOCK_SEQPACKET for virtio
transport.
As SOCK_SEQPACKET guarantees to save record boundaries, so to
do it, two new packet operations were added: first for start of record
and second to mark end of record(SEQ_BEGIN and SEQ_END later). Also,
both operations carries metadata - to maintain boundaries and payload
integrity. Metadata is introduced by adding special header with two
fields - message id and message length:
struct virtio_vsock_seq_hdr {
__le32 msg_id;
__le32 msg_len;
} __attribute__((packed));
This header is transmitted as payload of SEQ_BEGIN and SEQ_END
packets(buffer of second virtio descriptor in chain) in the same way as
data transmitted in RW packets. Payload was chosen as buffer for this
header to avoid touching first virtio buffer which carries header of
packet, because someone could check that size of this buffer is equal
to size of packet header. To send record, packet with start marker is
sent first(it's header carries length of record and id),then all data
is sent as usual 'RW' packets and finally SEQ_END is sent(it carries
id of message, which is equal to id of SEQ_BEGIN), also after sending
SEQ_END id is incremented. On receiver's side,size of record is known
from packet with start record marker. To check that no packets were
dropped by transport, 'msg_id's of two sequential SEQ_BEGIN and SEQ_END
are checked to be equal and length of data between two markers is
compared to then length in SEQ_BEGIN header.
Now as packets of one socket are not reordered neither on
vsock nor on vhost transport layers, such markers allows to restore
original record on receiver's side. If user's buffer is smaller that
record length, when all out of size data is dropped.
Maximum length of datagram is not limited as in stream socket,
because same credit logic is used. Difference with stream socket is
that user is not woken up until whole record is received or error
occurred. Implementation also supports 'MSG_EOR' and 'MSG_TRUNC' flags.
Tests also implemented.
Thanks to stsp2@yandex.ru for encouragements and initial design
recommendations.
Arseny Krasnov (22):
af_vsock: update functions for connectible socket
af_vsock: separate wait data loop
af_vsock: separate receive data loop
af_vsock: implement SEQPACKET receive loop
af_vsock: separate wait space loop
af_vsock: implement send logic for SEQPACKET
af_vsock: rest of SEQPACKET support
af_vsock: update comments for stream sockets
virtio/vsock: set packet's type in virtio_transport_send_pkt_info()
virtio/vsock: simplify credit update function API
virtio/vsock: dequeue callback for SOCK_SEQPACKET
virtio/vsock: fetch length for SEQPACKET record
virtio/vsock: add SEQPACKET receive logic
virtio/vsock: rest of SOCK_SEQPACKET support
virtio/vsock: SEQPACKET feature bit
vhost/vsock: SEQPACKET feature bit support
virtio/vsock: SEQPACKET feature bit support
virtio/vsock: setup SEQPACKET ops for transport
vhost/vsock: setup SEQPACKET ops for transport
vsock/loopback: setup SEQPACKET ops for transport
vsock_test: add SOCK_SEQPACKET tests
virtio/vsock: update trace event for SEQPACKET
drivers/vhost/vsock.c | 22 +-
include/linux/virtio_vsock.h | 22 +
include/net/af_vsock.h | 10 +
.../events/vsock_virtio_transport_common.h | 48 +-
include/uapi/linux/virtio_vsock.h | 19 +
net/vmw_vsock/af_vsock.c | 589 +++++++++++------
net/vmw_vsock/virtio_transport.c | 18 +
net/vmw_vsock/virtio_transport_common.c | 364 ++++++++--
net/vmw_vsock/vsock_loopback.c | 13 +
tools/testing/vsock/util.c | 32 +-
tools/testing/vsock/util.h | 3 +
tools/testing/vsock/vsock_test.c | 126 ++++
12 files changed, 1013 insertions(+), 253 deletions(-)
v5 -> v6:
General changelog:
- virtio transport specific callbacks which send SEQ_BEGIN or
SEQ_END now hidden inside virtio transport. Only enqueue,
dequeue and record length callbacks are provided by transport.
- virtio feature bit for SEQPACKET socket support introduced:
VIRTIO_VSOCK_F_SEQPACKET.
- 'msg_cnt' field in 'struct virtio_vsock_seq_hdr' renamed to
'msg_id' and used as id.
Per patch changelog:
- 'af_vsock: separate wait data loop':
1) Commit message updated.
2) 'prepare_to_wait()' moved inside while loop(thanks to
Jorgen Hansen).
Marked 'Reviewed-by' with 1), but as 2) I removed R-b.
- 'af_vsock: separate receive data loop': commit message
updated.
Marked 'Reviewed-by' with that fix.
- 'af_vsock: implement SEQPACKET receive loop': style fixes.
- 'af_vsock: rest of SEQPACKET support':
1) 'module_put()' added when transport callback check failed.
2) Now only 'seqpacket_allow()' callback called to check
support of SEQPACKET by transport.
- 'af_vsock: update comments for stream sockets': commit message
updated.
Marked 'Reviewed-by' with that fix.
- 'virtio/vsock: set packet's type in send':
1) Commit message updated.
2) Parameter 'type' from 'virtio_transport_send_credit_update()'
also removed in this patch instead of in next.
- 'virtio/vsock: dequeue callback for SOCK_SEQPACKET': SEQPACKET
related state wrapped to special struct.
- 'virtio/vsock: update trace event for SEQPACKET': format strings
now not broken by new lines.
v4 -> v5:
- patches reorganized:
1) Setting of packet's type in 'virtio_transport_send_pkt_info()'
is moved to separate patch.
2) Simplifying of 'virtio_transport_send_credit_update()' is
moved to separate patch and before main virtio/vsock patches.
- style problem fixed
- in 'af_vsock: separate receive data loop' extra 'release_sock()'
removed
- added trace event fields for SEQPACKET
- in 'af_vsock: separate wait data loop':
1) 'vsock_wait_data()' removed 'goto out;'
2) Comment for invalid data amount is changed.
- in 'af_vsock: rest of SEQPACKET support', 'new_transport' pointer
check is moved after 'try_module_get()'
- in 'af_vsock: update comments for stream sockets', 'connect-oriented'
replaced with 'connection-oriented'
- in 'loopback/vsock: setup SEQPACKET ops for transport',
'loopback/vsock' replaced with 'vsock/loopback'
v3 -> v4:
- SEQPACKET specific metadata moved from packet header to payload
and called 'virtio_vsock_seq_hdr'
- record integrity check:
1) SEQ_END operation was added, which marks end of record.
2) Both SEQ_BEGIN and SEQ_END carries counter which is incremented
on every marker send.
- af_vsock.c: socket operations for STREAM and SEQPACKET call same
functions instead of having own "gates" differs only by names:
'vsock_seqpacket/stream_getsockopt()' now replaced with
'vsock_connectible_getsockopt()'.
- af_vsock.c: 'seqpacket_dequeue' callback returns error and flag that
record ready. There is no need to return number of copied bytes,
because case when record received successfully is checked at virtio
transport layer, when SEQ_END is processed. Also user doesn't need
number of copied bytes, because 'recv()' from SEQPACKET could return
error, length of users's buffer or length of whole record(both are
known in af_vsock.c).
- af_vsock.c: both wait loops in af_vsock.c(for data and space) moved
to separate functions because now both called from several places.
- af_vsock.c: 'vsock_assign_transport()' checks that 'new_transport'
pointer is not NULL and returns 'ESOCKTNOSUPPORT' instead of 'ENODEV'
if failed to use transport.
- tools/testing/vsock/vsock_test.c: rename tests
v2 -> v3:
- patches reorganized: split for prepare and implementation patches
- local variables are declared in "Reverse Christmas tree" manner
- virtio_transport_common.c: valid leXX_to_cpu() for vsock header
fields access
- af_vsock.c: 'vsock_connectible_*sockopt()' added as shared code
between stream and seqpacket sockets.
- af_vsock.c: loops in '__vsock_*_recvmsg()' refactored.
- af_vsock.c: 'vsock_wait_data()' refactored.
v1 -> v2:
- patches reordered: af_vsock.c related changes now before virtio vsock
- patches reorganized: more small patches, where +/- are not mixed
- tests for SOCK_SEQPACKET added
- all commit messages updated
- af_vsock.c: 'vsock_pre_recv_check()' inlined to
'vsock_connectible_recvmsg()'
- af_vsock.c: 'vsock_assign_transport()' returns ENODEV if transport
was not found
- virtio_transport_common.c: transport callback for seqpacket dequeue
- virtio_transport_common.c: simplified
'virtio_transport_recv_connected()'
- virtio_transport_common.c: send reset on socket and packet type
mismatch.
Signed-off-by: Arseny Krasnov <redacted>
--
2.25.1
Hello, great, no problem!
Thanks
On 10.03.2021 13:06, Stefano Garzarella wrote:
Hi Arseny,
thanks for this new version.
It's a busy week for me, but I hope to review this series by the end of
this week :-)
Thanks,
Stefano
On Sun, Mar 07, 2021 at 08:57:19PM +0300, Arseny Krasnov wrote:
quoted
This patchset implements support of SOCK_SEQPACKET for virtio
transport.
As SOCK_SEQPACKET guarantees to save record boundaries, so to
do it, two new packet operations were added: first for start of record
and second to mark end of record(SEQ_BEGIN and SEQ_END later). Also,
both operations carries metadata - to maintain boundaries and payload
integrity. Metadata is introduced by adding special header with two
fields - message id and message length:
struct virtio_vsock_seq_hdr {
__le32 msg_id;
__le32 msg_len;
} __attribute__((packed));
This header is transmitted as payload of SEQ_BEGIN and SEQ_END
packets(buffer of second virtio descriptor in chain) in the same way as
data transmitted in RW packets. Payload was chosen as buffer for this
header to avoid touching first virtio buffer which carries header of
packet, because someone could check that size of this buffer is equal
to size of packet header. To send record, packet with start marker is
sent first(it's header carries length of record and id),then all data
is sent as usual 'RW' packets and finally SEQ_END is sent(it carries
id of message, which is equal to id of SEQ_BEGIN), also after sending
SEQ_END id is incremented. On receiver's side,size of record is known
from packet with start record marker. To check that no packets were
dropped by transport, 'msg_id's of two sequential SEQ_BEGIN and SEQ_END
are checked to be equal and length of data between two markers is
compared to then length in SEQ_BEGIN header.
Now as packets of one socket are not reordered neither on
vsock nor on vhost transport layers, such markers allows to restore
original record on receiver's side. If user's buffer is smaller that
record length, when all out of size data is dropped.
Maximum length of datagram is not limited as in stream socket,
because same credit logic is used. Difference with stream socket is
that user is not woken up until whole record is received or error
occurred. Implementation also supports 'MSG_EOR' and 'MSG_TRUNC' flags.
Tests also implemented.
Thanks to stsp2@yandex.ru for encouragements and initial design
recommendations.
Arseny Krasnov (22):
af_vsock: update functions for connectible socket
af_vsock: separate wait data loop
af_vsock: separate receive data loop
af_vsock: implement SEQPACKET receive loop
af_vsock: separate wait space loop
af_vsock: implement send logic for SEQPACKET
af_vsock: rest of SEQPACKET support
af_vsock: update comments for stream sockets
virtio/vsock: set packet's type in virtio_transport_send_pkt_info()
virtio/vsock: simplify credit update function API
virtio/vsock: dequeue callback for SOCK_SEQPACKET
virtio/vsock: fetch length for SEQPACKET record
virtio/vsock: add SEQPACKET receive logic
virtio/vsock: rest of SOCK_SEQPACKET support
virtio/vsock: SEQPACKET feature bit
vhost/vsock: SEQPACKET feature bit support
virtio/vsock: SEQPACKET feature bit support
virtio/vsock: setup SEQPACKET ops for transport
vhost/vsock: setup SEQPACKET ops for transport
vsock/loopback: setup SEQPACKET ops for transport
vsock_test: add SOCK_SEQPACKET tests
virtio/vsock: update trace event for SEQPACKET
drivers/vhost/vsock.c | 22 +-
include/linux/virtio_vsock.h | 22 +
include/net/af_vsock.h | 10 +
.../events/vsock_virtio_transport_common.h | 48 +-
include/uapi/linux/virtio_vsock.h | 19 +
net/vmw_vsock/af_vsock.c | 589 +++++++++++------
net/vmw_vsock/virtio_transport.c | 18 +
net/vmw_vsock/virtio_transport_common.c | 364 ++++++++--
net/vmw_vsock/vsock_loopback.c | 13 +
tools/testing/vsock/util.c | 32 +-
tools/testing/vsock/util.h | 3 +
tools/testing/vsock/vsock_test.c | 126 ++++
12 files changed, 1013 insertions(+), 253 deletions(-)
v5 -> v6:
General changelog:
- virtio transport specific callbacks which send SEQ_BEGIN or
SEQ_END now hidden inside virtio transport. Only enqueue,
dequeue and record length callbacks are provided by transport.
- virtio feature bit for SEQPACKET socket support introduced:
VIRTIO_VSOCK_F_SEQPACKET.
- 'msg_cnt' field in 'struct virtio_vsock_seq_hdr' renamed to
'msg_id' and used as id.
Per patch changelog:
- 'af_vsock: separate wait data loop':
1) Commit message updated.
2) 'prepare_to_wait()' moved inside while loop(thanks to
Jorgen Hansen).
Marked 'Reviewed-by' with 1), but as 2) I removed R-b.
- 'af_vsock: separate receive data loop': commit message
updated.
Marked 'Reviewed-by' with that fix.
- 'af_vsock: implement SEQPACKET receive loop': style fixes.
- 'af_vsock: rest of SEQPACKET support':
1) 'module_put()' added when transport callback check failed.
2) Now only 'seqpacket_allow()' callback called to check
support of SEQPACKET by transport.
- 'af_vsock: update comments for stream sockets': commit message
updated.
Marked 'Reviewed-by' with that fix.
- 'virtio/vsock: set packet's type in send':
1) Commit message updated.
2) Parameter 'type' from 'virtio_transport_send_credit_update()'
also removed in this patch instead of in next.
- 'virtio/vsock: dequeue callback for SOCK_SEQPACKET': SEQPACKET
related state wrapped to special struct.
- 'virtio/vsock: update trace event for SEQPACKET': format strings
now not broken by new lines.
v4 -> v5:
- patches reorganized:
1) Setting of packet's type in 'virtio_transport_send_pkt_info()'
is moved to separate patch.
2) Simplifying of 'virtio_transport_send_credit_update()' is
moved to separate patch and before main virtio/vsock patches.
- style problem fixed
- in 'af_vsock: separate receive data loop' extra 'release_sock()'
removed
- added trace event fields for SEQPACKET
- in 'af_vsock: separate wait data loop':
1) 'vsock_wait_data()' removed 'goto out;'
2) Comment for invalid data amount is changed.
- in 'af_vsock: rest of SEQPACKET support', 'new_transport' pointer
check is moved after 'try_module_get()'
- in 'af_vsock: update comments for stream sockets', 'connect-oriented'
replaced with 'connection-oriented'
- in 'loopback/vsock: setup SEQPACKET ops for transport',
'loopback/vsock' replaced with 'vsock/loopback'
v3 -> v4:
- SEQPACKET specific metadata moved from packet header to payload
and called 'virtio_vsock_seq_hdr'
- record integrity check:
1) SEQ_END operation was added, which marks end of record.
2) Both SEQ_BEGIN and SEQ_END carries counter which is incremented
on every marker send.
- af_vsock.c: socket operations for STREAM and SEQPACKET call same
functions instead of having own "gates" differs only by names:
'vsock_seqpacket/stream_getsockopt()' now replaced with
'vsock_connectible_getsockopt()'.
- af_vsock.c: 'seqpacket_dequeue' callback returns error and flag that
record ready. There is no need to return number of copied bytes,
because case when record received successfully is checked at virtio
transport layer, when SEQ_END is processed. Also user doesn't need
number of copied bytes, because 'recv()' from SEQPACKET could return
error, length of users's buffer or length of whole record(both are
known in af_vsock.c).
- af_vsock.c: both wait loops in af_vsock.c(for data and space) moved
to separate functions because now both called from several places.
- af_vsock.c: 'vsock_assign_transport()' checks that 'new_transport'
pointer is not NULL and returns 'ESOCKTNOSUPPORT' instead of 'ENODEV'
if failed to use transport.
- tools/testing/vsock/vsock_test.c: rename tests
v2 -> v3:
- patches reorganized: split for prepare and implementation patches
- local variables are declared in "Reverse Christmas tree" manner
- virtio_transport_common.c: valid leXX_to_cpu() for vsock header
fields access
- af_vsock.c: 'vsock_connectible_*sockopt()' added as shared code
between stream and seqpacket sockets.
- af_vsock.c: loops in '__vsock_*_recvmsg()' refactored.
- af_vsock.c: 'vsock_wait_data()' refactored.
v1 -> v2:
- patches reordered: af_vsock.c related changes now before virtio vsock
- patches reorganized: more small patches, where +/- are not mixed
- tests for SOCK_SEQPACKET added
- all commit messages updated
- af_vsock.c: 'vsock_pre_recv_check()' inlined to
'vsock_connectible_recvmsg()'
- af_vsock.c: 'vsock_assign_transport()' returns ENODEV if transport
was not found
- virtio_transport_common.c: transport callback for seqpacket dequeue
- virtio_transport_common.c: simplified
'virtio_transport_recv_connected()'
- virtio_transport_common.c: send reset on socket and packet type
mismatch.
Signed-off-by: Arseny Krasnov <redacted>
--
2.25.1
On Sun, Mar 07, 2021 at 08:58:39PM +0300, Arseny Krasnov wrote:
This prepares af_vsock.c for SEQPACKET support: some functions such
as setsockopt(), getsockopt(), connect(), recvmsg(), sendmsg() are
shared between both types of sockets, so rename them in general
manner.
Signed-off-by: Arseny Krasnov <redacted>
---
net/vmw_vsock/af_vsock.c | 64 +++++++++++++++++++++-------------------
1 file changed, 34 insertions(+), 30 deletions(-)
On Sun, Mar 07, 2021 at 08:59:01PM +0300, Arseny Krasnov wrote:
This moves wait loop for data to dedicated function, because later it
will be used by SEQPACKET data receive loop. While moving the code
around, let's update an old comment.
Signed-off-by: Arseny Krasnov <redacted>
---
net/vmw_vsock/af_vsock.c | 156 +++++++++++++++++++++------------------
1 file changed, 84 insertions(+), 72 deletions(-)
On Sun, Mar 07, 2021 at 08:59:45PM +0300, Arseny Krasnov wrote:
This adds receive loop for SEQPACKET. It looks like receive loop for
STREAM, but there is a little bit difference:
1) It doesn't call notify callbacks.
2) It doesn't care about 'SO_SNDLOWAT' and 'SO_RCVLOWAT' values, because
there is no sense for these values in SEQPACKET case.
3) It waits until whole record is received or error is found during
receiving.
4) It processes and sets 'MSG_TRUNC' flag.
So to avoid extra conditions for two types of socket inside one loop, two
independent functions were created.
Signed-off-by: Arseny Krasnov <redacted>
---
include/net/af_vsock.h | 5 +++
net/vmw_vsock/af_vsock.c | 95 +++++++++++++++++++++++++++++++++++++++-
2 files changed, 99 insertions(+), 1 deletion(-)
On Sun, Mar 07, 2021 at 09:00:26PM +0300, Arseny Krasnov wrote:
quoted hunk
This adds some logic to current stream enqueue function for SEQPACKET
support:
1) Use transport's seqpacket enqueue callback.
2) Return value from enqueue function is whole record length or error
for SOCK_SEQPACKET.
Signed-off-by: Arseny Krasnov <redacted>
---
include/net/af_vsock.h | 2 ++
net/vmw_vsock/af_vsock.c | 22 ++++++++++++++++------
2 files changed, 18 insertions(+), 6 deletions(-)
* responsibility to check how many bytes we were able to send.
*/
- written = transport->stream_enqueue(
- vsk, msg,
- len - total_written);
+ if (sk->sk_type == SOCK_SEQPACKET) {
+ written = transport->seqpacket_enqueue(vsk,
+ msg, msg->msg_flags,
I think we can avoid to pass 'msg->msg_flags', since the transport can
access it through the 'msg' pointer, right?
quoted hunk
+ len - total_written);
+ } else {
+ written = transport->stream_enqueue(vsk,
+ msg, len - total_written);
+ }
if (written < 0) {
err = -ENOMEM;
goto out_err;
On Sun, Mar 07, 2021 at 08:59:45PM +0300, Arseny Krasnov wrote:
quoted hunk
This adds receive loop for SEQPACKET. It looks like receive loop for
STREAM, but there is a little bit difference:
1) It doesn't call notify callbacks.
2) It doesn't care about 'SO_SNDLOWAT' and 'SO_RCVLOWAT' values, because
there is no sense for these values in SEQPACKET case.
3) It waits until whole record is received or error is found during
receiving.
4) It processes and sets 'MSG_TRUNC' flag.
So to avoid extra conditions for two types of socket inside one loop, two
independent functions were created.
Signed-off-by: Arseny Krasnov <redacted>
---
include/net/af_vsock.h | 5 +++
net/vmw_vsock/af_vsock.c | 95 +++++++++++++++++++++++++++++++++++++++-
2 files changed, 99 insertions(+), 1 deletion(-)
return err;
}
+static int __vsock_seqpacket_recvmsg(struct sock *sk, struct msghdr *msg,
+ size_t len, int flags)
+{
+ const struct vsock_transport *transport;
+ const struct iovec *orig_iov;
+ unsigned long orig_nr_segs;
+ bool msg_ready;
+ struct vsock_sock *vsk;
+ size_t record_len;
+ long timeout;
+ int err = 0;
+ DEFINE_WAIT(wait);
+
+ vsk = vsock_sk(sk);
+ transport = vsk->transport;
+
+ timeout = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
+ orig_nr_segs = msg->msg_iter.nr_segs;
+ orig_iov = msg->msg_iter.iov;
+ msg_ready = false;
+ record_len = 0;
+
+ while (1) {
+ err = vsock_wait_data(sk, &wait, timeout, NULL, 0);
+
+ if (err <= 0) {
+ /* In case of any loop break(timeout, signal
+ * interrupt or shutdown), we report user that
+ * nothing was copied.
+ */
+ err = 0;
+ break;
+ }
+
+ if (record_len == 0) {
+ record_len =
+ transport->seqpacket_seq_get_len(vsk);
+
+ if (record_len == 0)
+ continue;
+ }
+
+ err = transport->seqpacket_dequeue(vsk, msg, flags, &msg_ready);
In order to simplify the transport interface, can we do the work of
seqpacket_seq_get_len() at the beginning of seqpacket_dequeue()?
So in this way seqpacket_dequeue() can return the 'record_len' or an
error.
On Sun, Mar 07, 2021 at 09:02:31PM +0300, Arseny Krasnov wrote:
quoted hunk
This adds transport callback which tries to fetch record begin marker
from socket's rx queue. It is called from af_vsock.c before reading data
packets of record.
Signed-off-by: Arseny Krasnov <redacted>
---
include/linux/virtio_vsock.h | 1 +
net/vmw_vsock/virtio_transport_common.c | 53 +++++++++++++++++++++++++
2 files changed, 54 insertions(+)
On Sun, Mar 07, 2021 at 09:01:05PM +0300, Arseny Krasnov wrote:
This replaces 'stream' to 'connection oriented' in comments as
SEQPACKET is also connection oriented.
Signed-off-by: Arseny Krasnov <redacted>
---
net/vmw_vsock/af_vsock.c | 31 +++++++++++++++++--------------
1 file changed, 17 insertions(+), 14 deletions(-)
On Sun, Mar 07, 2021 at 09:01:22PM +0300, Arseny Krasnov wrote:
This moves passing type of packet from 'info' structure to 'virtio_
transport_send_pkt_info()' function. There is no need to set type of
packet which differs from type of socket. Since at current time only
stream type is supported, set it directly in 'virtio_transport_send_
pkt_info()', so callers don't need to set it.
Signed-off-by: Arseny Krasnov <redacted>
---
net/vmw_vsock/virtio_transport_common.c | 19 +++++--------------
1 file changed, 5 insertions(+), 14 deletions(-)
On Sun, Mar 07, 2021 at 09:02:31PM +0300, Arseny Krasnov wrote:
quoted
This adds transport callback which tries to fetch record begin marker
from socket's rx queue. It is called from af_vsock.c before reading data
packets of record.
Signed-off-by: Arseny Krasnov <redacted>
---
include/linux/virtio_vsock.h | 1 +
net/vmw_vsock/virtio_transport_common.c | 53 +++++++++++++++++++++++++
2 files changed, 54 insertions(+)
On Sun, Mar 07, 2021 at 08:59:45PM +0300, Arseny Krasnov wrote:
quoted
This adds receive loop for SEQPACKET. It looks like receive loop for
STREAM, but there is a little bit difference:
1) It doesn't call notify callbacks.
2) It doesn't care about 'SO_SNDLOWAT' and 'SO_RCVLOWAT' values, because
there is no sense for these values in SEQPACKET case.
3) It waits until whole record is received or error is found during
receiving.
4) It processes and sets 'MSG_TRUNC' flag.
So to avoid extra conditions for two types of socket inside one loop, two
independent functions were created.
Signed-off-by: Arseny Krasnov <redacted>
---
include/net/af_vsock.h | 5 +++
net/vmw_vsock/af_vsock.c | 95 +++++++++++++++++++++++++++++++++++++++-
2 files changed, 99 insertions(+), 1 deletion(-)
return err;
}
+static int __vsock_seqpacket_recvmsg(struct sock *sk, struct msghdr *msg,
+ size_t len, int flags)
+{
+ const struct vsock_transport *transport;
+ const struct iovec *orig_iov;
+ unsigned long orig_nr_segs;
+ bool msg_ready;
+ struct vsock_sock *vsk;
+ size_t record_len;
+ long timeout;
+ int err = 0;
+ DEFINE_WAIT(wait);
+
+ vsk = vsock_sk(sk);
+ transport = vsk->transport;
+
+ timeout = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
+ orig_nr_segs = msg->msg_iter.nr_segs;
+ orig_iov = msg->msg_iter.iov;
+ msg_ready = false;
+ record_len = 0;
+
+ while (1) {
+ err = vsock_wait_data(sk, &wait, timeout, NULL, 0);
+
+ if (err <= 0) {
+ /* In case of any loop break(timeout, signal
+ * interrupt or shutdown), we report user that
+ * nothing was copied.
+ */
+ err = 0;
+ break;
+ }
+
+ if (record_len == 0) {
+ record_len =
+ transport->seqpacket_seq_get_len(vsk);
+
+ if (record_len == 0)
+ continue;
+ }
+
+ err = transport->seqpacket_dequeue(vsk, msg, flags, &msg_ready);
In order to simplify the transport interface, can we do the work of
seqpacket_seq_get_len() at the beginning of seqpacket_dequeue()?
So in this way seqpacket_dequeue() can return the 'record_len' or an
error.
On Sun, Mar 07, 2021 at 09:00:26PM +0300, Arseny Krasnov wrote:
quoted
This adds some logic to current stream enqueue function for SEQPACKET
support:
1) Use transport's seqpacket enqueue callback.
2) Return value from enqueue function is whole record length or error
for SOCK_SEQPACKET.
Signed-off-by: Arseny Krasnov <redacted>
---
include/net/af_vsock.h | 2 ++
net/vmw_vsock/af_vsock.c | 22 ++++++++++++++++------
2 files changed, 18 insertions(+), 6 deletions(-)
* responsibility to check how many bytes we were able to send.
*/
- written = transport->stream_enqueue(
- vsk, msg,
- len - total_written);
+ if (sk->sk_type == SOCK_SEQPACKET) {
+ written = transport->seqpacket_enqueue(vsk,
+ msg, msg->msg_flags,
I think we can avoid to pass 'msg->msg_flags', since the transport can
access it through the 'msg' pointer, right?
Ack
quoted
+ len - total_written);
+ } else {
+ written = transport->stream_enqueue(vsk,
+ msg, len - total_written);
+ }
if (written < 0) {
err = -ENOMEM;
goto out_err;
On Sun, Mar 07, 2021 at 09:02:01PM +0300, Arseny Krasnov wrote:
quoted hunk
This adds transport callback and it's logic for SEQPACKET dequeue.
Callback fetches RW packets from rx queue of socket until whole record
is copied(if user's buffer is full, user is not woken up). This is done
to not stall sender, because if we wake up user and it leaves syscall,
nobody will send credit update for rest of record, and sender will wait
for next enter of read syscall at receiver's side. So if user buffer is
full, we just send credit update and drop data. If during copy SEQ_BEGIN
was found(and not all data was copied), copying is restarted by reset
user's iov iterator(previous unfinished data is dropped).
Signed-off-by: Arseny Krasnov <redacted>
---
include/linux/virtio_vsock.h | 13 +++
include/uapi/linux/virtio_vsock.h | 16 ++++
net/vmw_vsock/virtio_transport_common.c | 116 ++++++++++++++++++++++++
3 files changed, 145 insertions(+)
On Sun, Mar 07, 2021 at 09:02:50PM +0300, Arseny Krasnov wrote:
quoted hunk
This modifies current receive logic for SEQPACKET support:
1) Inserts 'SEQ_BEGIN' packet to socket's rx queue.
2) Inserts 'RW' packet to socket's rx queue, but without merging with
buffer of last packet in queue.
3) Performs check for packet and socket types on receive(if mismatch,
then reset connection).
Signed-off-by: Arseny Krasnov <redacted>
---
net/vmw_vsock/virtio_transport_common.c | 63 +++++++++++++++++--------
1 file changed, 44 insertions(+), 19 deletions(-)
}
EXPORT_SYMBOL_GPL(virtio_transport_deliver_tap_pkt);
+static u16 virtio_transport_get_type(struct sock *sk)
+{
+ if (sk->sk_type == SOCK_STREAM)
+ return VIRTIO_VSOCK_TYPE_STREAM;
+ else
+ return VIRTIO_VSOCK_TYPE_SEQPACKET;
+}
+
/* This function can only be used on connecting/connected sockets,
* since a socket assigned to a transport is required.
*
goto out;
}
- /* Try to copy small packets into the buffer of last packet queued,
- * to avoid wasting memory queueing the entire buffer with a small
- * payload.
- */
- if (pkt->len <= GOOD_COPY_LEN && !list_empty(&vvs->rx_queue)) {
- struct virtio_vsock_pkt *last_pkt;
+ if (le16_to_cpu(pkt->hdr.type) == VIRTIO_VSOCK_TYPE_STREAM) {
+ /* Try to copy small packets into the buffer of last packet queued,
+ * to avoid wasting memory queueing the entire buffer with a small
+ * payload.
+ */
+ if (pkt->len <= GOOD_COPY_LEN && !list_empty(&vvs->rx_queue)) {
+ struct virtio_vsock_pkt *last_pkt;
- last_pkt = list_last_entry(&vvs->rx_queue,
- struct virtio_vsock_pkt, list);
+ last_pkt = list_last_entry(&vvs->rx_queue,
+ struct virtio_vsock_pkt, list);
- /* If there is space in the last packet queued, we copy the
- * new packet in its buffer.
- */
- if (pkt->len <= last_pkt->buf_len - last_pkt->len) {
- memcpy(last_pkt->buf + last_pkt->len, pkt->buf,
- pkt->len);
- last_pkt->len += pkt->len;
- free_pkt = true;
- goto out;
+ /* If there is space in the last packet queued, we copy the
+ * new packet in its buffer.
+ */
+ if (pkt->len <= last_pkt->buf_len - last_pkt->len) {
+ memcpy(last_pkt->buf + last_pkt->len, pkt->buf,
+ pkt->len);
+ last_pkt->len += pkt->len;
+ free_pkt = true;
+ goto out;
+ }
}
}
On Sun, Mar 07, 2021 at 09:03:09PM +0300, Arseny Krasnov wrote:
quoted hunk
This adds rest of logic for SEQPACKET:
1) SEQPACKET specific functions which send SEQ_BEGIN/SEQ_END.
Note that both functions may sleep to wait enough space for
SEQPACKET header.
2) SEQ_BEGIN/SEQ_END in TAP packet capture.
3) Send SHUTDOWN on socket close for SEQPACKET type.
4) Set SEQPACKET packet type during send.
5) Set MSG_EOR in flags for SEQPACKET during send.
6) 'seqpacket_allow' flag to virtio transport.
Signed-off-by: Arseny Krasnov <redacted>
---
include/linux/virtio_vsock.h | 8 +++
net/vmw_vsock/virtio_transport_common.c | 87 ++++++++++++++++++++++++-
2 files changed, 93 insertions(+), 2 deletions(-)
break;
case VIRTIO_VSOCK_OP_CREDIT_UPDATE:
case VIRTIO_VSOCK_OP_CREDIT_REQUEST:
+ case VIRTIO_VSOCK_OP_SEQ_BEGIN:
+ case VIRTIO_VSOCK_OP_SEQ_END:
hdr->op = cpu_to_le16(AF_VSOCK_OP_CONTROL);
break;
default:
@@ -187,7 +189,12 @@ static int virtio_transport_send_pkt_info(struct vsock_sock *vsk,
I suspect we should increment next_tx_msg_id even in case of an error to
avoid issues with packets with same IDs, so in case of error I would do:
if (/* error */) {
written = -1;
goto out;
}
Then we can add the 'out' label and the id increment:
out:
vvs->seqpacket_state.next_tx_msg_id++;
On Sun, Mar 07, 2021 at 09:03:41PM +0300, Arseny Krasnov wrote:
This adds handling of SEQPACKET bit: if guest sets features with
this bit cleared, then SOCK_SEQPACKET support will be disabled.
Signed-off-by: Arseny Krasnov <redacted>
---
drivers/vhost/vsock.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
I think is better to move this patch after we set the seqpackets ops,
so we are really able to handle SEQPACKET traffic.
Hi Arseny,
On Sun, Mar 07, 2021 at 08:57:19PM +0300, Arseny Krasnov wrote:
This patchset implements support of SOCK_SEQPACKET for virtio
transport.
As SOCK_SEQPACKET guarantees to save record boundaries, so to
do it, two new packet operations were added: first for start of record
and second to mark end of record(SEQ_BEGIN and SEQ_END later). Also,
both operations carries metadata - to maintain boundaries and payload
integrity. Metadata is introduced by adding special header with two
fields - message id and message length:
struct virtio_vsock_seq_hdr {
__le32 msg_id;
__le32 msg_len;
} __attribute__((packed));
This header is transmitted as payload of SEQ_BEGIN and SEQ_END
packets(buffer of second virtio descriptor in chain) in the same way as
data transmitted in RW packets. Payload was chosen as buffer for this
header to avoid touching first virtio buffer which carries header of
packet, because someone could check that size of this buffer is equal
to size of packet header. To send record, packet with start marker is
sent first(it's header carries length of record and id),then all data
is sent as usual 'RW' packets and finally SEQ_END is sent(it carries
id of message, which is equal to id of SEQ_BEGIN), also after sending
SEQ_END id is incremented. On receiver's side,size of record is known
from packet with start record marker. To check that no packets were
dropped by transport, 'msg_id's of two sequential SEQ_BEGIN and SEQ_END
are checked to be equal and length of data between two markers is
compared to then length in SEQ_BEGIN header.
Now as packets of one socket are not reordered neither on
vsock nor on vhost transport layers, such markers allows to restore
original record on receiver's side. If user's buffer is smaller that
record length, when all out of size data is dropped.
Maximum length of datagram is not limited as in stream socket,
because same credit logic is used. Difference with stream socket is
that user is not woken up until whole record is received or error
occurred. Implementation also supports 'MSG_EOR' and 'MSG_TRUNC' flags.
Tests also implemented.
Thanks to stsp2@yandex.ru for encouragements and initial design
recommendations.
Arseny Krasnov (22):
af_vsock: update functions for connectible socket
af_vsock: separate wait data loop
af_vsock: separate receive data loop
af_vsock: implement SEQPACKET receive loop
af_vsock: separate wait space loop
af_vsock: implement send logic for SEQPACKET
af_vsock: rest of SEQPACKET support
af_vsock: update comments for stream sockets
virtio/vsock: set packet's type in virtio_transport_send_pkt_info()
virtio/vsock: simplify credit update function API
virtio/vsock: dequeue callback for SOCK_SEQPACKET
virtio/vsock: fetch length for SEQPACKET record
virtio/vsock: add SEQPACKET receive logic
virtio/vsock: rest of SOCK_SEQPACKET support
virtio/vsock: SEQPACKET feature bit
vhost/vsock: SEQPACKET feature bit support
virtio/vsock: SEQPACKET feature bit support
virtio/vsock: setup SEQPACKET ops for transport
vhost/vsock: setup SEQPACKET ops for transport
vsock/loopback: setup SEQPACKET ops for transport
vsock_test: add SOCK_SEQPACKET tests
virtio/vsock: update trace event for SEQPACKET
drivers/vhost/vsock.c | 22 +-
include/linux/virtio_vsock.h | 22 +
include/net/af_vsock.h | 10 +
.../events/vsock_virtio_transport_common.h | 48 +-
include/uapi/linux/virtio_vsock.h | 19 +
net/vmw_vsock/af_vsock.c | 589 +++++++++++------
net/vmw_vsock/virtio_transport.c | 18 +
net/vmw_vsock/virtio_transport_common.c | 364 ++++++++--
net/vmw_vsock/vsock_loopback.c | 13 +
tools/testing/vsock/util.c | 32 +-
tools/testing/vsock/util.h | 3 +
tools/testing/vsock/vsock_test.c | 126 ++++
12 files changed, 1013 insertions(+), 253 deletions(-)
v5 -> v6:
General changelog:
- virtio transport specific callbacks which send SEQ_BEGIN or
SEQ_END now hidden inside virtio transport. Only enqueue,
dequeue and record length callbacks are provided by transport.
- virtio feature bit for SEQPACKET socket support introduced:
VIRTIO_VSOCK_F_SEQPACKET.
- 'msg_cnt' field in 'struct virtio_vsock_seq_hdr' renamed to
'msg_id' and used as id.
Per patch changelog:
- 'af_vsock: separate wait data loop':
1) Commit message updated.
2) 'prepare_to_wait()' moved inside while loop(thanks to
Jorgen Hansen).
Marked 'Reviewed-by' with 1), but as 2) I removed R-b.
- 'af_vsock: separate receive data loop': commit message
updated.
Marked 'Reviewed-by' with that fix.
- 'af_vsock: implement SEQPACKET receive loop': style fixes.
- 'af_vsock: rest of SEQPACKET support':
1) 'module_put()' added when transport callback check failed.
2) Now only 'seqpacket_allow()' callback called to check
support of SEQPACKET by transport.
- 'af_vsock: update comments for stream sockets': commit message
updated.
Marked 'Reviewed-by' with that fix.
- 'virtio/vsock: set packet's type in send':
1) Commit message updated.
2) Parameter 'type' from 'virtio_transport_send_credit_update()'
also removed in this patch instead of in next.
- 'virtio/vsock: dequeue callback for SOCK_SEQPACKET': SEQPACKET
related state wrapped to special struct.
- 'virtio/vsock: update trace event for SEQPACKET': format strings
now not broken by new lines.
I left a bunch of comments in the patches, I hope they are easy to fix
:-)
Thanks for the changelogs. About 'per patch changelog', it is very
useful!
Just a suggestion, I think is better to include them in each patch after
the '---' to simplify the review.
You can use git-notes(1) or you can simply edit the format-patch and add
the changelog after the 3 dashes, so that they are ignored when the
patch is applied.
Thanks,
Stefano
Hi Arseny,
On Sun, Mar 07, 2021 at 08:57:19PM +0300, Arseny Krasnov wrote:
quoted
This patchset implements support of SOCK_SEQPACKET for virtio
transport.
As SOCK_SEQPACKET guarantees to save record boundaries, so to
do it, two new packet operations were added: first for start of record
and second to mark end of record(SEQ_BEGIN and SEQ_END later). Also,
both operations carries metadata - to maintain boundaries and payload
integrity. Metadata is introduced by adding special header with two
fields - message id and message length:
struct virtio_vsock_seq_hdr {
__le32 msg_id;
__le32 msg_len;
} __attribute__((packed));
This header is transmitted as payload of SEQ_BEGIN and SEQ_END
packets(buffer of second virtio descriptor in chain) in the same way as
data transmitted in RW packets. Payload was chosen as buffer for this
header to avoid touching first virtio buffer which carries header of
packet, because someone could check that size of this buffer is equal
to size of packet header. To send record, packet with start marker is
sent first(it's header carries length of record and id),then all data
is sent as usual 'RW' packets and finally SEQ_END is sent(it carries
id of message, which is equal to id of SEQ_BEGIN), also after sending
SEQ_END id is incremented. On receiver's side,size of record is known
from packet with start record marker. To check that no packets were
dropped by transport, 'msg_id's of two sequential SEQ_BEGIN and SEQ_END
are checked to be equal and length of data between two markers is
compared to then length in SEQ_BEGIN header.
Now as packets of one socket are not reordered neither on
vsock nor on vhost transport layers, such markers allows to restore
original record on receiver's side. If user's buffer is smaller that
record length, when all out of size data is dropped.
Maximum length of datagram is not limited as in stream socket,
because same credit logic is used. Difference with stream socket is
that user is not woken up until whole record is received or error
occurred. Implementation also supports 'MSG_EOR' and 'MSG_TRUNC' flags.
Tests also implemented.
Thanks to stsp2@yandex.ru for encouragements and initial design
recommendations.
Arseny Krasnov (22):
af_vsock: update functions for connectible socket
af_vsock: separate wait data loop
af_vsock: separate receive data loop
af_vsock: implement SEQPACKET receive loop
af_vsock: separate wait space loop
af_vsock: implement send logic for SEQPACKET
af_vsock: rest of SEQPACKET support
af_vsock: update comments for stream sockets
virtio/vsock: set packet's type in virtio_transport_send_pkt_info()
virtio/vsock: simplify credit update function API
virtio/vsock: dequeue callback for SOCK_SEQPACKET
virtio/vsock: fetch length for SEQPACKET record
virtio/vsock: add SEQPACKET receive logic
virtio/vsock: rest of SOCK_SEQPACKET support
virtio/vsock: SEQPACKET feature bit
vhost/vsock: SEQPACKET feature bit support
virtio/vsock: SEQPACKET feature bit support
virtio/vsock: setup SEQPACKET ops for transport
vhost/vsock: setup SEQPACKET ops for transport
vsock/loopback: setup SEQPACKET ops for transport
vsock_test: add SOCK_SEQPACKET tests
virtio/vsock: update trace event for SEQPACKET
drivers/vhost/vsock.c | 22 +-
include/linux/virtio_vsock.h | 22 +
include/net/af_vsock.h | 10 +
.../events/vsock_virtio_transport_common.h | 48 +-
include/uapi/linux/virtio_vsock.h | 19 +
net/vmw_vsock/af_vsock.c | 589 +++++++++++------
net/vmw_vsock/virtio_transport.c | 18 +
net/vmw_vsock/virtio_transport_common.c | 364 ++++++++--
net/vmw_vsock/vsock_loopback.c | 13 +
tools/testing/vsock/util.c | 32 +-
tools/testing/vsock/util.h | 3 +
tools/testing/vsock/vsock_test.c | 126 ++++
12 files changed, 1013 insertions(+), 253 deletions(-)
v5 -> v6:
General changelog:
- virtio transport specific callbacks which send SEQ_BEGIN or
SEQ_END now hidden inside virtio transport. Only enqueue,
dequeue and record length callbacks are provided by transport.
- virtio feature bit for SEQPACKET socket support introduced:
VIRTIO_VSOCK_F_SEQPACKET.
- 'msg_cnt' field in 'struct virtio_vsock_seq_hdr' renamed to
'msg_id' and used as id.
Per patch changelog:
- 'af_vsock: separate wait data loop':
1) Commit message updated.
2) 'prepare_to_wait()' moved inside while loop(thanks to
Jorgen Hansen).
Marked 'Reviewed-by' with 1), but as 2) I removed R-b.
- 'af_vsock: separate receive data loop': commit message
updated.
Marked 'Reviewed-by' with that fix.
- 'af_vsock: implement SEQPACKET receive loop': style fixes.
- 'af_vsock: rest of SEQPACKET support':
1) 'module_put()' added when transport callback check failed.
2) Now only 'seqpacket_allow()' callback called to check
support of SEQPACKET by transport.
- 'af_vsock: update comments for stream sockets': commit message
updated.
Marked 'Reviewed-by' with that fix.
- 'virtio/vsock: set packet's type in send':
1) Commit message updated.
2) Parameter 'type' from 'virtio_transport_send_credit_update()'
also removed in this patch instead of in next.
- 'virtio/vsock: dequeue callback for SOCK_SEQPACKET': SEQPACKET
related state wrapped to special struct.
- 'virtio/vsock: update trace event for SEQPACKET': format strings
now not broken by new lines.
I left a bunch of comments in the patches, I hope they are easy to fix
:-)
Thank you, yes, there are still small fixes.
Thanks for the changelogs. About 'per patch changelog', it is very
useful!
Just a suggestion, I think is better to include them in each patch after
the '---' to simplify the review.
Ack
You can use git-notes(1) or you can simply edit the format-patch and add
the changelog after the 3 dashes, so that they are ignored when the
patch is applied.
Thanks,
Stefano
Hi Arseny,
On Sun, Mar 07, 2021 at 08:57:19PM +0300, Arseny Krasnov wrote:
quoted
This patchset implements support of SOCK_SEQPACKET for virtio
transport.
As SOCK_SEQPACKET guarantees to save record boundaries, so to
do it, two new packet operations were added: first for start of record
and second to mark end of record(SEQ_BEGIN and SEQ_END later). Also,
both operations carries metadata - to maintain boundaries and payload
integrity. Metadata is introduced by adding special header with two
fields - message id and message length:
struct virtio_vsock_seq_hdr {
__le32 msg_id;
__le32 msg_len;
} __attribute__((packed));
This header is transmitted as payload of SEQ_BEGIN and SEQ_END
packets(buffer of second virtio descriptor in chain) in the same way as
data transmitted in RW packets. Payload was chosen as buffer for this
header to avoid touching first virtio buffer which carries header of
packet, because someone could check that size of this buffer is equal
to size of packet header. To send record, packet with start marker is
sent first(it's header carries length of record and id),then all data
is sent as usual 'RW' packets and finally SEQ_END is sent(it carries
id of message, which is equal to id of SEQ_BEGIN), also after sending
SEQ_END id is incremented. On receiver's side,size of record is known
from packet with start record marker. To check that no packets were
dropped by transport, 'msg_id's of two sequential SEQ_BEGIN and SEQ_END
are checked to be equal and length of data between two markers is
compared to then length in SEQ_BEGIN header.
Now as packets of one socket are not reordered neither on
vsock nor on vhost transport layers, such markers allows to restore
original record on receiver's side. If user's buffer is smaller that
record length, when all out of size data is dropped.
Maximum length of datagram is not limited as in stream socket,
because same credit logic is used. Difference with stream socket is
that user is not woken up until whole record is received or error
occurred. Implementation also supports 'MSG_EOR' and 'MSG_TRUNC' flags.
Tests also implemented.
Thanks to stsp2@yandex.ru for encouragements and initial design
recommendations.
Arseny Krasnov (22):
af_vsock: update functions for connectible socket
af_vsock: separate wait data loop
af_vsock: separate receive data loop
af_vsock: implement SEQPACKET receive loop
af_vsock: separate wait space loop
af_vsock: implement send logic for SEQPACKET
af_vsock: rest of SEQPACKET support
af_vsock: update comments for stream sockets
virtio/vsock: set packet's type in virtio_transport_send_pkt_info()
virtio/vsock: simplify credit update function API
virtio/vsock: dequeue callback for SOCK_SEQPACKET
virtio/vsock: fetch length for SEQPACKET record
virtio/vsock: add SEQPACKET receive logic
virtio/vsock: rest of SOCK_SEQPACKET support
virtio/vsock: SEQPACKET feature bit
vhost/vsock: SEQPACKET feature bit support
virtio/vsock: SEQPACKET feature bit support
virtio/vsock: setup SEQPACKET ops for transport
vhost/vsock: setup SEQPACKET ops for transport
vsock/loopback: setup SEQPACKET ops for transport
vsock_test: add SOCK_SEQPACKET tests
virtio/vsock: update trace event for SEQPACKET
drivers/vhost/vsock.c | 22 +-
include/linux/virtio_vsock.h | 22 +
include/net/af_vsock.h | 10 +
.../events/vsock_virtio_transport_common.h | 48 +-
include/uapi/linux/virtio_vsock.h | 19 +
net/vmw_vsock/af_vsock.c | 589 +++++++++++------
net/vmw_vsock/virtio_transport.c | 18 +
net/vmw_vsock/virtio_transport_common.c | 364 ++++++++--
net/vmw_vsock/vsock_loopback.c | 13 +
tools/testing/vsock/util.c | 32 +-
tools/testing/vsock/util.h | 3 +
tools/testing/vsock/vsock_test.c | 126 ++++
12 files changed, 1013 insertions(+), 253 deletions(-)
v5 -> v6:
General changelog:
- virtio transport specific callbacks which send SEQ_BEGIN or
SEQ_END now hidden inside virtio transport. Only enqueue,
dequeue and record length callbacks are provided by transport.
- virtio feature bit for SEQPACKET socket support introduced:
VIRTIO_VSOCK_F_SEQPACKET.
- 'msg_cnt' field in 'struct virtio_vsock_seq_hdr' renamed to
'msg_id' and used as id.
Per patch changelog:
- 'af_vsock: separate wait data loop':
1) Commit message updated.
2) 'prepare_to_wait()' moved inside while loop(thanks to
Jorgen Hansen).
Marked 'Reviewed-by' with 1), but as 2) I removed R-b.
- 'af_vsock: separate receive data loop': commit message
updated.
Marked 'Reviewed-by' with that fix.
- 'af_vsock: implement SEQPACKET receive loop': style fixes.
- 'af_vsock: rest of SEQPACKET support':
1) 'module_put()' added when transport callback check failed.
2) Now only 'seqpacket_allow()' callback called to check
support of SEQPACKET by transport.
- 'af_vsock: update comments for stream sockets': commit message
updated.
Marked 'Reviewed-by' with that fix.
- 'virtio/vsock: set packet's type in send':
1) Commit message updated.
2) Parameter 'type' from 'virtio_transport_send_credit_update()'
also removed in this patch instead of in next.
- 'virtio/vsock: dequeue callback for SOCK_SEQPACKET': SEQPACKET
related state wrapped to special struct.
- 'virtio/vsock: update trace event for SEQPACKET': format strings
now not broken by new lines.
I left a bunch of comments in the patches, I hope they are easy to fix
:-)
Thank you, yes, there are still small fixes.
So one more question, this is final review for this version of patchset and can
prepare next version with fixes? All other patches will reviewed in next version?
Thank You
quoted
Thanks for the changelogs. About 'per patch changelog', it is very
useful!
Just a suggestion, I think is better to include them in each patch after
the '---' to simplify the review.
Ack
quoted
You can use git-notes(1) or you can simply edit the format-patch and add
the changelog after the 3 dashes, so that they are ignored when the
patch is applied.
Thanks,
Stefano
On Tue, Mar 16, 2021 at 06:37:31AM +0300, Arseny Krasnov wrote:
On 15.03.2021 18:22, Arseny Krasnov wrote:
quoted
On 15.03.2021 14:40, Stefano Garzarella wrote:
quoted
Hi Arseny,
On Sun, Mar 07, 2021 at 08:57:19PM +0300, Arseny Krasnov wrote:
quoted
This patchset implements support of SOCK_SEQPACKET for virtio
transport.
As SOCK_SEQPACKET guarantees to save record boundaries, so to
do it, two new packet operations were added: first for start of record
and second to mark end of record(SEQ_BEGIN and SEQ_END later). Also,
both operations carries metadata - to maintain boundaries and payload
integrity. Metadata is introduced by adding special header with two
fields - message id and message length:
struct virtio_vsock_seq_hdr {
__le32 msg_id;
__le32 msg_len;
} __attribute__((packed));
This header is transmitted as payload of SEQ_BEGIN and SEQ_END
packets(buffer of second virtio descriptor in chain) in the same way as
data transmitted in RW packets. Payload was chosen as buffer for this
header to avoid touching first virtio buffer which carries header of
packet, because someone could check that size of this buffer is equal
to size of packet header. To send record, packet with start marker is
sent first(it's header carries length of record and id),then all data
is sent as usual 'RW' packets and finally SEQ_END is sent(it carries
id of message, which is equal to id of SEQ_BEGIN), also after sending
SEQ_END id is incremented. On receiver's side,size of record is known
from packet with start record marker. To check that no packets were
dropped by transport, 'msg_id's of two sequential SEQ_BEGIN and SEQ_END
are checked to be equal and length of data between two markers is
compared to then length in SEQ_BEGIN header.
Now as packets of one socket are not reordered neither on
vsock nor on vhost transport layers, such markers allows to restore
original record on receiver's side. If user's buffer is smaller that
record length, when all out of size data is dropped.
Maximum length of datagram is not limited as in stream socket,
because same credit logic is used. Difference with stream socket is
that user is not woken up until whole record is received or error
occurred. Implementation also supports 'MSG_EOR' and 'MSG_TRUNC' flags.
Tests also implemented.
Thanks to stsp2@yandex.ru for encouragements and initial design
recommendations.
Arseny Krasnov (22):
af_vsock: update functions for connectible socket
af_vsock: separate wait data loop
af_vsock: separate receive data loop
af_vsock: implement SEQPACKET receive loop
af_vsock: separate wait space loop
af_vsock: implement send logic for SEQPACKET
af_vsock: rest of SEQPACKET support
af_vsock: update comments for stream sockets
virtio/vsock: set packet's type in virtio_transport_send_pkt_info()
virtio/vsock: simplify credit update function API
virtio/vsock: dequeue callback for SOCK_SEQPACKET
virtio/vsock: fetch length for SEQPACKET record
virtio/vsock: add SEQPACKET receive logic
virtio/vsock: rest of SOCK_SEQPACKET support
virtio/vsock: SEQPACKET feature bit
vhost/vsock: SEQPACKET feature bit support
virtio/vsock: SEQPACKET feature bit support
virtio/vsock: setup SEQPACKET ops for transport
vhost/vsock: setup SEQPACKET ops for transport
vsock/loopback: setup SEQPACKET ops for transport
vsock_test: add SOCK_SEQPACKET tests
virtio/vsock: update trace event for SEQPACKET
drivers/vhost/vsock.c | 22 +-
include/linux/virtio_vsock.h | 22 +
include/net/af_vsock.h | 10 +
.../events/vsock_virtio_transport_common.h | 48 +-
include/uapi/linux/virtio_vsock.h | 19 +
net/vmw_vsock/af_vsock.c | 589 +++++++++++------
net/vmw_vsock/virtio_transport.c | 18 +
net/vmw_vsock/virtio_transport_common.c | 364 ++++++++--
net/vmw_vsock/vsock_loopback.c | 13 +
tools/testing/vsock/util.c | 32 +-
tools/testing/vsock/util.h | 3 +
tools/testing/vsock/vsock_test.c | 126 ++++
12 files changed, 1013 insertions(+), 253 deletions(-)
v5 -> v6:
General changelog:
- virtio transport specific callbacks which send SEQ_BEGIN or
SEQ_END now hidden inside virtio transport. Only enqueue,
dequeue and record length callbacks are provided by transport.
- virtio feature bit for SEQPACKET socket support introduced:
VIRTIO_VSOCK_F_SEQPACKET.
- 'msg_cnt' field in 'struct virtio_vsock_seq_hdr' renamed to
'msg_id' and used as id.
Per patch changelog:
- 'af_vsock: separate wait data loop':
1) Commit message updated.
2) 'prepare_to_wait()' moved inside while loop(thanks to
Jorgen Hansen).
Marked 'Reviewed-by' with 1), but as 2) I removed R-b.
- 'af_vsock: separate receive data loop': commit message
updated.
Marked 'Reviewed-by' with that fix.
- 'af_vsock: implement SEQPACKET receive loop': style fixes.
- 'af_vsock: rest of SEQPACKET support':
1) 'module_put()' added when transport callback check failed.
2) Now only 'seqpacket_allow()' callback called to check
support of SEQPACKET by transport.
- 'af_vsock: update comments for stream sockets': commit message
updated.
Marked 'Reviewed-by' with that fix.
- 'virtio/vsock: set packet's type in send':
1) Commit message updated.
2) Parameter 'type' from 'virtio_transport_send_credit_update()'
also removed in this patch instead of in next.
- 'virtio/vsock: dequeue callback for SOCK_SEQPACKET': SEQPACKET
related state wrapped to special struct.
- 'virtio/vsock: update trace event for SEQPACKET': format strings
now not broken by new lines.
I left a bunch of comments in the patches, I hope they are easy to fix
:-)
Thank you, yes, there are still small fixes.
So one more question, this is final review for this version of patchset and can
prepare next version with fixes? All other patches will reviewed in next version?
For me yes, the other patches seem okay, but I would like to see them
later, with the right order and the last things fixed.
Maybe to merge all this we should wait for the agreement from the specs
patch.
Thanks,
Stefano