5

How independent is the handling of UDP send and receive on same socket in Linux kernel? The use case I have is a worker thread sending UDP test traffic on (up to) 1000 sockets, and receiving the UDP replies in another worker thread. The receiver will be an epoll loop that also receives hardware send and receive timestamps on the socket error queue.

To clarify, when doing a sendmsg() syscall, will this temporarily block (or generate EAGAIN/EWOULDBLOCK) on the receiver thread receiving on the same socket? (i.e. if the send and receive happen to overlap in time) All sockets are set to non-blocking mode.

Another question is granularity of locking in the kernel - if I send and receive with sendmmsg/recvmmsg, is a lock for that socket locked once per sendmmsg, or once per UDP datagram in the sendmmsg?

UPDATE: I took a look at the original patch for sendmmsg in Linux kernel, seems the main benefit is avoiding multiple transitions user-kernel space. If any locking is done, it is probably done inside the individual calls to __sys_sendmsg: https://lwn.net/Articles/441169/

4
  • I must say, I didn't expect such a fine question when reading the title. Feb 16, 2017 at 14:37
  • Considering how utterly trivial UDP is, I would be really surprised if a UDP send would interfere with a UDP receive. For TCP, the hidden ACK's couple sending and receiving, but UDP is little more than IP plus a port number.
    – MSalters
    Feb 16, 2017 at 15:01
  • @MSalters: I agree, sending and receiving are pretty independent. If I have time, I will delve into my updated kernel repo and see if I find out anything more. Feb 16, 2017 at 15:13
  • @MSalters A quick look in net/ipv4/udp.c shows that there is a fast, lockless path for the non-corking case (for udp send). I am not well versed enough in the kernel to say for sure if that means no locking at all for normal UDP packets. Feb 16, 2017 at 15:20

1 Answer 1

0

Each system call is thread independent. So, as far as you don't involve per process kernel data, both will run independently without disturbing each other.

Another different thing is what the kernel does with system calls related to the same inode (in this case, the virtual node assigned to the socket you use to communicate) To serialize and make atomic calls to the filesystem, the kernel normally does an inode lock during the whole system call (being this a read, write or ioctl system call) that stands for the whole system call (even if you do a unique write call to write a zillion bytes, the inode is blocked during the execution of the whole system call)

In the tcp-ip stack, this is made at the socket level, and is controlled in your case by the specific AF_INET socket class software. As udp is concerned, sending a packet or receiving doesn't affect shared resources that need to be locked, but you'll have to look at your udp implementation (or socket level) to see if some locking is done and what the granularity is. Normally, the lock, should it be, should be used only during the load/unload of the udp buffers (normally there aren't buffers in udp, as the socket and network card driver are enough to supply enough buffer resources.

9
  • 1
    An inode is a structure in a Unix file system. A socket is not an inode. The term you are looki for is 'file descriptor'.
    – user207421
    Feb 17, 2017 at 7:37
  • @Luis Colorado what do you mean 'normally there aren't buffers in UDP', you have the per-socket bufffer in the kernel (necessary to adjust e.g. for high-rate sockets (setsockopt SO_SNDBUF)), and you have the whole queuing system that comes before the NIC ring buffers. Feb 17, 2017 at 8:13
  • @ErikAlapää, yes... but no buffering is done in the UDP module... buffering is done in the socket module for incoming packets and in the driver buffer for outgoing packets. The udp module has no buffering resources, as packets flow directly from user space to the driver. UDP only selects wich driver will store the packet until the driver is ready to transmit the packet. UDP passes packets directly to IP (and IP to the driver) module with no buffering. Feb 18, 2017 at 9:02
  • @EJP, some unix implementations call it vnode, others call it incore i-node, others do it differently. But all of them do the same thing, to block the *-node during the system call execution to provide atomicity of system calls. An AF_UNIX socket is a socket implemented using filesystem i-nodes, the same as normal files, and named fifos where implemented in some implementations as a special kind of socket (in that case they even allocate disk blocks to implement buffering between readers and writers) Feb 18, 2017 at 9:11
  • @EJP, on the other side a file table entry (a file descriptor is only the index in the per-process file table) is a diferent structure allocated only to provide a file pointer (the point where the next write will be done) to solve the problem of processes sharing or not file pointers. File table entry only stores there the file pointer and some of the fcntl(2) flags. It is right that today's implementations use a diferent structure pointed to from the file entry for sockets than for filesystem inodes, but for locking purposes, both are the same thing. Feb 18, 2017 at 9:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.