4

I've written a networking server that uses OpenSSL for SSL/TLS. The server sends and receives large blocks of data and performs various transformations in between. For performance reasons, transformations are done mainly using vector information (see iovec from POSIX) that avoids expensive memory moves (memcpy() etc.). When data are ready to be sent, I use writev() POSIX function that gathers data from the memory using these vectors and it sends that usually as one network packet.

Now with OpenSSL, it is not entirely possible because OpenSSL offers only SSL_write() function as far as I know. That means I have to call this function for every vector entry I want to send. It, unfortunately, causes that every vectored chunk of data is transmitted in its own SSL frame, and that introduces unwanted and unnecessary network overhead.

My question is: Is there SSL_writev() equivalent of writev()? Or in general, is there a technique how I can tell to OpenSSL to stash SSL_write() data into a one SSL application record (type 22) without sending it (and then of course some kind of flush() function)?

Edit: As discussed below, a viable approach is to consolidate vectored data into a big chunk prior a final single SSL_write() call. There is however connected overhead with 2 copies (1st during consolidation, 2nd when SSL_write() performs AES encryption). Theoretical SSL_writev() call doesn't introduce this overhead.

11
  • I think you will need a "pull up" function. I.e., one that combines multiple buffers into one.
    – jww
    Jul 5, 2016 at 14:22
  • That's exactly what I do today. But it is quite expensive b/c it moves a large chunks of data in the memory.
    – Ales Teska
    Jul 5, 2016 at 17:57
  • 1
    @ateska But it is quite expensive b/c it moves a large chunks of data in the memory. On Linux, writev() is actually implemented as just a wrapper around write() that allocates a temporary buffer, copies the writev() buffers into the temp buffer, then calls write(). If you're running on Linux and writev() is working for you without SSL, just write your own SSL_writev() wrapper. Jul 5, 2016 at 18:25
  • 2
    @AndrewHenle - good point with writev() implementation. I was hoping that it uses scatter/gather kernel feature.
    – Ales Teska
    Jul 5, 2016 at 18:59
  • 2
    @jww - agree, an efficient copying is important. Yet, it still means that SSL version will do 2 copies: 1st is "pull up", 2nd is AES (or similar) encryption during SSL write. Both can be indeed 'accelerated' by SSE4/AVX and AES-NI respectively. I'm looking for consolidating that into a 1 copy, that is an original idea behind SSL_writev().
    – Ales Teska
    Jul 5, 2016 at 19:04

1 Answer 1

-1

You can use a BIO_f_buffer() to achieve this. Wrap your network layer BIO in a BIO_f_buffer() filter BIO and set that as the write BIO for your SSL object. This will cause all data written out to stay in the buffer until you issue a flush on it.

7
  • BTW, you may want to wait to do this until after the handshake has completed - otherwise you will have to manually issue "flush" commands for each flight of handshake messages that are exchanged. Jul 5, 2016 at 19:22
  • Agree, but it will still create an SSL application record (type 23) for each call of SSL_write. That means an unwanted network overhead that I want to avoid.
    – Ales Teska
    Jul 5, 2016 at 21:15
  • Yes it will - although from your description it sounded like your main concern was to avoid multiple network packets. That is different to a TLS record. Multiple records can be contained within a single TCP packet, or split up across many. By buffering and flushing in the way I propose that gives the network layer the best opportunity to transmit the data across the network in the most efficient way possible. The overhead is then limited to the additional record header bytes (5 bytes) plus the MAC size (depends on the ciphersuite). If your aim is to reduce the number of records then... Jul 5, 2016 at 21:58
  • ...you can do the same thing in reverse, i.e. put a BIO_f_buffer() in front of an SSL BIO and flush it through to the SSL layer when you are ready. Jul 5, 2016 at 21:59
  • 1
    The general problem I have with this answer is that there is an unnecessary sacrifice in either the network overhead or a mem-copy operation. My goal is to avoid both (because I have a working solution on this level already). From an architectural point of view, it is quite straightforward: ...
    – Ales Teska
    Aug 13, 2016 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.