4

I'm failing to read more than 65536 bytes into a buffer from a file using boost::asio::windows::stream_handle asynchronously.

Starting from 65537th byte the buffer contains the the data from the very beginning of the file, rather than the expected data.

Here is a code example, which reproduces the issue:

auto handle = ::CreateFile(L"BigFile.xml", GENERIC_READ, FILE_SHARE_READ, nullptr, OPEN_EXISTING, FILE_FLAG_OVERLAPPED, nullptr);
boost::asio::io_service ios;

boost::asio::windows::stream_handle streamHandle(ios, handle);

const auto to_read_bytes = 100000;
char buffer[to_read_bytes];

boost::asio::async_read(streamHandle, boost::asio::buffer(buffer, to_read_bytes), [](auto &ec, auto read) {
    std::cout << "Bytes read: " << read << std::endl;
});

ios.run();

auto bufferBegin = std::string(buffer, 38);
auto bufferCorrupted = std::string(buffer + 65536, 38);   // <- it contains bytes from the beginning of the file

std::cout << "offset 0: " << bufferBegin << std::endl;
std::cout << "offset 65536: " << bufferCorrupted << std::endl;   

::CloseHandle(handle);

That code produces an output:

> Bytes read: 100000  
> offset 0: <?xml version="1.0" encoding="UTF-8"?>  
> offset 65536: <?xml version="1.0" encoding="UTF-8"?>

The source file is bigger than 65536.

This is reproducible with boost 1.61 + VS2015. Also that issue was in boost 1.55 + VS2010.
Operating systems are: Windows 7 and Windows Server 2008R2.

My questions are:
1. Is that the known limitation in boost::asio or in WinAPI?
2. If it is the known limitation, what would be the safe size of the buffer to read data? Is it safe to have a buffer of size 65536, or it should be smaller?

7
  • This is not a limit in the windows API -- the limit that is present there (in older API methods) is 2^32 not 2^16. Aug 8, 2016 at 16:20
  • You may want to specify what versions of software you are using (boost, windows, and visual C++)
    – nate
    Aug 8, 2016 at 16:46
  • @nate Yeah, definitely I need to specify the versions. Thanks for reminding. I updated the question accordingly. Aug 8, 2016 at 17:40
  • The problem might be with stack limit which is usually 100,000. Use std::string which allocates memory on heap, instead of using char buffer[100000]; which allocates on stack. Or just use to_read_bytes = 4096 it doesn't get faster if you increase that buffer size. Aug 8, 2016 at 18:52
  • @ BarmakShemirani Nope, it is not stack limit. The same behavior is when the buffer is allocated in the heap. Aug 8, 2016 at 19:25

2 Answers 2

6

As Tanner Sansbury says, you opened a file with FILE_FLAG_OVERLAPPED but you're trying to use it as a stream. It is not.

async_read() is basically this loop in asio/impl/read.hpp:

for (;;)
{
    stream_.async_read_some(buffers_, ASIO_MOVE_CAST(read_op)(*this));

    buffers_.consume(bytes_transferred);
    total_transferred_ += bytes_transferred;

    if (!ec && bytes_transferred == 0)
        break;
}

The actual maximum number of bytes that will be read in one call comes from completion_condition.hpp:

enum default_max_transfer_size_t { default_max_transfer_size = 65536 };

The problem is the async_read_some() call above. You'll notice that there's no offset to tell it where to start reading. Because you are using asynchronous reads (also called "overlapped" on Windows), an offset has to be specified for every read.

This is where it ends up, in asio/detail/impl/win_iocp_handle_service.ipp:

DWORD bytes_transferred = 0;
op->Offset = offset & 0xFFFFFFFF;
op->OffsetHigh = (offset >> 32) & 0xFFFFFFFF;
BOOL ok = ::ReadFile(impl.handle_, buffer.data(),
    static_cast<DWORD>(buffer.size()),
    &bytes_transferred, op);

op->Offset and op->OffsetHigh are always 0. The pointer inside your buffer will advance correctly, but every chunk will be read from the start of the file.

There's an async_read_some_at() that's available, which you should use instead, as well as windows::random_access_handle. This will set the Offset and OffsetHigh members correctly. You will have to keep track of the number of bytes read yourself.

The documentation for the OVERLAPPED structure says this:

The Offset and OffsetHigh members together represent a 64-bit file position. It is a byte offset from the start of the file or file-like device, and it is specified by the user; the system will not modify these values. The calling process must set this member before passing the OVERLAPPED structure to functions that use an offset, such as the ReadFile or WriteFile (and related) functions.

There's also this part in Synchronous and Asynchronous I/O:

The system does not maintain the file pointer on asynchronous handles to files and devices that support file pointers (that is, seeking devices), therefore the file position must be passed to the read and write functions in the related offset data members of the OVERLAPPED structure. For more information, see WriteFile and ReadFile.

4
  • 3
    Your answer definitely deserves more that one upvote. I submitted the bug report here: svn.boost.org/trac/boost/ticket/12383 . And I'll add a link to your answer to the bug report later. Thanks for help :) Aug 9, 2016 at 16:35
  • 1
    @AlexanderStepaniuk The main repo is on github, it's just boostified when released. You may have a quicker answer if you file the bug report on the github tracker. Not sure though.
    – isanae
    Aug 9, 2016 at 16:40
  • @AlexanderStepaniuk Well, I got the how, but he got the why: your file isn't a stream. You should accept his answer, I'll modify my post so it doesn't suggest it's a bug anymore.
    – isanae
    Aug 10, 2016 at 0:26
  • ok, I'll accept another answer, as he provides the working solution to fix. But your answer is also very valuable as it explains the root cause in details. Aug 10, 2016 at 10:37
4

This is neither a limitation of Asio, Windows, nor buffer sizes. Rather, Asio is performing exactly what it has been told to do within its specifications: it is reading 100000 bytes from a regular file as-if it was a stream. With windows::stream_handle:

  • async_read() will be composed of zero or more intermediate async_read_some() operations until either the number of bytes requested by the application has been transferred, or until an error occurs

    This operation is implemented in terms of zero or more calls to the stream's async_read_some function, and is known as a composed operation.

  • async_read_some() operations may read less than the number of requested bytes

    The read operation may not read all of the requested number of bytes.

  • each intermediate async_read_some() operation will read from the start of the stream

As the file handle being used is not truly a stream, but rather a regular file, consider using the windows::random_access_handle and async_read_at(device, 0, ...). The Random-Access HANDLEs documentation notes:

Boost.Asio provides Windows-specific classes that permit asynchronous read and write operations to be performed on HANDLEs that refer to regular files.

When using windows::random_access_handle and async_read_at():

  • async_read_at() will be composed of zero or more intermediate async_read_some_at() operations until either the number of bytes requested by the application has been transferred, or until an error occurs
  • async_read_some_at() operations may read less than the number of requested bytes
  • each intermediate async_read_some_at() operation will use an offset corresponding to end of the previous read when reading from the device (e.g. the initial offset + current bytes transferred)
4
  • Thanks, for answer. Now it's clear how to read big file asynchronously. But I still think it's a bug in Asio. At least a bug in the documentation, where this behavior should be mentioned. As a simple operation gives wrong result without any compile-time or run-time notification. Aug 10, 2016 at 10:41
  • @Alexander I believe the behavior is documented. I doubt any compile-time errors are possible. However, by querying for file information by HANDLE, it may be possible for stream_handle::assign() to fail with a not-supported error, assuming there is no compelling use case to support stream_handle for regular files. Aug 10, 2016 at 13:53
  • I cannot find any warnings in the documentation about the limitation that async reading more than 2^16 bytes won't work. boost.org/doc/libs/1_61_0/boost/asio/windows/… - here documented method async_read_some() , which says nothing about it. And here mentioned usage async_read() when we need to get guaranteed amount of data read. Aug 10, 2016 at 16:23
  • @Alexander There is no limitation. The async_read() composed operations can asynchronously read more than 65536 bytes. The example in the question even indicates that it successfully read 10000: the first async_read_some() reads 65536 bytes (bytes 0 to 65536 in the file), and the second async_read_some() reads 34464 bytes (bytes 0 to 34464 in the file). I understand that this is not the desired behavior for your use case, but it is for a stream. The documentation explicitly states windows::random_access_handle should be used for regular files. Aug 10, 2016 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.