11

I consider replacing Stream-based IO in our application with System.IO.Pipelines to avoid unnecessary memory allocation (considered first RecyclableMemoryStream but it seems to be discontinued). But in some places I still have to use Stream because of the interface imposed by an external library. So my PipeWriter will need wrap its data in a Stream.

I didn't find much on this topic, but found a suggestion to use decorator pattern (C# Stream Pipe (Stream Spy)) in an answer to a different question. I am not sure it would be a right to hide Pipelines behind a Stream wrapper but can't find anything else that will let me piped data to a stream. Am I missing something?

UPDATE. Here's an example using SSH.NET open source library to upload a file to an FTP server (https://gist.github.com/DavidDeSloovere/96f3a827b54f20d52bcfda4fe7a16a0b):

using (var fileStream = new FileStream(uploadfile, FileMode.Open))
    {
        Console.WriteLine("Uploading {0} ({1:N0} bytes)", uploadfile, fileStream.Length);
        client.BufferSize = 4 * 1024; // bypass Payload error large files
        client.UploadFile(fileStream, Path.GetFileName(uploadfile));
    }

Note that we open a FileStream to read a file and then pass a Stream reference to an SftpClient. Can I use System.IO.Pipelines here to reduce memory allocation? I will still need to provide a Stream for SftpClient.

2
  • Can you please explain a little more what do you wish to achieve.
    – Gilad
    Dec 16, 2018 at 13:38
  • @Gilad I updated my question with the example. Dec 16, 2018 at 14:13

2 Answers 2

12

Disclaimer: I'm no expert, just putting together the pieces...


The answer (as of Janurary 2019) seems to be: there is no official support for this.

System.IO.Pipelines was created primarily for networking use cases. In fact, the pipelines code released in 2.1 had no support for any endpoints:

Here we need a bit of caveat and disclaimer: the pipelines released in .NET Core 2.1 do not include any endpoint implementations.

There is a proposed design for an API for generic stream adapter but that is part of the .NET Core 3.0 Milestone.

There even seems to be some reticence to implementing file-based pipelines access (AKA a FileStream pipelines equivalent). This is particularly disappointing since I too was hoping for pipelines powered file I/O.

I think your best bet at the moment is using the UsePipe() methods in https://github.com/AArnott/Nerdbank.Streams


Update: Here's another example I just found https://github.com/tulis/system-io-pipelines-demo/tree/master/src/SystemIoPipelinesDemo/SystemIoPipelinesDemo


Update: I had a go at making a Pipeline based file reader. You can read all about it here: https://github.com/atruskie/Pipelines.File.Unofficial

Essentially, from a performance perspective, using a pipeline stream adapter like Nerdbank.Streams is a good way to go!

4
  • Thank very much for the clarification. It's disappointing that there is no support for file streams but at least I understand now why I wasn't able to find any examples of how to use pipelines with them. Jan 16, 2019 at 18:15
  • @Anthony Truskinger I wondered why synchronous I/O is significantly faster than async I/O in your benchmarks, and I think I spotted the reason for it: your file streams are not opened with FileOptions.Asynchronous. See github.com/atruskie/Pipelines.File.Unofficial/issues/1
    – feO2x
    Mar 24, 2019 at 9:56
  • I've updated the benchmarks to test with and without FileOptions.Asynchronous Mar 26, 2019 at 5:02
  • 1
    I haven't worked on any of this in a while, but from what I read from the GitHub link above (github.com/dotnet/corefx/issues/27246) it appears some adapters should be present in .NET Core 3.0 Jan 29, 2020 at 4:33
2

It seems to be a way now to leverage AsStream() extension off of PipeReader:

using (var fileStream = new FileStream(uploadfile, FileMode.Open))
    {
        Console.WriteLine("Uploading {0} ({1:N0} bytes)", uploadfile, fileStream.Length);
        client.BufferSize = 4 * 1024; // bypass Payload error large files
        var pipeReader = PipeReader.Create(fileStream);
        client.UploadFile(pipeReader.AsStream(), Path.GetFileName(uploadfile));
    }

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.