268

I am using a library, ya-csv, that expects either a file or a stream as input, but I have a string.

How do I convert that string into a stream in Node?

12 Answers 12

219

As @substack corrected me in #node, the new streams API in Node v10 makes this easier:

const Readable = require('stream').Readable;
const s = new Readable();
s._read = () => {}; // redundant? see update below
s.push('your text here');
s.push(null);

… after which you can freely pipe it or otherwise pass it to your intended consumer.

It's not as clean as the resumer one-liner, but it does avoid the extra dependency.

(Update: in v0.10.26 through v9.2.1 so far, a call to push directly from the REPL prompt will crash with a not implemented exception if you didn't set _read. It won't crash inside a function or a script. If inconsistency makes you nervous, include the noop.)

11
  • 6
    From the docs (link): "All Readable stream implementations must provide a _read method to fetch data from the underlying resource."
    – Felix Rabe
    Jun 11, 2014 at 21:38
  • 2
    @eye_mew you need to require('stream') first
    – Jim Jones
    Dec 5, 2015 at 20:15
  • 11
    Why do you push null into the stream's buffer?
    – dopatraman
    Feb 23, 2016 at 19:22
  • 7
    @dopatraman null tells the stream that it has finished reading all the data and to close the stream Jul 30, 2016 at 0:14
  • 3
    Looks like you shouldn’t do it this way. Quoting the docs: “The readable.push() method is intended be called only by Readable Implementers, and only from within the readable._read() method.” Apr 6, 2018 at 11:16
164

Do not use Jo Liss's resumer answer. It will work in most cases, but in my case it lost me a good 4 or 5 hours bug finding. There is no need for third party modules to do this.

NEW ANSWER:

var Readable = require('stream').Readable

var s = new Readable()
s.push('beep')    // the string you want
s.push(null)      // indicates end-of-file basically - the end of the stream

This should be a fully compliant Readable stream. See here for more info on how to use streams properly.

OLD ANSWER: Just use the native PassThrough stream:

var stream = require("stream")
var a = new stream.PassThrough()
a.write("your string")
a.end()

a.pipe(process.stdout) // piping will work as normal
/*stream.on('data', function(x) {
   // using the 'data' event works too
   console.log('data '+x)
})*/
/*setTimeout(function() {
   // you can even pipe after the scheduler has had time to do other things
   a.pipe(process.stdout) 
},100)*/

a.on('end', function() {
    console.log('ended') // the end event will be called properly
})

Note that the 'close' event is not emitted (which is not required by the stream interfaces).

2
  • 2
    @Finn You don't need the parens in javascript if there aren't any args
    – B T
    Apr 6, 2018 at 18:21
  • 2
    dont' use "var" in 2018! but const
    – stackdave
    Dec 12, 2018 at 9:27
147

From node 10.17, stream.Readable have a from method to easily create streams from any iterable (which includes array literals):

const { Readable } = require("stream")

const readable = Readable.from(["input string"])

readable.on("data", (chunk) => {
  console.log(chunk) // will be called once with `"input string"`
})

Note that at least between 10.17 and 12.3, a string is itself a iterable, so Readable.from("input string") will work, but emit one event per character. Readable.from(["input string"]) will emit one event per item in the array (in this case, one item).

Also note that in later nodes (probably 12.3, since the documentation says the function was changed then), it is no longer necessary to wrap the string in an array.

https://nodejs.org/api/stream.html#stream_stream_readable_from_iterable_options

2
  • 2
    According to stream.Readable.from, Calling Readable.from(string) or Readable.from(buffer) will not have the strings or buffers be iterated to match the other streams semantics for performance reasons.
    – abbr
    Mar 1, 2020 at 0:04
  • 1
    My bad. The function was added in 10.7, and behaved the way I originally described. Sometime since, strings no longer need to be wrapped in arrays (since 12.3, it no longer iterates each character individually).
    – Fizker
    May 17, 2020 at 21:32
35

Just create a new instance of the stream module and customize it according to your needs:

var Stream = require('stream');
var stream = new Stream();

stream.pipe = function(dest) {
  dest.write('your string');
  return dest;
};

stream.pipe(process.stdout); // in this case the terminal, change to ya-csv

or

var Stream = require('stream');
var stream = new Stream();

stream.on('data', function(data) {
  process.stdout.write(data); // change process.stdout to ya-csv
});

stream.emit('data', 'this is my string');
2
  • 13
    This code breaks stream conventions. pipe() is supposed to return the destination stream, at very least.
    – greim
    Jun 30, 2014 at 16:02
  • 2
    The end event isn't called if you use this code. This is not a good way to create a stream that can be used generally.
    – B T
    Sep 3, 2014 at 17:28
12

Edit: Garth's answer is probably better.

My old answer text is preserved below.


To convert a string to a stream, you can use a paused through stream:

through().pause().queue('your string').end()

Example:

var through = require('through')

// Create a paused stream and buffer some data into it:
var stream = through().pause().queue('your string').end()

// Pass stream around:
callback(null, stream)

// Now that a consumer has attached, remember to resume the stream:
stream.resume()
4
  • I couldn't get zeMirco's solution to work for my use case, but resumer worked quite well. Thanks!
    – mpen
    Sep 9, 2013 at 16:13
  • The @substack resumer suggestion worked very well for me. Thanks!
    – Garth Kidd
    Feb 28, 2014 at 3:28
  • 2
    Resumer is great, but the "auto-resumes the stream on nextTick" can yield surprises if you expect you can pass the stream to unknown consumers! I had some code that piped a content stream to a file if a db save of metadata succeeded. That was a lurking bug, it happened to succeed when the db write returned success immediately! I later refactored things to be inside an async block, and boom, the stream was never readable. Lesson: if you don't know who's going to consume your stream, stick to the through().pause().queue('string').end() technique. Apr 10, 2014 at 16:56
  • 2
    I spent about 5 hours debugging my code because I used the resumer part of this answer. It'd be great if you could like.. remove it
    – B T
    Jan 22, 2015 at 1:53
11

There's a module for that: https://www.npmjs.com/package/string-to-stream

var str = require('string-to-stream')
str('hi there').pipe(process.stdout) // => 'hi there' 
3
9

Another solution is passing the read function to the constructor of Readable (cf doc stream readeable options)

var s = new Readable({read(size) {
    this.push("your string here")
    this.push(null)
  }});

you can after use s.pipe for exemple

4
  • what is the purpose of the return at the end? Dec 25, 2017 at 21:46
  • "always return something (or nothing)" , this exemple from the documentation . Dec 26, 2017 at 8:15
  • In JS, if a function doesn't have a return it is an equivalent to your empty return. Could you please provide a link where you have found it? Dec 26, 2017 at 13:56
  • you should right . I said that more for best practice. I want to return nothing , it 's not a mistake . So i remove the line. Jan 4, 2018 at 10:59
6

in coffee-script:

class StringStream extends Readable
  constructor: (@str) ->
    super()

  _read: (size) ->
    @push @str
    @push null

use it:

new StringStream('text here').pipe(stream1).pipe(stream2)
5

I got tired of having to re-learn this every six months, so I just published an npm module to abstract away the implementation details:

https://www.npmjs.com/package/streamify-string

This is the core of the module:

const Readable = require('stream').Readable;
const util     = require('util');

function Streamify(str, options) {

  if (! (this instanceof Streamify)) {
    return new Streamify(str, options);
  }

  Readable.call(this, options);
  this.str = str;
}

util.inherits(Streamify, Readable);

Streamify.prototype._read = function (size) {

  var chunk = this.str.slice(0, size);

  if (chunk) {
    this.str = this.str.slice(size);
    this.push(chunk);
  }

  else {
    this.push(null);
  }

};

module.exports = Streamify;

str is the string that must be passed to the constructor upon invokation, and will be outputted by the stream as data. options are the typical options that may be passed to a stream, per the documentation.

According to Travis CI, it should be compatible with most versions of node.

1
  • 2
    When I posted this initially, I did not include the relevant code, which I was told is frowned upon. Dec 14, 2016 at 22:12
5

Heres a tidy solution in TypeScript:

import { Readable } from 'stream'

class ReadableString extends Readable {
    private sent = false

    constructor(
        private str: string
    ) {
        super();
    }

    _read() {
        if (!this.sent) {
            this.push(Buffer.from(this.str));
            this.sent = true
        }
        else {
            this.push(null)
        }
    }
}

const stringStream = new ReadableString('string to be streamed...')
1

In a NodeJS, you can create a readable stream in a few ways:

SOLUTION 1

You can do it with fs module. The function fs.createReadStream() allows you to open up a readable stream and all you have to do is pass the path of the file to start streaming in.

const fs = require('fs');

const readable_stream = fs.createReadStream('file_path');

SOLUTION 2

If you don't want to create file, you can create an in-memory stream and do something with it (for example, upload it somewhere). ​You can do this with stream module. You can import Readable from stream module and you can create a readable stream. When creating an object, you can also implement read() method which is used to read the data out of the internal buffer. If no data available to be read, null is returned. The optional size argument specifies a specific number of bytes to read. If the size argument is not specified, all of the data contained in the internal buffer will be returned.

const Readable = require('stream').Readable;

const readable_stream = new Readable({
  ​read(size) {
   ​// ...
​  }
});

SOLUTION 3

When you are fetching something over the network, that can be fetched like stream (for example you are fetching a PDF document from some API).

const axios = require('axios');

const readable_stream = await axios({
  method: 'get',
  url: "pdf_resource_url",
  responseType: 'stream'
}).data;

SOLUTION 4

Third party packages can support creating of streams as a feature. That is a way with aws-sdk package that is usually used for uploading files to S3.

const file = await s3.getObject(params).createReadStream();
2
  • 4
    These solutions explain various ways to create a stream, but none of them the question, which is asking how to convert a string into a stream.
    – cjol
    Jan 10, 2022 at 14:54
  • Maybe, but it still helped me to fix my own (similar) problem.
    – raarts
    Jan 23, 2022 at 10:16
0

JavaScript is duck-typed, so if you just copy a readable stream's API, it'll work just fine. In fact, you can probably not implement most of those methods or just leave them as stubs; all you'll need to implement is what the library uses. You can use Node's pre-built EventEmitter class to deal with events, too, so you don't have to implement addListener and such yourself.

Here's how you might implement it in CoffeeScript:

class StringStream extends require('events').EventEmitter
  constructor: (@string) -> super()

  readable: true
  writable: false

  setEncoding: -> throw 'not implemented'
  pause: ->    # nothing to do
  resume: ->   # nothing to do
  destroy: ->  # nothing to do
  pipe: -> throw 'not implemented'

  send: ->
    @emit 'data', @string
    @emit 'end'

Then you could use it like so:

stream = new StringStream someString
doSomethingWith stream
stream.send()
3
  • I get this: TypeError: string is not a function at String.CALL_NON_FUNCTION (native) when I use it like new StringStream(str).send()
    – pathikrit
    Oct 6, 2012 at 2:24
  • Just because JavaScript uses duck typing doesn't mean you should reinvent the wheel. Node already provides an implementation for streams. Just create a new instance of stream.Readable like @Garth Kidd suggested.
    – Sukima
    Oct 2, 2014 at 3:15
  • 5
    @Sukima: stream.Readable did not exist when I wrote this answer.
    – icktoofay
    Oct 2, 2014 at 4:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.