12

By default Ruby opens $stdin and $stdout in buffered mode. This means you can't use Ruby to perform a grep-like operation filtering text. Is there any way to force Ruby to use line-oriented mode? I've seen various solutions including popen3 (which does buffered-mode only) and pty (which doesn't separately handle $stdout and $stderr, which I require).

How do I do this? Python seems to have the same lack.

4
  • Dear kind reply-ers: thanks for your help. Please verify your solution works - and a hint that all the obvious answers probably don't, due to unexpected and aggressive buffering.
    – Peter
    Aug 10, 2011 at 0:19
  • How do you verify that the I/O is not buffered ? Is it like a perceptible slowness in I/O. Sorry if it sounds like a stupid question.
    – Ocaj Nires
    Aug 10, 2011 at 19:34
  • Buffering isn't slowness but failure to process partial data at all. Best way is to write a simple grep-like function and test it interactively.
    – Peter
    Aug 10, 2011 at 19:38
  • I tried just that and compared it against unix grep. I also read up about line oriented vs. buffered IO. Take a look at this pastie which include ruby grep and benchmark against unix grep. From the benchmark in the pastie, except for ruby grep being extremely slow, I still don't understand the failure to process partial data part in this test. How do you test that interactively ?
    – Ocaj Nires
    Aug 10, 2011 at 20:49

4 Answers 4

4
+100

It looks like your best bet is to use STDOUT.syswrite and STDOUT.sysread - the following seemed to have reasonably good performance, despite being ugly code:

STDIN.sync = true
STDOUT.syswrite "Looking for #{ARGV[0]}\n"

def next_line
  mybuff = @overflow || ""
  until mybuff[/\n/]
    mybuff += STDIN.sysread(8)
  end
  overflow = mybuff.split("\n")
  out, *others = overflow
  @overflow = others.join("\n")
  out
rescue EOFError => e
  false  # NB: There's a bug here, see below
end

line = next_line
while line
  STDOUT.syswrite "#{line}\n" if line =~ /#{ARGV[0]}/i
  line = next_line
end

Note: Not sure you need #sync with #sysread, but if so you should probably sync STDOUT too. Also, it reads 8 bytes at a time into mybuff - you should experiment with this value, it's highly inefficient / CPU heavy. Lastly, this code is hacky and needs a refactor, but it works - tested it using ls -l ~/* | ruby rgrep.rb doc (where 'doc' is the search term)


Second note: Apparently, I was so busy trying to get it to perform well, I failed to get it to perform correctly! As Dmitry Shevkoplyas has noted, if there is text in @overflow when EOFError is raised, that text will be lost. I believe if you replace the catch with the following, it should fix the problem:

rescue EOFError => e
  return false unless @overflow && @overflow.length > 0
  output = @overflow
  @overflow = ""
  output
end

(if you found that helpful, please upvote Dmitry's answer!)

4
  • 1
    this seems to be working well, using the full sysread / syswrite thing. thanks!
    – Peter
    Aug 11, 2011 at 20:21
  • 1
    The accepted answer by user208769 is good, but has one flaw: under certain conditions you will loose last line. I'll show how to reproduce it and how to fix it in my answer: link Dec 6, 2016 at 15:27
  • @DmitryShevkoplyas: Thanks for the suggested edit - unfortunately it got rejected (not sure why, most of the reasons given were nonsense.) As a rule, I try to avoid empty () at the end of methods, and like having the code self document - if I think the variable might be a non-object, I'd rather test for that explicitly. But thank you for the further edit, and sorry it got rejected out of hand by over-zealous mods!
    – user208769
    Dec 13, 2016 at 11:23
  • 1
    no problem. I only added empty () because my edit was rejected with some nonsense like "edit must be 6 byte long" (I wonder how do we edit simple grammar when only 1 byte should be changed).. anyways - your final edit looks good! Thx for your solution! Dec 13, 2016 at 16:01
4

You can always turn on autoflush on any stream you want:

STDOUT.sync = true

This will have the effect of committing any writes immediately.

Most languages have this feature, but they always call it something a little different.

3
  • 2
    Just because ruby isn't buffering doesn't mean that the OS below isn't buffering. I suspect that this may be a two-part solution, what the second part is though, I'm looking forward to seeing.
    – Paul Rubel
    Aug 3, 2011 at 16:32
  • 1
    The OS only buffers because the application receiving the data may not be ready to read it the instant you write it. This is unavoidable, and is simply how pipes work. Ruby, like most languages, including C, buffers file output for performance reasons, but this can be either bypassed by using syswrite or tweaked using sync.
    – tadman
    Aug 3, 2011 at 18:36
  • 1
    This is way more elegant than STDOUT.syswrite, thank you! Probably better to use $stdout.sync = true though; see stackoverflow.com/questions/6671716/ruby-stdout-vs-stdout for why. Jul 25, 2012 at 10:36
2

You can call $stdout.flush after you've printed your line, and call $stdin.readline to fetch one line.

2

The accepted answer by user208769 is good, but has one flaw: under certain conditions you will loose last line. I'll show how to reproduce it and how to fix it below:

To reproduce the "last line lost" bug:

mkdir deleteme
touch deleteme/1 deleteme/2 deleteme/3
ls deleteme/ | ./rgrep.rb ''
Looking for
1
2

as you can see the "3" file is missing from the rgrep output. Surprisingly for the different filename length it would work differently though! Look:

rm -fr deleteme/
mkdir deleteme
touch deleteme/11 deleteme/22 deleteme/33
ls deleteme/ | ./rgrep.rb ''
Looking for
11
22
33

Now the third file is present! What a bug! Isn't she a beauty!!?
One can only imagine how much damage this random behaviour can cause.

To fix the bug we'd modify rescue portion slightly:

#!/usr/bin/env ruby
STDIN.sync = true
STDOUT.syswrite "Looking for #{ARGV[0]}\n"

def next_line
  mybuff = @overflow || ""
  until mybuff[/\n/]
    mybuff += STDIN.sysread(8)
  end
  overflow = mybuff.split("\n")
  out, *others = overflow
  @overflow = others.join("\n")
  out
rescue EOFError => e
  if @overflow.to_s.size > 0
    leftover_line = @overflow
    @overflow = ''
    return leftover_line
  else
    false
  end
end

line = next_line
while line
  STDOUT.syswrite "#{line}\n" if line =~ /#{ARGV[0]}/i
  line = next_line
end

I'll leave the "why" portion out of this post as an exercise for curious ones as otherwise it wont be digested properly (and this post is already way too long for my 1st post ever;) heh..

1
  • Good catch! And a great first post ever :D
    – user208769
    Dec 7, 2016 at 22:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.