75

What's the best (most efficient) way to parse a tab-delimited file in Ruby?

4 Answers 4

118

The Ruby CSV library lets you specify the field delimiter. Ruby 1.9 uses FasterCSV. Something like this would work:

require "csv"
parsed_file = CSV.read("path-to-file.csv", col_sep: "\t")
5
  • 9
    Beware that this approach will fail if any of the tab-separated values contains a double-quote. The StrictTsv suggestion in the other answer is more robust. Nov 1, 2016 at 1:49
  • If you have a file with incorrect use of double-quotes in it, you can substitute " for some other character that you don't expect to ever be in the file. parsed_file = CSV.read("path-to-file.csv", { col_sep: "\t", quote_char: '}') Source docs.
    – Aaron Gray
    Oct 10, 2018 at 14:18
  • 2
    Another approach is to use the liberal_parsing option. parsed_file = CSV.read("path-to-file.csv", { col_sep: "\t", liberal_parsing: true) Source docs.
    – Aaron Gray
    Nov 7, 2018 at 5:21
  • If you want to read double quotes exactly as they appear in the file, simply set quote_char to nil: parsed_file = CSV.read("path-to-file.csv", col_sep: "\t", quote_char: nil). This is more robust and elegant than using a character you think won't appear in the file.
    – kxmh42
    Jul 18, 2019 at 7:59
  • Latest version of CSV library uses doesn't want you to use the surrounding curly braces and will complain about it if you do with "warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call"
    – cdmo
    Nov 24, 2020 at 16:10
34

The rules for TSV are actually a bit different from CSV. The main difference is that CSV has provisions for sticking a comma inside a field and then using quotation characters and escaping quotes inside a field. I wrote a quick example to show how the simple response fails:

require 'csv'
line = 'boogie\ttime\tis "now"'
begin
  line = CSV.parse_line(line, col_sep: "\t")
  puts "parsed correctly"
rescue CSV::MalformedCSVError
  puts "failed to parse line"
end

begin
  line = CSV.parse_line(line, col_sep: "\t", quote_char: "Ƃ")
  puts "parsed correctly with random quote char"
rescue CSV::MalformedCSVError
  puts "failed to parse line with random quote char"
end

#Output:
# failed to parse line
# parsed correctly with random quote char

If you want to use the CSV library you could used a random quote character that you don't expect to see if your file (the example shows this), but you could also use a simpler methodology like the StrictTsv class shown below to get the same effect without having to worry about field quotations.

# The main parse method is mostly borrowed from a tweet by @JEG2
class StrictTsv
  attr_reader :filepath
  def initialize(filepath)
    @filepath = filepath
  end

  def parse
    open(filepath) do |f|
      headers = f.gets.strip.split("\t")
      f.each do |line|
        fields = Hash[headers.zip(line.split("\t"))]
        yield fields
      end
    end
  end
end

# Example Usage
tsv = Vendor::StrictTsv.new("your_file.tsv")
tsv.parse do |row|
  puts row['named field']
end

The choice of using the CSV library or something more strict just depends on who is sending you the file and whether they are expecting to adhere to the strict TSV standard.

Details about the TSV standard can be found at http://en.wikipedia.org/wiki/Tab-separated_values

5
  • Please include code snippets with the answer, not in an external gist. That gist now appears to be down, which is a real shame. Feb 1, 2015 at 19:26
  • 4
    @JezenThomas thanks for the heads up. I pulled all the code samples inline to fix the issue of having to go look at the gist
    – mmmries
    Feb 2, 2015 at 5:37
  • great answer. 👍 . I'm surprised how horribly \d with the CSV parser fails.
    – a2f0
    Nov 27, 2017 at 2:35
  • 1
    This is minor but lead me slightly astray. line = 'boogie\ttime\tis "now"' results in a string with double escaped tab characters so I thought the failure could be due to that when I had actually just written my test incorrectly. To get the test string intended use line = "boogie\ttime\tis \"now\"" or "boogie\ttime\tis " + '"now"' You can test it with puts. The first results in boogie\ttime\tis "now" while the latter two result in boogie time is "now" (tabs don't show up well here but will in your console). Thanks for the comprehensive answer 👍
    – Aaron
    Mar 12, 2018 at 11:10
  • I just discovered that at least in Ruby 2.5.0 this is specifically related to TSV but the entire CSV library and spec due to the placement of the quotes. Both of the following will fail CSV.parse("foo,bar,and \"baz\" quotes") and CSV.parse("foo\tbar\tand \"baz\" quotes", col_sep: "\t"). It appears the quotes are only valid if they surround the entire column's contents so that you can include the column separator character. The follow two will parse fine CSV.parse("foo\tbar\t\"and baz\tquotes\"", col_sep: "\t") and CSV.parse("foo,bar,\"and baz,quotes\"")
    – Aaron
    Mar 12, 2018 at 11:19
6

There are actually two different kinds of TSV files.

  1. TSV files that are actually CSV files with a delimiter set to Tab. This is something you'll get when you e.g. save an Excel spreadsheet as "UTF-16 Unicode Text". Such files use CSV quoting rules, which means that fields may contain tabs and newlines, as long as they are quoted, and literal double quotes are written twice. The easiest way to parse everything correctly is to use the csv gem:

    use 'csv'
    parsed = CSV.read("file.tsv", col_sep: "\t")
    
  2. TSV files conforming to the IANA standard. Tabs and newlines are not allowed as field values, and there is no quoting whatsoever. This is something you will get when you e.g. select a whole Excel spreadsheet and paste it into a text file (beware: it will get messed up if some cells do contain tabs or newlines). Such TSV files can be easily parsed line-by-line with a simple line.rstrip.split("\t", -1) (note -1, which prevents split from removing empty trailing fields). If you want to use the csv gem, simply set quote_char to nil:

    use 'csv'
    parsed = CSV.read("file.tsv", col_sep: "\t", quote_char: nil)
    
2
  • In practice I finds that quote_char: nil gets me a undefined method encode' for nil:NilClass (NoMethodError)` in Ruby 2.7.0. Another SO thread suggests using "\0", or liberal_parsing: true, which works better for me, but both might still fail with IANA TSV containing escaped characters: stackoverflow.com/a/41644206/2960236
    – wu-lee
    Jan 5, 2022 at 20:05
  • I was looking for the split("\t", -1)
    – Fravadona
    Feb 12, 2022 at 11:14
0

I like mmmries answer. HOWEVER, I hate the way that ruby strips off any empty values off of the end of a split. It isn't stripping off the newline at the end of the lines, either.

Also, I had a file with potential newlines within a field. So, I rewrote his 'parse' as follows:

def parse
  open(filepath) do |f|
    headers = f.gets.strip.split("\t")
    f.each do |line|
      myline=line
      while myline.scan(/\t/).count != headers.count-1
        myline+=f.gets
      end
      fields = Hash[headers.zip(myline.chomp.split("\t",headers.count))]
      yield fields
    end
  end
end

This concatenates any lines as necessary to get a full line of data, and always returns the full set of data (without potential nil entries at the end).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.