39

I have a script that is appending new fields to an existing CSV, however ^M characters are appearing at the end of the old lines so the new fields end up on a new row instead of the same one. How do I remove ^M characters from a CSV file using Perl?

1

11 Answers 11

52

^M is carriage return. You can do this:

$str =~ s/\r//g
28

Or a 1-liner:

perl -p -i -e 's/\r\n$/\n/g' file1.txt file2.txt ... filen.txt
3
  • 3
    It's so easy to remember this one as Perl Pie.
    – dreamlax
    Mar 16, 2009 at 19:02
  • On windows passing *.txt with this command does not work. It gives: Can't open *.txt: Invalid argument. Anyone?
    – mgouin
    Dec 10, 2015 at 19:26
  • No need for global 'g' as '$' matches only end of line. Mar 20, 2017 at 12:07
15

You found out you can also do this:

$line=~ tr/\015//d;
1
  • 1
    not as readable as \r - anyone looking at that (or yourself in a year's time) would be glad of a comment stating what it does
    – plusplus
    Jun 21, 2011 at 8:12
8

Slightly unrelated, but to remove ^M from the command line using Perl, do this:

perl -p -i -e "s/\r\n/\n/g" file.name
6

I prefer a more general solution that will work with either DOS or Unix input. Assuming the input is from STDIN:

while (defined(my $ln = <>))
  {
    chomp($ln);
    chop($ln) if ($ln =~ m/\r$/);

    # filter and write
  }
3

This one liner replaces all the ^M characters:

dos2unix <file-name>

You can call this from inside Perl or directly on your Unix prompt.

2

To convert DOS style to UNIX style line endings:

for ($line in <FILEHANDLE>) {
   $line =~ s/\r\n$/\n/;
}

Or, to remove UNIX and/or DOS style line endings:

for ($line in <FILEHANDLE>) {
   $line =~ s/\r?\n$//;
}
2
  • wouldn't that remove the newlines, too? Mar 16, 2009 at 14:54
  • I guess that depends on your goal. I edited to show both strategies.
    – spoulson
    Mar 16, 2009 at 15:23
1

This is what solved my problem. ^M is a carriage return, and it can be easily avoided in a Perl script.

while(<INPUTFILE>)
{
     chomp;
     chop($_) if ($_ =~ m/\r$/);
}
1
  • Does that remove ^M from a CSV file? Changing the input file? Does it create some output file that will not have them? Dec 7, 2017 at 0:23
0

Little script I have for that. A modification of it helped to filter out some other non-printable characters in cross-platform legacy files.

#!/usr/bin/perl
# run this as
# convert_dos2unix.pl < input_file > output_file
undef $/;
$_ = <>;
s/\r//ge;
print;
0

perl command to convert dos line ending to unix line ending with backup of the original file:

perl -pi.bak -e 's/\r\n/\n/g' filename

This command generates filename with unix line ending and leaves the original file as filename.bak.

-1

In vi hit :.

Then s/Control-VControl-M//g.

Control-V Control-M are obviously those keys. Don't spell it out.

1
  • 1
    It's a bad idea to include non-printing characters like carriage return verbatim in source code like this. Far better to use the \r escape that is (a) easy to see and (b) won't get lost if the source is reformatted.
    – Denis Howe
    Dec 2, 2015 at 17:16

Not the answer you're looking for? Browse other questions tagged or ask your own question.