104

When I run a particular SQL script in Unix environments, I see a '^M' character at the end of each line of the SQL script as it is echoed to the command line.
I don't know on which OS the SQL script was initially created.

What is causing this and how do I fix it?

17 Answers 17

81

It's caused by the DOS/Windows line-ending characters. Like Andy Whitfield said, the Unix command dos2unix will help fix the problem. If you want more information, you can read the man pages for that command.

2
  • 3
    On some systems (i.e. Ubuntu) the name of this command is "fromdos" Commented Dec 4, 2012 at 16:12
  • 6
    You can get the tool on OSX very easy with brew install dos2unix when you have homebrew installed
    – philipp
    Commented Jun 12, 2013 at 17:04
81

Fix line endings in vi by running the following:

:set fileformat=unix

:w

2
  • 2
    This is a brilliant answer. Many thanks. (saved installing dos2unix, a tool I'd probably only use once)
    – James
    Commented Feb 4, 2014 at 22:29
  • 3
    this doesnt remove the ^Ms for some reason. reference file: /etc/timidity/fluidr3_gm.cfg.
    – phil294
    Commented Apr 6, 2017 at 1:11
41

The cause is the difference between how a Windows-based based OS and a Unix based OS store the end-of-line markers.

Windows based operating systems, thanks to their DOS heritage, store an end-of-line as a pair of characters - 0x0D0A (carriage return + line feed). Unix-based operating systems just use 0x0A (a line feed). The ^M you're seeing is a visual representation of 0x0D (a carriage return).

dos2unix will help with this. You probably also need to adjust the source of the scripts to be 'Unix-friendly'.

5
  • I wouldn't say current versions of Windows have any kind of DOS heritage. They still have compatibility restraints, though.
    – Joey
    Commented Mar 8, 2013 at 12:30
  • This is the easy way, is you do an automatic conversion tool. Thank's
    – Pjl
    Commented Apr 9, 2015 at 13:57
  • But why ^M? Why the '^'? Why the 'M'?
    – 1737973
    Commented Jul 28, 2015 at 4:47
  • 1
    Because it's a "control character". "^" is the visual representation of clicking the control key. Underneath its just specific bytes, the ^ is how the editor represents them.
    – Hejazzman
    Commented Aug 5, 2015 at 22:10
  • 1
    Upvoted for the clear and informative answer. By the way, I just tried all different CSV options in Excel "Save as..." but all of them produce \r which I can see by the ^M on Linux with cat -e filename.csv. Also tried Calc from Apache OpenOffice which is now on the Microsoft Store, and same result. Conclusion, I'll use Excel or Calc for editing convenience and just get rid of ^M with dos2unix before git commit.
    – Nagev
    Commented Jan 21, 2022 at 15:27
25

The easiest way is to use vi. I know that sounds terrible but its simple and already installed on most UNIX environments. The ^M is a new line from Windows/DOS environment.

from the command prompt: $ vi filename

Then press ":" to get to command mode.

Search and Replace all Globally is :%s/^M//g "Press and hold control then press V then M" which will replace ^M with nothing.

Then to write and quit enter ":wq" Done!

4
  • 1
    How to replace it in emacs?
    – herbertD
    Commented Nov 1, 2013 at 3:09
  • 4
    Thanks for the expanation on how to type the ^M character! I would replace it with \r instead. So I did :%s/^M/\r/g
    – aharris88
    Commented Nov 24, 2014 at 20:05
  • This doesn't work for nvim inside the Windows Subsystem for Linux (WSL).
    – 71GA
    Commented Apr 1, 2022 at 13:06
  • @herbertD, maybe this will help for Emacs emacswiki.org/emacs/DosToUnix. dos2unix also comes by default with Doom Emacs
    – eho
    Commented Jun 30, 2022 at 2:08
14

Try using dos2unix to strip off the ^M.

10

In vi, do a :%s/^M//g

To get the ^M hold the CTRL key, press V then M (Both while holding the control key) and the ^M will appear. This will find all occurrences and replace them with nothing.

1
  • 2
    To replace the ^M with a unix friendly line break: :%s/^M/\r/g
    – Gary Oak
    Commented Feb 26, 2014 at 19:46
8

The SQL script was originally created on a Windows OS. The '^M' characters are a result of Windows and Unix having different ideas about what to use for an end-of-line character. You can use perl at the command line to fix this.

perl -pie 's/\r//g' filename.txt
3
  • Sure, you CAN use perl, but would you suggest perl over dos2unix? Commented Sep 15, 2008 at 17:11
  • 2
    I'm just providing an alternative, since four people already said to use dos2unix. Commented Sep 15, 2008 at 17:17
  • 2
    Yes, I found this useful because I am on a backward workstation working in an office with a prehistoric IT department. Except I used a variation: perl -pi -e "s/\x0D/\n/g" file.csv
    – Rimian
    Commented Feb 10, 2010 at 23:36
7

The ^M is typically caused by the Windows operator newlines, and translated onto Unix looks like a ^M. The command dos2unix should remove them nicely

dos2unix [options] [-c convmode] [-o file ...] [-n infile outfile ...]

5
C:\tmp\text>dos2unix hello.txt helloUNIX.txt

Sed is even more widely available and can do this kind of thing also if dos2unix is not installed

C:\tmp\text>sed s/\r// hello.txt > helloUNIX.txt  

You could also try tr:

cat hello.txt | tr -d \r > helloUNIX2.txt  

Here are the results:

C:\tmp\text>dumphex hello.txt  
00000000h: 48 61 68 61 0D 0A 68 61 68 61 0D 0A 68 61 68 61 Haha..haha..haha  
00000010h: 0D 0A 0D 0A 68 61 68 61 0D 0A                   ....haha..  

C:\tmp\text>dumphex helloUNIX.txt  
00000000h: 48 61 68 61 0A 68 61 68 61 0A 68 61 68 61 0A 0A Haha.haha.haha..  
00000010h: 68 61 68 61 0A                                  haha.  

C:\tmp\text>dumphex helloUNIX2.txt  
00000000h: 48 61 68 61 0A 68 61 68 61 0A 68 61 68 61 0A 0A Haha.haha.haha..  
00000010h: 68 61 68 61 0A                                  haha.  
5

An alternative to dos2unix command would be using standard utilities like sed.

For example, dos to unix:

sed 's/\r$//' dos.txt > unix.txt

unix to dos:

sed 's/$/\r/' unix.txt > dos.txt
4

To replace ^M characters in vi editor use below

open the text file say t1.txt

vi t1.txt

Enter command mode by pressing shift + :

then press keys as mentioned %s/^M/\r/g

in above ^M is not (shift + 6)M instead it is (ctrl + V)(ctrl + M)
1
  • Your last line is what I was missing from all the previous answers. I kept getting 'no matches found bc I was doing shift+6, so I did what every hacker would and circumvented my misunderstanding with my own solution: record a macro to do $ to go to end of each line and then press x, just repeat macro for num of lines in file.
    – darethas
    Commented Aug 13, 2013 at 5:46
1

You can remove ^M from the files directly via sed command, e.g.:

sed -i'.bak' s/\r//g *.*

If you're happy with the changes, remove the .bak files:

rm -v *.bak
1

Convert DOS/Windows (\r\n) line endings to Unix (\n) line endings, with tr:

tr '\r\n' '\n' < dosFile.txt > unixFile.txt

Post about replacing newlines from the Unix command line

0

od -a $file is useful to explore those types of question on Linux (similar to dumphex in the above).

0

In Perl, if you don't want to set the $/ variable and use chomp() you can also do:

$var =~ /\r\n//g;

My two cents

0

As already explained, Windows programs like to terminate lines with CRLF, i.e. \r\n instead of the Unix/Linux standard \n. Since I don't need all the features of dos2unix I replaced it by adding the following to my ~/.bashrc which removes the \r:

function win2unix() {
    tmp=$(mktemp) && tr -d '\r' < $1 > $tmp && mv $tmp $1
}

Now when I want to get rid of those ^M characters created e.g. when I export a CSV file from Excel or Calc, I can just do something like:

win2unix filename.csv

You could also use sed or something else, of course. By the way, I use cat -e $filename to visualize the ^M endings.

-1

Another vi command that'll do: :%s/.$// This removes the last character of each line in the file. The drawback to this search and replace command is that it doesn't care what the last character is, so be careful not to call it twice.

1
  • Why mention it if you know it's not reliable?
    – minexew
    Commented Feb 7, 2014 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.