33

I'm tailing logs and they output \n instead of newlines.

I thought I'd pipe the tail to awk and do a simple replace, however I cannot seem to escape the newline in the regex. Here I'm demonstrating my problem with cat instead of tail:

test.txt:

John\nDoe
Sara\nConnor
cat test.txt | awk -F'\\n' '{ print $1 "\n" $2 }'

Desired output:

John
Doe
Sara
Connor

Actual output:

John\nDoe

Sara\nConnor

So it looks like \\n does not match the \n between the first and last names in test.txt but instead the newline at the end of each line.

It looks like \\n is not the right way of escaping in the terminal right? This way of escaping works fine in e.g. Sublime Text:

regex working in ST3

7 Answers 7

47

How about this?

$ cat file
John\nDoe
Sara\nConnor

$ awk '{gsub(/\\n/,"\n")}1' file
John
Doe
Sara
Connor
6
  • Why you are using awk? sed seems more appropriate for editing files.
    – hek2mgl
    Jul 4, 2014 at 12:41
  • 10
    op tagged the awk tag. Jul 4, 2014 at 12:42
  • 2
    You could change "\n" with RS and get awk '{gsub(/\\n/,RS)}1'
    – Jotne
    Jul 4, 2014 at 12:42
  • 2
    Maybe I should not have tagged this awk but I did, and this answer works. However the sed seems better for this task? The problem is sed behaves weird on Mac OSX, see @DarkDust and @Ed Morton answers.
    – Cotten
    Jul 4, 2014 at 13:28
  • 2
    @Cotten i'm not compel you. Go for anyother solution which seems best for you. Jul 4, 2014 at 13:30
13

Using GNU's sed, the solution is pretty simple as @hek2mgl already answered (and that IMHO is the way it should work everywhere, but unfortunately doesn't).

But it's bit tricky when doing it on Mac OS X and other *BSD UNIXes.

The best way looks like this:

sed 's/\\n/\'$'\n''/g' <<< 'ABC\n123'

Then of course there's still AWK, @AvinashRaj has the correct answer if you'd like to use that.

8
  • 1
    Err, no. It's the other way around: GNU's sed is extending POSIX sed; that's why it needs --posix in the first place.
    – DarkDust
    Jul 4, 2014 at 13:20
  • 1
    I think your syntax is a little off - it should be sed 's/\\n/\'$'\n''/g' to insert a literal newline (generated by $'\n') in the script before sed processes it. I'm not sure what the shell is making of the standalone $ between the 2 halves of your sed script ('s/\\n/\' and '\n/g'). Jul 4, 2014 at 13:43
  • 2
    So, let's actually look at the POSIX standard: the problem is that the standard does not specify whether second part of s (the "replacement") shall interpret \n or not. Since it's not a BRE and the "\" has special meaning here I'd say it shouldn't. The BSD sed's POSIX notes state that historic versions didn't and discarded the "\" (see point 16). So both are POSIX compatible since the standard doesn't specify the behavior.
    – DarkDust
    Jul 4, 2014 at 13:45
  • 2
    Yeah, what's happening is that the shell is evaluating $'\n/g' before sed tries to execute the script and that expands to a literal newline followed by /g so it "works" by co-incidence. It would not work with different characters after the / that the shell would expand - /g just happens to be harmless. Jul 4, 2014 at 13:51
  • 1
    +1, but note that the use of an ANSI C-quoted string, $'\n', makes the solution shell-dependent; fortunately, though, popular shells (bash, zsh, ksh) DO support that, but POSIX-features-only ones such as dash do not.
    – mklement0
    Jul 4, 2014 at 19:38
9

Why use either awk or sed for this? Use perl!

perl -pe 's/\\n/\n/g' file

By using perl you avoid having to think about posix compliance, and it will typically give better performance, and it will be consistent across all (most) platforms.

1
  • because not every system has perl installed for example. Sep 30, 2020 at 16:18
9

This will work with any sed on any system as it is THE portable way to use newlines in sed:

$ sed 's/\\n/\
/' file
John
Doe
Sara
Connor

If it is possible for your input to contain a line like foo\\nbar and the \\ is intended to be an escaped backslash then you cannot use a simple substitution approach like you've asked for.

4
  • 1
    Yes, you are right. Could you explain why GNU sed --posix behaves differently than BSD sed (which also claims to be POSIX compatible). Is the problem caused by GNU sed --posix not working properly or by BSD sed not being POSIX compatible?
    – hek2mgl
    Jul 4, 2014 at 13:25
  • Sorry, no, I'd need to read the POSIX spec for sed to figure that one out and life's too short... I will say , though, that BSD awk is broken in some ways (e.g. parsing of unparenthesized ternary expressions in print statements) so maybe their sed is too? Jul 4, 2014 at 13:28
  • 1
    I would need too, thought maybe you know it already... :) thx . +1 for the solution
    – hek2mgl
    Jul 4, 2014 at 13:29
  • @hek2mgl, as per POSIX, The meaning of a <backslash> immediately followed by any character other than '&', <backslash>, a digit, or the delimiter character used for this command, is unspecified., so the GNU sed behaviour is conformant, as would be a sed that outputs \n or one that reboots your computer. A script that uses \n there would be non-conformant (for that very reason that the behaviour is unspecified). Nov 22, 2015 at 12:52
7

I have struggled with this problem before, but I discovered the cleanest way is to use the builtin printf

printf "$(cat file.txt)" | less

Here is a real world example dealing with aws iam embeded json policy in the output, the file file.txt contains:

{
  "registryId": "111122223333",
  "repositoryName": "awesome-repo",
  "policyText": "{\n  \"Version\" : \"2008-10-17\",\n  \"Statement\" : [ {\n    \"Sid\" : \"AllowPushPull\",\n    \"Effect\" : \"Allow\",\n    \"Principal\" : {\n      \"AWS\" : [ \"arn:aws:iam::444455556666:root\", \"arn:aws:iam::444455556666:user/johndoe\" ]\n    },\n    \"Action\" : [ \"ecr:BatchCheckLayerAvailability\", \"ecr:BatchGetImage\", \"ecr:CompleteLayerUpload\", \"ecr:DescribeImages\", \"ecr:DescribeRepositories\", \"ecr:GetDownloadUrlForLayer\", \"ecr:InitiateLayerUpload\", \"ecr:PutImage\", \"ecr:UploadLayerPart\" ]\n  } ]\n}"
}

after applying the above (without the less) you get:

{
  "registryId": "111122223333",
  "repositoryName": "awesome-repo",
  "policyText": "{
  "Version" : "2008-10-17",
  "Statement" : [ {
    "Sid" : "AllowPushPull",
    "Effect" : "Allow",
    "Principal" : {
      "AWS" : [ "arn:aws:iam::444455556666:root", "arn:aws:iam::444455556666:user/johndoe" ]
    },
    "Action" : [ "ecr:BatchCheckLayerAvailability", "ecr:BatchGetImage", "ecr:CompleteLayerUpload", "ecr:DescribeImages", "ecr:DescribeRepositories", "ecr:GetDownloadUrlForLayer", "ecr:InitiateLayerUpload", "ecr:PutImage", "ecr:UploadLayerPart" ]
  } ]
}"
}

Note that the value for "policyText" is itself a string containing json.

2
  • Brilliant, simple, and works great.
    – Sym
    Mar 11, 2022 at 16:22
  • Terrible if your text happens to contain a percent symbol (%). Use printf "$(sed 's/%/%%/g' file.txt)" instead
    – roaima
    Jun 8, 2022 at 22:27
5

I would use sed:

sed 's/\\n/\n/g' file
11
  • That does not work for me: sed 's/\\n/\n/g' test.txt output: line1: JohnnDoe line2: SaranConnor
    – Cotten
    Jul 4, 2014 at 12:44
  • It should. Try: sed 's/\\n/\n/g' <<< 'ABC\n123'
    – hek2mgl
    Jul 4, 2014 at 12:46
  • sed 's/\\n/\n/g' <<< 'ABC\n123' gives me ABCn123. Is this a plattform issue? I'm on OS X using zsh
    – Cotten
    Jul 4, 2014 at 12:50
  • sed --posix 's/\\n/\n/g' <<< 'ABC\n123' works. Srry, use Linux. I'm not the OSX support I'm tired of supporting this and will never get why a Mac should be used for hacking. (its not against you personally)
    – hek2mgl
    Jul 4, 2014 at 12:55
  • 1
    BSD's sed doesn't have a --posix, it is POSIX conform already. GNU's sed has a lot of (useful) extensions to POSIX that --posix is supposed to disable. I found an answer with a solution that works on all systems (tested on Mac and Linux).
    – DarkDust
    Jul 4, 2014 at 13:18
0

In addition to the accepted answer, OP asked about tail, and on some unix variants, eg ubuntu you need to add -W interactive to awk

tail -f error.log | awk -W interactive '{gsub(/\\n/,"\n")}1'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.