80

I'm using sed for updating my JSON configuration file in the runtime. Sometimes, when the pattern doesn't match in the JSON file, sed still exits with return code 0.

Returning 0 means successful completion, but why does sed return 0 if it doesn't find the proper pattern and update the file? Is there a workaround for that?

8
  • 9
    Yes, as far as sed is concerned it did its job: it tried to edit the file. It will return an error code if the file is not readable or something like that. In short, I don't think sed is the best tool for deciding whether a pattern appears in a file.
    – cnicutar
    Apr 12, 2013 at 7:07
  • @cnicutar many people post comment as answer, but I think you just put answer in comment area. and I guess OP doesn't only want to check pattern match, he wants to do something if pattern match.. like /pat/s/foo/bar/...
    – Kent
    Apr 12, 2013 at 7:21
  • @Kent That's why I didn't post an answer. I explained why sed does this but was unable to come up with an acceptable solution. Everything I could think of entailed calling a separate command before the sed, in order to decide if the file matches. Ideally there should be some solution (perhaps awk with gsub ?) that can do this in one command.
    – cnicutar
    Apr 12, 2013 at 7:25
  • @cnicutar absolutely reasonable, I didn't notice the "workaround" request..... can I answer this question? well, sure the explanation of return code part would be same as yours...
    – Kent
    Apr 12, 2013 at 7:30
  • 1
    This is a duplicate of How to check if `sed` has changed a file May 10, 2015 at 0:13

9 Answers 9

83

as @cnicutar commented, the return code of a command means if the command was executed successfully. has nothing to do with the logic you implemented in the codes/scripts.

so if you have:

echo "foo"|sed '/bar/ s/a/b/'

sed will return 0 but if you write some syntax/expression errors, or the input/file doesn't exist, sed cannot execute your request, sed will return 1.

workaround

this is actually not workaround. sed has q command: (from man page):

 q [exit-code]

here you can define exit-code as you want. For example '/foo/!{q100}; {s/f/b/}' will exit with code 100 if foo isn't present, and otherwise perform the substitution f->b and exit with code 0.

Matched case:

kent$  echo "foo" | sed  '/foo/!{q100}; {s/f/b/}'
boo
kent$  echo $?
0

Unmatched case:

kent$ echo "trash" | sed  '/foo/!{q100}; {s/f/b/}'
trash
kent$ echo $?
100

I hope this answers your question.

edit

I must add that, the above example is just for one-line processing. I don't know your exact requirement. when you want to get exit 1. one-line unmatched or the whole file. If whole file unmatching case, you may consider awk, or even do a grep before your text processing...

13
  • 15
    This answer is great for GNU sed, but note that the q option is not portable to BSD (Mac) sed. May 15, 2014 at 20:09
  • 3
    Also it is good for one line input. But if you want process file and want known if on any line replacement had been done - it is not the solution. Does anyone known how to deal in that situation?
    – Hubbitus
    Jan 6, 2016 at 18:20
  • 2
    @MohammedNoureldin it is really common usage in sed. like seq 20|sed '/7/!d' read the man page.
    – Kent
    Jul 24, 2017 at 13:11
  • 4
    Thanks for the useful answer. However, I'm not sure about your and @cnicutar 's explanation about return codes. After all, grep on linux fails with a return code of -1 when the grep pattern cannot be found. As far as I can see, this is incompatible with sed's philosophy of retcode: grep also "tried its best" to find a (correctly specified) pattern, there were no errors or missing input files, and yet it resulted in an error code, whereas for sed everything was fine. In the end, we're left guessing which commands interpret what as an error and there's no overall answer...
    – Andres F.
    Jan 9, 2018 at 18:12
  • 2
    It seems pretty clear that the original request was to find a way for sed to "either modify some line (possibly multiple lines) or tell me you could not do it". Suggestion like "run grep first" is not useful, if sed is used to modify stdin. So the search is for a more general solution.
    – igorlord
    May 10, 2019 at 0:31
60

This might work for you (GNU sed):

sed '/search-string/{s//replacement-string/;h};${x;/./{x;q0};x;q1}' file

If the search-string is found it will be replaced with replacement-string and at end-of-file sed will exit with 0 return code. If no substitution takes place the return code will be 1.

A more detailed explanation:

In sed the user has two registers at his disposal: the pattern space (PS) in which the current line is loaded into (minus the linefeed) and a spare register called the hold space (HS) which is initially empty.

The general idea is to use the HS as a flag to indicate if a substitution has taken place. If the HS is still empty at the end of the file, then no changes have been made, otherwise changes have occurred.

The command /search-string/ matches search-string with whatever is in the PS and if it is found to contain the search-string the commands between the following curly braces are executed.

Firstly the substitution s//replacement-string/ (sed uses the last regexp i.e. the search-string, if the lefthand-side is empty, so s//replacement-string is the same as s/search-string/replacement-string/) and following this the h command makes a copy of the PS and puts it in the HS.

The sed command $ is used to recognise the last line of a file and the following then occurs.

First the x command swaps the two registers, so the HS becomes the PS and the PS becomes the HS.

Then the PS is searched for any character /./ (. means match any character) remember the HS (now the PS) was initially empty until a substitution took place. If the condition is true the x is again executed followed by q0 command which ends all sed processing and sets the return code to 0. Otherwise the x command is executed and the return code is set to 1.

N.B. although the q quits sed processing it does not prevent the PS from being reassembled by sed and printed as per normal.

Another alternative:

sed '/search-string/!ba;s//replacement-string/;h;:a;$!b;p;x;/./Q;Q1' file

or:

sed '/search-string/,${s//replacement-string/;b};$q1' file
6
  • 1
    +1 I was having trouble getting Kent's example to work. I also tried your last one and could not get it to work either. I'm sure it has something to do with the search string I'm using. However, the first one in your thread works great! Jun 21, 2013 at 19:38
  • The first solution doesn't work for the case when the file is empty... It should exit with 1 (no match) and it actually exits with 0. Despite this edge condition, it took me 10 minutes with the man page just to figure out what you were doing. Impressive! Mar 15, 2015 at 14:08
  • Would you explain your sed commands in details? I mean each parameter, because it is totally unclear for me how does it work Jul 24, 2017 at 0:21
  • This can be simplified a lot by inverting your logic. Just q1 on match, otherwise let it fail with 0 exit status as usual at the end.
    – robert
    Jan 18, 2018 at 16:43
  • Perfect!!! This satisfied all requirements: allow editing multiple lines; return 1 iff no lines matched; can be used for editing stdin (so no "just do a grep first").
    – igorlord
    May 10, 2019 at 1:00
32

These answers are all too complicated. What is wrong with writing a bit of shell script that uses grep to figure out if the thing you want to replace is there then using sed to replace it?

grep -q $TARGET_STRING $file
if [ $? -eq 0 ]
then
    echo "$file contains the old site"
    sed -e "s|${TARGET_STRING}|${NEW_STRING}|g" ....
fi
6
  • 1
    This also doesn't work on, say, Windows without Cygwin ;-)
    – Cameron
    Sep 8, 2016 at 17:30
  • 1
    Well for one you would be scanning the file twice... (you would probably introduce it to the cache so might not be that bad, but it depends factors I don't know well enough, and I suspect for large enough files this will be a problem). Jul 2, 2018 at 22:21
  • @OhadSchneider - the initial question specified that they were working with a JSON configuration file. How likely is it that a JSON configuration file is going to be large enough for the double-scan to matter? May 14, 2020 at 15:05
  • 3
    I like this because when I come back in a month and forget the two hours I put into learning the sed syntax I will more easily understand my code as "do a simple string replace and see if that new string is there or else exit"
    – Drakes
    Jan 26, 2021 at 17:05
11

For 1 line of input. To avoid repeating the /pattern/:

When s succeeds to substitute, use t to jump conditionally to a label, e.g. x. Otherwise use q to quit with an exit code, e.g. 100:

's/pattern/replacement/;tx;q100;:x'

Example:

$ echo 1 > one
$ < one sed 's/1/replaced-it/;tx;q1;:x'
replaced-it
$ echo $?
0
$ < one sed 's/999/replaced-it/;tx;q100;:x'
1
$ echo $?
100

https://www.gnu.org/software/sed/manual/html_node/Branching-and-flow-control.html

4
  • 5
    From the man page: “if label is omitted, branch to end of script”. Looks like s///; t; q 128 is even shorter. Jun 19, 2021 at 10:14
  • 2
    Notice that this uses GNU extensions which are not portable to more regular sed variants.
    – tripleee
    Nov 24, 2021 at 10:43
  • @tripleee according to the man-page T is a GNU extension, but t is not.
    – GitProphet
    Aug 8 at 12:30
  • The numeric argument to q is what I was referring to.
    – tripleee
    Aug 8 at 14:45
3

Below is the pattern we use with sed -rn or sed -r.

The entire search and replace command ("s/.../.../...") is optional. If the search and replace is used, for speed and having already matched $matchRe, we use as fast a $searchRe value as possible, using . where the character does not need to be re-verified and .{$len} for fixed length sections of the pattern.

The return value for none found is $notFoundExit.

/$matchRe/{s/$searchRe/$replacement/$options; Q}; q$notFoundExit

For the following reasons:

  • No time wasted testing for both matched and unmatched case
  • No time wasted copying to or from buffers
  • No superfluous branches
  • Reasonable flexibility

Varying the case of Q commands will vary the behavior depending on when the exit should occur. Behaviors involving the application of Boolean logic to a multiple line input requires more complexity in the solution.

3

We have the answer above but it took some time for me work out what is happening. I am trying to provide a simple explanation for basic user of sed like me.

Lets consider the example:

echo "foo" | sed  '/foo/!{q100}; {s/f/b/}'

Here we have two sed commands. First one is '/foo/!{q100}' This command actually check the pattern matching and return exist code 100 if no match. Consider following examples, -n is used to silent the output so we only get exist code.

This example foo matches so exit code return is 0

echo "foo" | sed -n '/foo/!{q100}'; echo $?
0

This example input is foo and we try match boo so no match and exit code 100 is returned

echo "foo" | sed  -n '/boo/!{q100}'; echo $?
100

So if my requirement is only to check a pattern match or not I can use

echo "<input string>" | sed -n '/<pattern to match>/!{q<exit-code>}'

More examples:

echo "20200206" | sed -n '/[0-9]*/!{q100}' && echo "Matched" || echo "No Match"
Matched

echo "20200206" | sed -n '/[0-9]{2}/!{q100}' && echo "Matched" || echo "No Match"
No Match

Second command is '{s/f/b/}' is to replace the f in foo with b which I used many times.

1
  • 2
    If your requirement is only to check if a pattern matches or not, use grep. Mar 13, 2020 at 15:46
2

For any number of input lines:

sed --quiet 's/hello/HELLO/;t1;b2;:1;h;:2;p;${g;s/..*//;tok;q1;:ok}'

Fills hold space on match, and checks it after the last line. Returns status 1 if no match in file.

  • s/hello/HELLO - substitution to check for
  • t1 - jump to label 1 if substitution succeeded
  • b2 - jump to label 2 unconditionally
  • :1 - label 1
  • h - copy pattern to hold space (when substitution succeeded)
  • :2 - label 2
  • p - print pattern space, unconditionally
  • ${ ... } - match last line, evaluate block inside
  • g - copy hold space into pattern space (non-empty if first substitution succeded before)
  • s/..*// - dummy substitution, to set branch-flag
  • tok - jump to label ok (if dummy substitution succeeded on non-empty hold space)
  • q1 - exit with error status 1
  • :ok - label ok
1

As we already know, when sed fails to match then it simply returns its input string - no error has occurred. It is true that a difference between the input and output strings implies a match, but a match does not imply a difference in the strings; after all sed could have simply matched all of the input characters. The flaw is created in the following example

h=$(echo "$g" | sed 's/.*\(abc[[:digit:]]\).*/\1/g')    
if [ ! "$h" = "$g" ]; then    
  echo "1"   
else    
  echo "2"    
fi

where g=Xabc1 gives 1, while setting g=abc1 gives 2; yet both of these input strings are matched by sed! So, it can be hard to determine whether sed has matched or not. A solution:

h=$(echo "fix${g}ed" | sed 's/.*\(abc[[:digit:]]\).*/\1/g')
if [ ! "$h" = "fix${g}ed" ]; then    
  echo "1"   
else    
  echo "2"    
fi

in which case the 1 is printed if-and-only-if sed has matched.

0

I had wanted to truncate a file by quitting when the match was found (and exclude the matching line). This is handy when a process that adds lines at the end of the file may be re-run. "Q;Q1" didn't work but simply "Q1" did, as follows:

if sed -i '/text I wanted to find/Q1' file.txt then insert blank line at end of file + new lines fi insert just the new lines without the blank line

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.