34

I am trying to find a clever way to figure out if the file passed to sed has been altered successfully or not.

Basically, I want to know if the file has been changed or not without having to look at the file modification date.

The reason why I need this is because I need to do some extra stuff if sed has successfully replaced a pattern.

I currently have:

    grep -q $pattern $filename
    if [ $? -eq 0 ]
    then
        sed -i s:$pattern:$new_pattern: $filename
                # DO SOME OTHER STUFF HERE
    else
        # DO SOME OTHER STUFF HERE
    fi

The above code is a bit expensive and I would love to be able to use some hacks here.

9
  • 5
    Any time you are trying to do something "clever", you probably shouldn't do it. Aug 27, 2012 at 14:42
  • 5
    @WilliamPursell because the world was built with stupid inventions. Aug 27, 2012 at 14:43
  • What about writing the changes to a new file, and then diff-ing the original and the generated? By the way, shouldn't sed always replace the pattern if grep found it before?
    – AlvaroGMJ
    Aug 27, 2012 at 14:44
  • 1
    Can SOME OTHER STUFF be done with sed, too? Aug 27, 2012 at 14:56
  • 5
    sed's exit code does not reflect whether any matches were found.
    – chepner
    Aug 27, 2012 at 15:11

12 Answers 12

46

A bit late to the party but for the benefit of others, I found the 'w' flag to be exactly what I was looking for.

sed -i "s/$pattern/$new_pattern/w changelog.txt" "$filename"
if [ -s changelog.txt ]; then
    # CHANGES MADE, DO SOME STUFF HERE
else
    # NO CHANGES MADE, DO SOME OTHER STUFF HERE
fi

changelog.txt will contain each change (ie the changed text) on it's own line. If there were no changes, changelog.txt will be zero bytes.

A really helpful sed resource (and where I found this info) is http://www.grymoire.com/Unix/Sed.html.

13
  • 1
    This overwrites the changelog.txt file. Any idea how to append to it instead? May 10, 2015 at 0:19
  • Not directly, however some out-of-process file manipulation is certainly doable. May 11, 2015 at 2:36
  • 2
    I find the overwriting convenient... After every sed I can check without keeping in mind to delete the file. Oct 2, 2015 at 11:22
  • 4
    also works with s/search/replace/ w /dev/stdout - no need to create temporary files : ) see the solution by acecilia
    – milahu
    Oct 30, 2021 at 20:02
  • 2
    @pgr sed -i "s/a/b/g w /dev/stdout" file
    – Max_Payne
    May 15, 2022 at 17:22
10

I believe you may find these GNU sed extensions useful

t label

If a s/// has done a successful substitution since the last input line
was read and since the last t or T command, then branch to label; if
label is omitted, branch to end of script.

and

q [exit-code]

Immediately quit the sed script without processing any more input, except 
that if auto-print is not disabled the current pattern space will be printed. 
The exit code argument is a GNU extension.

It seems like exactly what are you looking for.

4
  • 39
    I don't understand how to use this Sep 29, 2014 at 9:57
  • 1
    I too do not understand. Hostmaster could you please explain where do we put this options? I'm running sed within an install script and I need to raise warning if sed doesn't manage to change configuration file. Oct 2, 2015 at 11:20
  • 5
    How to use t and q options with sed command explained here: askubuntu.com/a/1036918/250399. TLDR: sed -i 's/orig/repl/; t; q1' file.txt Dec 23, 2019 at 3:20
  • 10
    Important: looks like this works only for one-line patterns. For sed-ing a whole file, where the replacement is done somewhere in the middle, after sed parses the first line and the replacement is not found there, it will just print the first line unchanged and exit.
    – quezak
    Jun 5, 2020 at 9:12
10

This might work for you (GNU sed):

sed -i.bak '/'"$old_pattern"'/{s//'"$new_pattern"'/;h};${x;/./{x;q1};x}' file || echo changed

Explanation:

  • /'"$old_pattern"'/{s//'"$new_pattern"'/;h} if the pattern space (PS) contains the old pattern, replace it by the new pattern and copy the PS to the hold space (HS).
  • ${x;/./{x;q1};x} on encountering the last line, swap to the HS and test it for the presence of any string. If a string is found in the HS (i.e. a substitution has taken place) swap back to the original PS and exit using the exit code of 1, otherwise swap back to the original PS and exit with the exit code of 0 (the default).
0
6

You can diff the original file with the sed output to see if it changed:

sed -i.bak s:$pattern:$new_pattern: "$filename"
if ! diff "$filename" "$filename.bak" &> /dev/null; then
  echo "changed"
else
  echo "not changed"
fi
rm "$filename.bak"
4
  • 3
    Is diff going to be less expensive than the grep he's trying to replace?
    – chepner
    Aug 27, 2012 at 14:51
  • 1
    Hey, thank you for the solution, but I think doing a diff on each file i am gonna search through might be a bit cpu intensive. What do you think ? Aug 27, 2012 at 14:53
  • 1
    I think so too, this is not really efficient
    – perreal
    Aug 27, 2012 at 14:54
  • 3
    diff shows you what is different. If you just want to know if there is any difference cmp is sufficient. Also [ $? -ne 0] is useless. Just put the command into the if condition: if cmp "$filename" "$filename".bak; then.
    – ceving
    Mar 14, 2016 at 13:43
4

In macos I just do it as follows:

changes=""
changes+=$(sed -i '' "s/$to_replace/$replacement/g w /dev/stdout" "$f")
if [ "$changes" != "" ]; then
  echo "CHANGED!"
fi

I checked, and this is faster than md5, cksum and sha comparisons

1
  • limitation: can handle only one substitution. so this does not work: sed -i 's,a,b, w /dev/stdout ; s,b,a, w /dev/stdout' input.txt. see my workaround
    – milahu
    Oct 30, 2021 at 20:59
3

You could use awk instead:

awk '$0 ~ p { gsub(p, r); t=1} 1 END{ exit (!t) }' p="$pattern" r="$repl"

I'm ignoring the -i feature: you can use the shell do do redirections as necessary.

Sigh. Many comments below asking for basic tutorial on the shell. You can use the above command as follows:

if awk '$0 ~ p { gsub(p, r); t=1} 1 END{ exit (!t) }' \
        p="$pattern" r="$repl" "$filename" > "${filename}.new"; then
    cat "${filename}.new" > "${filename}"
    # DO SOME OTHER STUFF HERE
else
    # DO SOME OTHER STUFF HERE
fi

It is not clear to me if "DO SOME OTHER STUFF HERE" is the same in each case. Any similar code in the two blocks should be refactored accordingly.

8
  • 3
    You can't do in place modifications using only shell redirection. command file > file doesn't work (file is truncated at the moment the redirection is applied, which happens before the command is started)
    – AlvaroGMJ
    Aug 27, 2012 at 14:52
  • awk '...' $filename > tmp.txt; mv tmp.txt $filename. sed -i just hides the details of the temp file from you.
    – chepner
    Aug 27, 2012 at 14:56
  • 1
    @AlvaroGMJ: you cannot do in-place modifications using sed -i, but you certainly can do it with shell redirections. But you are right, you cannot do it with cmd file > file. Aug 27, 2012 at 15:02
  • 3
    Can you give a full example of the solution using awk? It's not clear what you mean about ignoring the -i feature or what sed has to do with this. Your code as it is doesnt appear to work
    – qodeninja
    Jun 3, 2015 at 15:50
  • Can you expend this answer ? What is "!t" ? What is "exit" ? Mar 11, 2020 at 14:08
0

I know it is a old question and using awk instead of sed is perhaps the best idea, but if one wants to stick with sed, an idea is to use the -w flag. The file argument to the w flag only contains the lines with a match. So, we only need to check that it is not empty.

0
perl -sple '$replaced++ if s/$from/$to/g;
                END{if($replaced != 0){ print "[Info]: $replaced replacement done in $ARGV(from/to)($from/$to)"}
                else {print "[Warning]: 0 replacement done in $ARGV(from/to)($from/$to)"}}' -- -from="FROM_STRING" -to="$DESIRED_STRING" </file/name>

Example: The command will produce the following output, stating the number of changes made/file.

perl -sple '$replaced++ if s/$from/$to/g;
END{if($replaced != 0){ print "[Info]: $replaced replacement done in $ARGV(from/to)($from/$to)"}
else {print "[Warning]: 0 replacement done in $ARGV(from/to)($from/$to)"}}' -- -from="timeout" -to="TIMEOUT" *
[Info]: 5 replacement done in main.yml(from/to)(timeout/TIMEOUT)
[Info]: 1 replacement done in task/main.yml(from/to)(timeout/TIMEOUT)
[Info]: 4 replacement done in defaults/main.yml(from/to)(timeout/TIMEOUT)
[Warning]: 0 replacement done in vars/main.yml(from/to)(timeout/TIMEOUT) 

Note: I have removed -i from the above command , so it will not update the files for the people who are just trying out the command. If you want to enable in-place replacements in the file add -i after perl in above command.

0

check if sed has changed MANY files

  • recursive replace of all files in one directory
  • produce a list of all modified files

workaround with two stages: match + replace

g='hello.*world'
s='s/hello.*world/bye world/g;'
d='./' # directory of input files
o='modified-files.txt'

grep -r -l -Z -E "$g" "$d" | tee "$o" | xargs -0 sed -i "$s"

the file paths in $o are zero-delimited

0
$ echo hi > abc.txt
$ sed "s/hi/bye/g; t; q1;" -i abc.txt && (echo "Changed") || (echo "Failed")
Changed
$ sed "s/hi/bye/g; t; q1;" -i abc.txt && (echo "Changed") || (echo "Failed")
Failed

https://askubuntu.com/questions/1036912/how-do-i-get-the-exit-status-when-using-the-sed-command/1036918#1036918

0

The problem with s/foo/bar/;t;q8 is that is quits after the first line. To set a proper exit code for multi-line or multi-substitution situations you can use the following:

sed -i "s/foo/bar/w /dev/stdout" dir/file.txt | grep -q .
echo $?

which uses -i to change the file in-place, the w command to send the successful substitutions to /dev/stdout, then we read that with grep . to see if output is empty. This returns an exit code of 1 when no match is found, and 0 when substitutions have been made. Use this in conjunction with set -e for example.

-1

Don't use sed to tell if it has changed a file; instead, use grep to tell if it is going to change a file, then use sed to actually change the file. Notice the single line of sed usage at the very end of the Bash function below:

# Usage: `gs_replace_str "regex_search_pattern" "replacement_string" "file_path"`
gs_replace_str() {
    REGEX_SEARCH="$1"
    REPLACEMENT_STR="$2"
    FILENAME="$3"

    num_lines_matched=$(grep -c -E "$REGEX_SEARCH" "$FILENAME")
    # Count number of matches, NOT lines (`grep -c` counts lines), 
    # in case there are multiple matches per line; see: 
    # https://superuser.com/questions/339522/counting-total-number-of-matches-with-grep-instead-of-just-how-many-lines-match/339523#339523
    num_matches=$(grep -o -E "$REGEX_SEARCH" "$FILENAME" | wc -l)

    # If num_matches > 0
    if [ "$num_matches" -gt 0 ]; then
        echo -e "\n${num_matches} matches found on ${num_lines_matched} lines in file"\
                "\"${FILENAME}\":"
        # Now show these exact matches with their corresponding line 'n'umbers in the file
        grep -n --color=always -E "$REGEX_SEARCH" "$FILENAME"
        # Now actually DO the string replacing on the files 'i'n place using the `sed` 
        # 's'tream 'ed'itor!
        sed -i "s|${REGEX_SEARCH}|${REPLACEMENT_STR}|g" "$FILENAME"
    fi
}

Place that in your ~/.bashrc file, for instance. Close and reopen your terminal and then use it.

Usage:

gs_replace_str "regex_search_pattern" "replacement_string" "file_path"

Example: replace do with bo so that "doing" becomes "boing" (I know, we should be fixing spelling errors not creating them :) ):

$ gs_replace_str "do" "bo" test_folder/test2.txt 

9 matches found on 6 lines in file "test_folder/test2.txt":
1:hey how are you doing today
2:hey how are you doing today
3:hey how are you doing today
4:hey how are you doing today  hey how are you doing today  hey how are you doing today  hey how are you doing today
5:hey how are you doing today
6:hey how are you doing today?
$SHLVL:3 

Screenshot of the output:

enter image description here

References:

  1. https://superuser.com/questions/339522/counting-total-number-of-matches-with-grep-instead-of-just-how-many-lines-match/339523#339523
  2. https://unix.stackexchange.com/questions/112023/how-can-i-replace-a-string-in-a-files/580328#580328

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.