How would I use sed to delete all lines in a text file that contain a specific string?
21 Answers
To remove the line and print the output to standard out:
sed '/pattern to match/d' ./infile
To directly modify the file – does not work with BSD sed:
sed -i '/pattern to match/d' ./infile
Same, but for BSD sed (Mac OS X and FreeBSD) – does not work with GNU sed:
sed -i '' '/pattern to match/d' ./infile
To directly modify the file (and create a backup) – works with BSD and GNU sed:
sed -i.bak '/pattern to match/d' ./infile
-
2How do I know what version of sed I have? GNU or non GNU? Mar 23, 2011 at 20:39
-
I dont't know why, but the least command resulted in an empty file :(– marquiesOct 4, 2015 at 21:49
-
1With GNU sed 4,2,2, the -i '' doesn't work at all. It then treats the pattern to match as the filename. Just remove the ''.– hookenzOct 19, 2015 at 22:29
-
1
-
6Note that if your pattern will contain forward slashes, you can use an alternate pattern delimiter but must then escape the first one, e.g.:
sed -i.bak "\#$pattern_variable_containing_slashes#d" ./infile
– EricMar 16, 2016 at 21:37
There are many other ways to delete lines with specific string besides sed
:
AWK
awk '!/pattern/' file > temp && mv temp file
Ruby (1.9+)
ruby -i.bak -ne 'print if not /test/' file
Perl
perl -ni.bak -e "print unless /pattern/" file
Shell (bash 3.2 and later)
while read -r line
do
[[ ! $line =~ pattern ]] && echo "$line"
done <file > o
mv o file
GNU grep
grep -v "pattern" file > temp && mv temp file
And of course sed
(printing the inverse is faster than actual deletion):
sed -n '/pattern/!p' file
-
4how to delete a particular line with a pattern and also the line immediately above it? I have a fine with thousands of such lines in between different data. Aug 6, 2013 at 22:41
-
2On OS/X, the shell variation doesn't preserve leading spaces, but the grep -v variation worked well for me. Feb 3, 2014 at 23:31
-
18the
sed
example have a different behaviour, it only greps! it should be something likesed -n -i '/pattern/!p' file
. Mar 28, 2014 at 16:41 -
11The grep version does not work when every line matches the pattern. Better do:
grep -v "pattern" file > temp; mv temp file
This might apply to some of the other examples depending on the return value. Jun 20, 2014 at 14:43 -
5"printing the inverse is faster than actual deletion" - Not on my machine (2012 MacBook Air, OS X 10.13.2). Create file:
seq -f %f 10000000 >foo.txt
. sed d:time sed -i '' '/6543210/d' foo.txt
real 0m9.294s. sed !p:time sed -i '' -n '/6543210/!p' foo.txt
real 0m13.671s. (For smaller files, the difference is larger.) Jan 22, 2018 at 2:11
You can use sed to replace lines in place in a file. However, it seems to be much slower than using grep for the inverse into a second file and then moving the second file over the original.
e.g.
sed -i '/pattern/d' filename
or
grep -v "pattern" filename > filename2; mv filename2 filename
The first command takes 3 times longer on my machine anyway.
-
32Voting up your answer too, just because you tried a performance comparison!– anuragwApr 12, 2013 at 7:11
-
6
-
3
-
3I'm curious what the performance difference would be if it were
sed '/pattern/d' filename > filename2; mv filename2 filename
– PeteApr 8, 2014 at 1:00 -
14(using ubuntu's /usr/share/dict/words) grep and mv: 0.010s | sed in place: 0.197s | sed and mv: 0.031s Feb 10, 2015 at 14:22
The easy way to do it, with GNU sed
:
sed --in-place '/some string here/d' yourfile
-
83A handy tip for others who stumble on this Q&A thread and are new to shell scripting: Short options are fine for one-time uses on the command line, but long options should be preferred in scripts since they're more readable.– DennisJan 12, 2015 at 10:45
-
4+1 for the --in-place flag. I need to test that out on permissions protected files. (have to do some user scrubbing.)– Bee KayMay 19, 2015 at 21:06
-
10Note that the long option is only available on GNU sed. Mac and BSD users will need to install gsed to do it this way.– MattMay 18, 2016 at 14:31
-
2Another tip: if your regex doesn't appear to match, try the
-r
option (or-E
, depending on your version). This enables the use of regex metacharacters+
,?
,{...}
and(...)
.– rjhSep 17, 2019 at 18:17 -
This is the correct answer when your disk no have more space and you can't copy the text to another file. This command do what was questioned? Dec 26, 2019 at 13:39
You may consider using ex
(which is a standard Unix command-based editor):
ex +g/match/d -cwq file
where:
+
executes given Ex command (man ex
), same as-c
which executeswq
(write and quit)g/match/d
- Ex command to delete lines with givenmatch
, see: Power of g
The above example is a POSIX-compliant method for in-place editing a file as per this post at Unix.SE and POSIX specifications for ex
.
The difference with sed
is that:
sed
is a Stream EDitor, not a file editor.BashFAQ
Unless you enjoy unportable code, I/O overhead and some other bad side effects. So basically some parameters (such as in-place/-i
) are non-standard FreeBSD extensions and may not be available on other operating systems.
-
5that's great... when I do
man ex
it gives me the man forvim
, it seemsex
is part of vim... if I understood right that means the pattern syntax formatch
is vimregex.com which is similar but different to POSIX and PCRE flavours? Nov 15, 2015 at 19:50 -
1
:g
is POSIX-compliant command with some slight differences. I assume PCRE was based on it.– kenorbJan 7, 2016 at 10:24 -
@kenorb "I/O overhead and some other bad side effects" could you elaborate? AFAIK
ex
is using a temp file, just like every other sane tool, besides idk usingdd
– CervEdJan 6, 2022 at 23:47
I was struggling with this on Mac. Plus, I needed to do it using variable replacement.
So I used:
sed -i '' "/$pattern/d" $file
where $file
is the file where deletion is needed and $pattern
is the pattern to be matched for deletion.
I picked the ''
from this comment.
The thing to note here is use of double quotes in "/$pattern/d"
. Variable won't work when we use single quotes.
-
4Mac
sed
requires a parameter after-i
, so if you don't want a backup, you still have to add an empty string:-i ''
– wisbuckyJan 12, 2017 at 0:51 -
2
You can also use this:
grep -v 'pattern' filename
Here -v
will print only other than your pattern (that means invert match).
To get a inplace like result with grep
you can do this:
echo "$(grep -v "pattern" filename)" >filename
-
4
I have made a small benchmark with a file which contains approximately 345 000 lines. The way with grep
seems to be around 15 times faster than the sed
method in this case.
I have tried both with and without the setting LC_ALL=C, it does not seem change the timings significantly. The search string (CDGA_00004.pdbqt.gz.tar) is somewhere in the middle of the file.
Here are the commands and the timings:
time sed -i "/CDGA_00004.pdbqt.gz.tar/d" /tmp/input.txt
real 0m0.711s
user 0m0.179s
sys 0m0.530s
time perl -ni -e 'print unless /CDGA_00004.pdbqt.gz.tar/' /tmp/input.txt
real 0m0.105s
user 0m0.088s
sys 0m0.016s
time (grep -v CDGA_00004.pdbqt.gz.tar /tmp/input.txt > /tmp/input.tmp; mv /tmp/input.tmp /tmp/input.txt )
real 0m0.046s
user 0m0.014s
sys 0m0.019s
-
Which platform are you on? Which versions of sed/perl/grep do you use?– hagelloFeb 21, 2018 at 6:01
-
The platform I use is Linux (Gentoo). The sed version is GNU sed v 4.2.2, the perl version perl 5 (I cant tell which revision I used at the time of the test), and grep (GNU) is version 3.0.– JadziaFeb 21, 2018 at 11:56
Delete lines from all files that match the match
grep -rl 'text_to_search' . | xargs sed -i '/text_to_search/d'
You can also delete a range of lines in a file. For example to delete stored procedures in a SQL file.
sed '/CREATE PROCEDURE.*/,/END ;/d' sqllines.sql
This will remove all lines between CREATE PROCEDURE and END ;.
I have cleaned up many sql files withe this sed command.
perl -i -nle'/regexp/||print' file1 file2 file3
perl -i.bk -nle'/regexp/||print' file1 file2 file3
The first command edits the file(s) inplace (-i).
The second command does the same thing but keeps a copy or backup of the original file(s) by adding .bk to the file names (.bk can be changed to anything).
Curiously enough, the accepted answer does not actually answer the question directly. The question asks about using sed to replace a string, but the answer seems to presuppose knowledge of how to convert an arbitrary string into a regex.
Many programming language libraries have a function to perform such a transformation, e.g.
python: re.escape(STRING)
ruby: Regexp.escape(STRING)
java: Pattern.quote(STRING)
But how to do it on the command line?
Since this is a sed-oriented question, one approach would be to use sed itself:
sed 's/\([\[/({.*+^$?]\)/\\\1/g'
So given an arbitrary string $STRING we could write something like:
re=$(sed 's/\([\[({.*+^$?]\)/\\\1/g' <<< "$STRING")
sed "/$re/d" FILE
or as a one-liner:
sed "/$(sed 's/\([\[/({.*+^$?]\)/\\\1/g' <<< "$STRING")/d"
with variations as described elsewhere on this page.
I found most of the answers not useful for me, If you use vim I found this very easy and straightforward:
:g/<pattern>/d
-
if your comfortable with
vim
as opposed tosed
you can useex
stackoverflow.com/a/33186317 beware that it's slower. What's nice aboutvim
is you can\v<pattern>
to avoid backslashitis– CervEdJan 6, 2022 at 23:52
Just in case someone wants to do it for exact matches of strings, you can use the -w
flag in grep - w for whole. That is, for example if you want to delete the lines that have number 11, but keep the lines with number 111:
-bash-4.1$ head file
1
11
111
-bash-4.1$ grep -v "11" file
1
-bash-4.1$ grep -w -v "11" file
1
111
It also works with the -f
flag if you want to exclude several exact patterns at once. If "blacklist" is a file with several patterns on each line that you want to delete from "file":
grep -w -v -f blacklist file
-
1A bit misleading.
-w, --word-regexp Select only those lines containing matches that form whole words.
vs.-x, --line-regexp Select only those matches that exactly match the whole line. For a regular expression pattern, this is like parenthesizing the pattern and then surrounding it with ^ and $.
– SaiOct 23, 2017 at 13:34
to show the treated text in console
cat filename | sed '/text to remove/d'
to save treated text into a file
cat filename | sed '/text to remove/d' > newfile
to append treated text info an existing file
cat filename | sed '/text to remove/d' >> newfile
to treat already treated text, in this case remove more lines of what has been removed
cat filename | sed '/text to remove/d' | sed '/remove this too/d' | more
the | more
will show text in chunks of one page at a time.
-
1With -i you can edit the file in place:
sed -i '/text to remove/d'
filename is the same as catfilename | sed '/text to remove/d'
Jan 28 at 12:49
cat filename | grep -v "pattern" > filename.1
mv filename.1 filename
You can use good old ed
to edit a file in a similar fashion to the answer that uses ex
. The big difference in this case is that ed
takes its commands via standard input, not as command line arguments like ex
can. When using it in a script, the usual way to accomodate this is to use printf
to pipe commands to it:
printf "%s\n" "g/pattern/d" w | ed -s filename
or with a heredoc:
ed -s filename <<EOF
g/pattern/d
w
EOF
-
I think this is the first time I've seen the standard editor recommended as a solution.– EricNov 2 at 16:05
This solution is for doing the same operation on multiple file.
for file in *.txt; do grep -v "Matching Text" $file > temp_file.txt; mv temp_file.txt $file; done