31

how to remove comment lines (as # bal bla ) and empty lines (lines without charecters) from file with one sed command?

THX lidia

1
  • I ran a test. paxdiablo vs Chris Johnson. The test file was squid.conf. squid.conf has 1000+ lines of comments for 15 lines of config. The paxdiablo method worked perfectly. It even deleted comments past a config on the same line. Results being exactly what you expect, if not better. Both thumbs up! The Chris Johnson improved method seemed to delete every line with a # regardless of where # was placed in the line. Resulting in lines being deleted that should not have been. Nice try Chris. A good motto I like to live by is "If is ain't broke, don't fix it". Dec 26, 2017 at 12:41

7 Answers 7

43

If you're worried about starting two sed processes in a pipeline for performance reasons, you probably shouldn't be, it's still very efficient. But based on your comment that you want to do in-place editing, you can still do that with distinct commands (sed commands rather than invocations of sed itself).

You can either use multiple -e arguments or separate commands with a semicolon, something like (just one of these, not both):

sed -i 's/#.*$//' -e '/^$/d' fileName
sed -i 's/#.*$//;/^$/d' fileName

The following transcript shows this in action:

pax> printf 'Line # with a comment\n\n# Line with only a comment\n' >file

pax> cat file
Line # with a comment

# Line with only a comment

pax> cp file filex ; sed -i 's/#.*$//;/^$/d' filex ; cat filex
Line

pax> cp file filex ; sed -i -e 's/#.*$//' -e '/^$/d' filex ; cat filex
Line

Note how the file is modified in-place even with two -e options. You can see that both commands are executed on each line. The line with a comment first has the comment removed then all is removed because it's empty.

In addition, the original empty line is also removed.

1
  • but if I want to use the sed -i ( how to integrated )
    – lidia
    Jul 28, 2010 at 5:57
17

@paxdiablo has a good answer but it can be improved.

(1) The '/^$/d' clause only matches 100% blank lines.

If you want to also match lines that are entirely whitespace (spaces, tabs etc.) use this instead:

'/^\s*$/d'

(2) The 's/#.*$//' clause only matches lines that start with the # character in column 0.

If you want to also match lines that have only whitespace before the first # use this instead:

'/^\s*#.*$/d'

The above criteria may not be universal (e.g. within a HEREDOC block, or in a Python multi-line string the different approaches could be significant), but in many cases the conventional definition of "blank" lines include whitespace-only, and "comment" lines include whitespace-then-#.

(3) Lastly, on OSX at least, the @paxdiablo solution in which the first clause turns comment lines into blank lines, and the second clause strips blank lines (including what were originally comments) doesn't work. It seems to be more portable to make both clauses /d delete actions as I've done.

The revised command incorporating the above is:

sed -e '/^\s*#.*$/d' -e '/^\s*$/d' inputFile
3
  • Is there a way to combine the two into one pattern? I tried many combinations but it does not work. So I want to know if there is a way, and if not, why? Thanks!
    – bruin
    Jan 14, 2017 at 15:13
  • Using #2 instead changes behaviour, as a line like this: not comment # comment, while previously turned into: not comment (with all spaces between not comment, and # left intact), now is completely untouched, and leaves the comments in file. What worked for me is: 's/\s*#.*$//'.
    – meeDamian
    Aug 25, 2019 at 20:12
  • 1
    @meeDamian be aware of what you're including -- your case doesn't work for input like: echo 'I need to echo "$ # and *"' | sed 's/\s*#.*$//' -- chops off in the middle of the quoted string, resulting in I need to echo "$. My regexes assume you don't have multi-line strings, HEREDOC blocks or the like with embedded # which may or may not be a good assumption, but will not change a line unless it's fully whitespace, or fully whitespace up to the first #. Yours chops off anything after whitespace-# even in a string. # is often used in format strings, so that seems pretty dangerous. Aug 25, 2019 at 21:32
7

This tiny jewel removes all # comments, no matter where they begin in a line (see caution below):

sed -e 's/\s*#.*$//'

Example:

text="
this is a # test
#this is a test
#this is a #test
this is # another #test
"

$echo "$text" | sed -e 's/\s*#.*$//'

this is a


this is

Next this removes any resulting blank lines:

$echo "$text" | sed -e 's/\s*#.*$//' | sed -e '/^\s*$/d'

Caution: Depending on the syntax and/or interpretation of the lines your processing, this might not be an appropriate solution, as it just stupidly removes end of lines, even if the '#' is part of your data or code. However, for use cases where you'll never use a hash except for as an end of line comment then it works fine. So just as with all coding, context must be taken into consideration.

4
  • Try that on an input file with a line like echo 'this is a hash symbol: #, my personal favorite.' && do_something_important. Jan 21, 2022 at 11:54
  • @Chris Johnson, Humm? Do you mean like this for example: $ echo 'this is a hash symbol: #, my personal favorite.' | sed -e 's/\s*#.*$//' && ls # a-second-comment ? Jan 22, 2022 at 15:20
  • No, what I’m saying is your approach will chop off everything after a # even when the symbol is not acting as a comment, and is therefore unsafe. Jan 22, 2022 at 15:25
  • @Chris Johnson, Right you are. Jan 22, 2022 at 17:01
3

Alternative variant, using grep:


cat file.txt | grep -Ev '(#.*$)|(^$)'
3
  • cat file.txt | grep -Ev '(^#.*$)|(^$)' to remove lines starting with "#"
    – demetrios
    Jul 28, 2010 at 6:09
  • This is very intuitive in terms of preservation of the original file. Sep 27, 2014 at 22:53
  • grep -v -E '(#.*$)|(^$)' file.txt -- Slay the useless cat antipattern!
    – ingyhere
    Aug 22, 2020 at 5:28
1

you can use awk

awk 'NF{gsub(/^[ \t]*#/,"");print}' file
1

First example(paxdiablo) is very good except its not change file, just output result. If you want to change it inline:

sudo sed -i 's/#.*$//;/^$/d' inputFile

0

On (one of) my linux boxes, sed understands extended regular expressions with the -r option, so:

sed -r '/(^\s*#)|(^\s*$)/d' squid.conf.installed

is very useful for showing all non-blank, non comment lines. The regex matches either start of line followed by zero or more spaces or tabs followed by either a hash or end of line, and deletes those matching lines from the input.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.