69

grep can't be fed "raw" strings when used from the command-line, since some characters need to be escaped to not be treated as literals. For example:

$ grep '(hello|bye)' # WON'T MATCH 'hello'
$ grep '\(hello\|bye\)' # GOOD, BUT QUICKLY BECOMES UNREADABLE

I was using printf to auto-escape strings:

$ printf '%q' '(some|group)\n'
\(some\|group\)\\n

This produces a bash-escaped version of the string, and using backticks, this can easily be passed to a grep call:

$ grep `printf '%q' '(a|b|c)'`

However, it's clearly not meant for this: some characters in the output are not escaped, and some are unnecessarily so. For example:

$ printf '%q' '(^#)'
\(\^#\)

The ^ character should not be escaped when passed to grep.

Is there a cli tool that takes a raw string and returns a bash-escaped version of the string that can be directly used as pattern with grep? How can I achieve this in pure bash, if not?

7
  • 5
    It's really not clear what you want to escape and what you don't. If you want to look for literal strings, use grep -F.
    – jordanm
    Aug 8, 2012 at 0:40
  • I tried to make the question clearer, please take another look.
    – salezica
    Aug 8, 2012 at 0:54
  • I still don't understand. How are you getting the strings into the shell to printf them in the first place? By that point they're just as ready to be passed to printf as to grep.
    – ephemient
    Aug 8, 2012 at 0:58
  • I added an example at the beginning
    – salezica
    Aug 8, 2012 at 1:01
  • 1
    Related: Escape a string for a sed replace pattern
    – hakre
    Jun 14, 2014 at 8:21

6 Answers 6

74

If you want to search for an exact string,

grep -F '(some|group)\n' ...

-F tells grep to treat the pattern as is, with no interpretation as a regex.

(This is often available as fgrep as well.)

8
  • fgrep is defined by POSIX, so it should be available, but is technically deprecated.
    – jordanm
    Aug 8, 2012 at 0:42
  • I tried to make the question clearer, please take another look.
    – salezica
    Aug 8, 2012 at 0:58
  • 1
    @jordanm A bit harsher than deprecated, even. It's marked LEGACY in POSIX.2, and has not been carried forward to any specification past 1997.
    – ephemient
    Aug 8, 2012 at 1:02
  • 2
    That's totally not what I meant :(
    – salezica
    Aug 8, 2012 at 16:34
  • 1
    @LLL that's because the pattern -B- looks like an option (indeed -B is a valid grep option), you need to use the -- special option to terminate option parsing; i.e. echo A-B-C | grep -F -- -B- (note that the quotes you used are unnecessary and do nothing in this case, and they are stripped by the shell before invoking grep)
    – kbolino
    Apr 3, 2018 at 17:44
35

If you are attempting to get grep to use Extended Regular Expression syntax, the way to do that is to use grep -E (aka egrep). You should also know about grep -F (aka fgrep) and, in newer versions of GNU Coreutils, grep -P.

Background: The original grep had a fairly small set of regex operators; it was Ken Thompson's original regular expression implementation. A new version with an extended repertoire was developed later, and for compatibility reasons, got a different name. With GNU grep, there is only one binary, which understands the traditional, basic RE syntax if invoked as grep, and ERE if invoked as egrep. Some constructs from egrep are available in grep by using a backslash escape to introduce special meaning.

Subsequently, the Perl programming language has extended the formalism even further; this regex dialect seems to be what most newcomers erroneously expect grep, too, to support. With grep -P, it does; but this is not yet widely supported on all platforms.

So, in grep, the following characters have a special meaning: ^$[]*.\

In egrep, the following characters also have a special meaning: ()|+?{}. (The braces for repetition were not in the original egrep.) The grouping parentheses also enable backreferences with \1, \2, etc.

In many versions of grep, you can get the egrep behavior by putting a backslash before the egrep specials. There are also special sequences like \<\>.

In Perl, a huge number of additional escapes like \w \s \d were introduced. In Perl 5, the regex facility was substantially extended, with non-greedy matching *? +? etc, non-grouping parentheses (?:...), lookaheads, lookbehinds, etc.

... Having said that, if you really do want to convert egrep regular expressions to grep regular expressions without invoking any external process, try ${regex/pattern/substitution} for each of the egrep special characters; but recognize that this does not handle character classes, negated character classes, or backslash escapes correctly.

4
32

When I use grep -E with user provided strings I escape them with this

ere_quote() {
    sed 's/[][\.|$(){}?+*^]/\\&/g' <<< "$*"
}

example run

ere_quote ' \ $ [ ] ( ) { } | ^ . ? + *'
# output
# \\ \$ \[ \] \( \) \{ \} \| \^ \. \? \+ \*

This way you may safely insert the quoted string in your regular expression.

e.g. if you wanted to find each line starting with the user content, with the user providing funny strings as .*

userdata=".*"
grep -E -- "^$(ere_quote "$userdata")" <<< ".*hello"
# if you have colors in grep you'll see only ".*" in red
3
  • The character class/set in your ere_quote function is missing the "/" character, as a result it won't escape it.
    – fholzer
    Oct 21, 2020 at 8:21
  • @fholzer why should "/" be quoted? It has no special meaning in a regexp. If you are using it as delimiter you can use different delimiters, or escape it, but that is different from being evalutated as part of a regexp Oct 26, 2020 at 0:34
  • While technically correct, my assumption was that the output of the ere_quote function would subsequently used in e.g. sed. While one could use a different delimiter, whatever delimiter would be chosen would again need to be escaped. So, true, while technically the slash holds no special meaning in regex in general, it might be worth noting that when ere_quote output is used later in the script with certain tools, it would make sense to amend the character class with the chosen delimiter of those tools, if necessary.
    – fholzer
    Oct 26, 2020 at 12:39
7

I think that previous answers are not complete because they miss one important thing, namely string which begin with dash (-). So while this won't work:

echo "A-B-C" | grep -F "-B-"

This one will:

echo "A-B-C" | grep -F -- "-B-"
1
  • grep has the -e option precisely so you can unambiguously pass in a pattern which starts with a dash. This has nothing to do with regex syntax per se.
    – tripleee
    Oct 18, 2020 at 19:12
2
quote() {
    sed 's/[^\^]/[&]/g;s/[\^]/\\&/g' <<< "$*"
}

Usage: grep [OPTIONS] "$(quote [STRING])"

This function has some substantial benefits:

  • quote is independent from the regex flavor. You can use quote's output in
    • grep (-G)` (BRE, the default)
    • grep -E (ERE)
    • grep -P (PCRE)
    • sed (-E) "s/$(quote [STRING])/.../" (as long as you don't use \, [, or ] instead of /).
  • quote even works in corner cases that are not directly quoting related, for instance
    • Leading - are quoted so that they aren't misinterpreted as options by grep.
    • Trailing spaces are quoted so that the aren't removed by $(...).

quote only fails if [STRING] contains linebreaks. But in general there is no fix for this since tools like grep and sed may not support linebreaks in their search pattern (even if they are written as \n).

Also, there is the drawback that the quoted output usually is three times longer than the unquoted input.

0

Just want to comment example below which shows that substring "-B" is iterpreted by grep as a command line option and the command failed.

echo "A-B-C" | grep -F "-B-"

grep has a special option for this case:

-e PATTERNS, --regexp=PATTERNS Use PATTERNS as the patterns. If this option is used multiple times or is combined with the -f (--file) option, search for all patterns given. This option can be used to protect a pattern beginning with “-”.

So a fix for the issue is:

echo "A-B-C" | grep -F -e "-B-" -
1
  • This seems to basically repeat LLL's answer from 2017, albeit with a slightly different solution.
    – tripleee
    Feb 16, 2023 at 8:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.