285

I'm running a grep to find any *.sql file that has the word select followed by the word customerName followed by the word from. This select statement can span many lines and can contain tabs and newlines.

I've tried a few variations on the following:

$ grep -liIr --include="*.sql" --exclude-dir="\.svn*" --regexp="select[a-zA-Z0-
9+\n\r]*customerName[a-zA-Z0-9+\n\r]*from"

This, however, just runs forever. Can anyone help me with the correct syntax please?

1
  • 6
    The grep you've indicated here runs forever because you have not specified any files to search at the end of the command... The '--include' is a filter of the files named and doesn't actually provide you any files to be filtered.
    – marklark
    Mar 4, 2013 at 17:01

3 Answers 3

627

Without the need to install the grep variant pcregrep, you can do a multiline search with grep.

$ grep -Pzo "(?s)^(\s*)\N*main.*?{.*?^\1}" *.c

Explanation:

-P activate perl-regexp for grep (a powerful extension of regular expressions)

-z Treat the input as a set of lines, each terminated by a zero byte (the ASCII NUL character) instead of a newline. That is, grep knows where the ends of the lines are, but sees the input as one big line. Beware this also adds a trailing NUL char if used with -o, see comments.

-o print only matching. Because we're using -z, the whole file is like a single big line, so if there is a match, the entire file would be printed; this way it won't do that.

In regexp:

(?s) activate PCRE_DOTALL, which means that . finds any character or newline

\N find anything except newline, even with PCRE_DOTALL activated

.*? find . in non-greedy mode, that is, stops as soon as possible.

^ find start of line

\1 backreference to the first group (\s*). This is a try to find the same indentation of method.

As you can imagine, this search prints the main method in a C (*.c) source file.

29
  • 16
    /bin/grep: The -P and -z options cannot be combined
    – Oli
    Sep 5, 2011 at 8:13
  • 9
    /bin/grep: PCRE does not support \L, \l, \N, \U, or \u
    – Oli
    Sep 5, 2011 at 8:14
  • 33
    -zo was enough for my multi-line needs, thanks! (upvoted.)
    – Szocske
    Oct 18, 2011 at 15:02
  • 21
    I recommend ''grep -Pazo'' instead of the unsafer ''-Pzo''. Explanation: the -z switch on non-ASCII files may trigger grep's "binary data" behaviour which changes the return values. Switch ''-a | --text'' prevents that.
    – rloth
    Jan 8, 2015 at 13:43
  • 11
    pipe the result to tr '\0' '\n' if you need the matches on separate lines!
    – t.animal
    Nov 26, 2018 at 16:09
214

I am not very good in grep. But your problem can be solved using AWK command. Just see

awk '/select/,/from/' *.sql

The above code will result from first occurence of select till first sequence of from. Now you need to verify whether returned statements are having customername or not. For this you can pipe the result. And can use awk or grep again.

2
8

Your fundamental problem is that grep works one line at a time - so it cannot find a SELECT statement spread across lines.

Your second problem is that the regex you are using doesn't deal with the complexity of what can appear between SELECT and FROM - in particular, it omits commas, full stops (periods) and blanks, but also quotes and anything that can be inside a quoted string.

I would likely go with a Perl-based solution, having Perl read 'paragraphs' at a time and applying a regex to that. The downside is having to deal with the recursive search - there are modules to do that, of course, including the core module File::Find.

In outline, for a single file:

$/ = "\n\n";    # Paragraphs

while (<>)
{
     if ($_ =~ m/SELECT.*customerName.*FROM/mi)
     {
         printf file name
         go to next file
     }
}

That needs to be wrapped into a sub that is then invoked by the methods of File::Find.

6
  • 2
    Grep does not work one line a time. It searches through the entire corpus for matches, and only when it finds a match does it go back to consider whether a newline is in the middle. That way, it doesn't have to scan through the corpus looking for new lines (which would slow it down significantly)
    – Squidly
    Nov 5, 2013 at 13:28
  • @MrBones: there's a chance that a modern implementaton of grep does as you say using mmap() to read the file into memory, but its mode of operation is defined by the POSIX specification for grep and it decidedly works in terms of lines. I'm not convinced though; if the file is multiple gigabytes, there is no need to memory map it all when you can simply read in a few kilobytes at a time (most files with lines have lines that are less than kilobytes long). Then there's JSON files, of course, but they're exceptional. Nov 5, 2013 at 14:02
  • 1
    It works in terms of lines, but it doesn't work one line at a time. There's not a loop doing some kind of (for line in lines: doesMatch(line)). It's more obvious when considering fgrep (fixed strings), and how boyer-moore works. mmap isn't really relevant
    – Squidly
    Nov 5, 2013 at 14:08
  • that's a lot of problems on top of the original problem, plus the problem of the regex! Aug 8, 2017 at 0:07
  • 1
    @Squidly Whether or not that's true does not change the fact that it considers a line at a time. How something is programmed doesn't equate to how it works does it?
    – Pryftan
    Sep 23, 2019 at 0:31

Not the answer you're looking for? Browse other questions tagged or ask your own question.