0

I am attempting to clean one file1.txt that contains always the same lines using file2.txt that contains a list of IP addresses I want to remove. The working script I have written I believe can be enhanced somehow to be faster in execution.

My script:

#!/bin/bash
IFS=$'\n'
for i in $(cat file1.txt); do
        for j in $(cat file2); do
                echo ${i} | grep -v ${j}
        done
done

I have tested the script with the following data set:

Amount of lines in file1.txt = 10,000
Amount of lines in file2.txt = 3

Scrit execution time: 
real    0m31.236s
user    0m0.820s
sys     0m6.816s

file1.txt content:

I3fSgGYBCBKtvxTb9EMz,1.1.2.3,45,This IP belongs to office space,1539760501,https://myoffice.com
I3fSgGYBCBKtvxTb9EMz,1.2.2.3,45,This IP belongs to office space,1539760502,https://myoffice.com
I3fSgGYBCBKtvxTb9EMz,1.3.2.3,45,This IP belongs to office space,1539760503,https://myoffice.com
I3fSgGYBCBKtvxTb9EMz,1.4.2.3,45,This IP belongs to office space,1539760504,https://myoffice.com
I3fSgGYBCBKtvxTb9EMz,1.5.2.3,45,This IP belongs to office space,1539760505,https://myoffice.com
... lots of other lines in the same format
I3fSgGYBCBKtvxTb9EMz,4.1.2.3,45,This IP belongs to office space,1539760501,https://myoffice.com

file2.txt content:

1.1.2.3
1.2.2.3
... lots of other IPs here
1.2.3.9

How can I improve those timings? I am confident that the files will grow over time. In my case I will run the script every hour from cron, therefore I would like to improve here.

0

1 Answer 1

3

You want to get rid of all lines in file1.txt that contains substrings which match file2.txt. to the rescue

grep -vFwf file2.txt file1.txt

The -w is need to avoid that 11.11.11.11 matches 111.11.11.111

-F, --fixed-strings, --fixed-regexp Interpret PATTERN as a list of fixed strings, separated by newlines, any of which is to be matched. (-F is specified by POSIX, --fixed-regexp is an obsoleted alias, please do not use it in new scripts.)

-f FILE, --file=FILE Obtain patterns from FILE, one per line. The empty file contains zero patterns and therefore matches nothing. (-f is specified by POSIX.)

-w, --word-regexp Select only those lines containing matches that form whole words. The test is that the matching substring must either be at the beginning of the line or preceded by a non-word constituent character. Similarly, it must be either at the end of the line or followed by a non-word constituent character. Word-constituent characters are letters, digits, and the underscore.

source: man grep


On a further note, here are a couple of pointers for your script:

  1. Don't use for loops to read files (http://mywiki.wooledge.org/DontReadLinesWithFor).
  2. Don't use cat (See How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?)
  3. Use quotes! (See Bash and Quotes)

This allows us to rewrite it as:

#!/bin/bash
while IFS=$'\n' read -r i; do
  while IFS=$'\n' read -r j; do
      echo "$i" | grep -v "$j"
  done < file2
done < file1

Now the problem is that you read file2 N times. Where N is the number of lines of file1. This is not really efficient. And luckily has the solution for us (see top).

3
  • Amazing, I was not aware of that!
    – creed
    Oct 17, 2018 at 8:11
  • @creed I have updated the command. Be aware of the -w flag, it is needed!
    – kvantour
    Oct 17, 2018 at 8:39
  • The while improvement is a great tip. I will stop using for.
    – creed
    Oct 17, 2018 at 12:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.