0

I'm having a trouble with one script that I've made, time execution is too long (like 24 min), but the time is variable (depends of log's) and in the near future the time will increase for sure.

The troube is in a nesteed for loop:

obtener_ErroresLanzados()
{
        #Buscamos los equipos del log lanzados_a_pendientes en los logs de instala_sw_qcc para ver el porque no se han lanzado.
        totalLanzadosPendientes=`cat $rutaTemporales/lanzados_a_pendientes.log | wc -l`;
        lanzadosPendientes=$(cat $rutaTemporales/lanzados_a_pendientes.log);
        #grep "$paqueteBuscado" instala_sw_qcc_2012*.log | cut -f 1 -d ":" > $rutaTemporales/logsErrores.log;
        find $rutaTrazas -name "instala_sw_qcc_2012*" | xargs grep -l "$rutaQcc/$paqueteBuscado" | xargs grep -l "ERROR \[" | cut -f 9 -d "/" > $rutaTemporales/logsErrores.log;
        logsErrores=$(cat $rutaTemporales/logsErrores.log);
        totalLogsErrores=`cat $rutaTemporales/logsErrores.log | wc -l`;

for (( j=1; j<=$totalLanzadosPendientes; j++ ))
    do
equipoBusqueda=`echo $lanzadosPendientes | cut -f $j -d " "`;
            for (( k=1; k<=$totalLogsErrores; k++ ))
            do
            logBusqueda=`echo $logsErrores | cut -f $k -d " "`;
            grep "ERROR \[$equipoBusqueda\]" $rutaTrazas/$logBusqueda >> $rutaTemporales/erroresPendientes.log;
            if [ $? -eq 0 ];then
                    break;
            fi;
                    done;
    done;
    cat $rutaTemporales/erroresPendientes.log | sed 's/  / /g' | sed '/No se ha podido/d' | cut -f 7-14 -d " " | sort -u > $rutaTemporales/erroresPendientes_Final.log;
}

The problem is with $totalLogsErrores that is larger than 20k...

Can I do this by other way?

Thanks you!

----------------------- EDIT 1 -----------------------

$ time find $rutaTrazas -name "instala_sw_qcc_2012*" | xargs grep -l "$rutaQcc/$paqueteBuscado" | xargs grep -l "ERROR [" | cut -f 9 -d "/" 

real 0m3.862s
user 0m0.959s
sys 0m2.941s 

$ du -h ../trazas/instala_sw_qcc_20120718091838.log 

4.0K ../trazas/instala_sw_qcc_20120718091838.log 

$ time grep error ../trazas/instala_sw_qcc_20120718091838.log 

real 0m0.001s
user 0m0.001s
sys 0m0.000s
3
  • Please, try to explain what do you want to achieve, and give some example of input and expected output. It would be easier to understand your question.
    – igustin
    Jul 18, 2012 at 8:21
  • 1
    Can't you use something like a perl script? There are a lot of binary calls here which implies a lot of "fork" calls, which are expensive.
    – Arcadien
    Jul 18, 2012 at 8:29
  • The problem is that first for may have like 100 repetitions and inside this for there are other for with more than 10k of repetitions, that's is 100 x 10000 reps... Jul 18, 2012 at 8:38

4 Answers 4

0

Inefficient Processing with Loops

It's almost impossible to figure out what you're really trying to do here, as there's no corpus or sample output showing what you're actually trying to parse. However, you can boil this problem down to inefficient processing and process forking.

Use AWK for Record-Oriented Operations

Log files are generally record-oriented, where each line is a record with multiple fields. If this is your use case, then AWK (or Perl/Ruby in AWK emulation mode) is usually the right tool for the job. This ensures that you handle each line only once, and reading lines and splitting fields is extremely efficient.

For example, with Bash 4 and GNU awk (a.k.a. gawk):

shopt -s globstar
awk 'BEGIN {error_count = 0}
     /ERROR/ {print $9; ++error_count}
     # other pattern/action pairs
     END {print "Total errors:", error_count}
    ' **/instala_sw_qcc_2012* > /path/to/output/file

You can have multiple pattern matches that are applied to each line, or direct output to individual files from inside awk if you really need to do that. However, by letting awk handle the loops and the line parsing in one process, you are likely to gain a great deal of efficiency.

0

To qualify this performance issue, you can try the following:

Assess the impact of your first find & grep commands:

$ time find $rutaTrazas -name "instala_sw_qcc_2012*" | xargs grep -l "$rutaQcc/$paqueteBuscado" | xargs grep -l "ERROR \[" | cut -f 9 -d "/" > $rutaTemporales/logsErrores.log;

In your nested for loops, assess the impact the grep. How big are the files? In your comment you mentioned 100*10000 repetition, which is huge if each grep took 4 ms.

The find becomes expensive when you have a lot of sub-directories and the grep becomes expensive when your files are large enough.

$ du -h file.out 
  20K   file.out


$ time grep ERROR file.out 

real    0m0.004s
user    0m0.000s
sys     0m0.003s

if you have 1000000 loops, that will take a lot :)

0
0

Yo are reading the whole file into a shell variable and then extracting every line with a separate cut process. This is very inefficient.

It is difficult to decipher what are you trying to do. Maybe you could replace the function with something like this:

$ cd $rutaTrazas
$ sed 's/^/ERROR \[/; s/$/\]/' $rutaTemporales/lanzados_a_pendientes.log > search_strings
$ xargs grep -F -f search_strings \
    < $rutaTemporales/logsErrores.log \
    >> $rutaTemporales/erroresPendientes.log
$ < $rutaTemporales/erroresPendientes.log \
    sed 's/  / /g' | sed '/No se ha podido/d' |
    cut -f 7-14 -d " " | 
    sort -u > $rutaTemporales/erroresPendientes_Final.log
0

useless use of cat : wc -l <file instead of cat file | wc -l

useless use of wc : while read line; do ...;done <file instead of for loop :

find $rutaTrazas -name "instala_sw_qcc_2012*" | xargs grep -l "$rutaQcc/$paqueteBuscado" | xargs grep -l "ERROR \[" | cut -f 9 -d "/" > $rutaTemporales/logsErrores.log;
while read equipoBusqueda; do
    while read logBusqueda; do
        grep "ERROR \[$equipoBusqueda\]" $rutaTrazas/$logBusqueda >> $rutaTemporales/erroresPendientes.log
        if [ $? -eq 0 ];then
            break; 
        fi;
    done <$rutaTemporales/logsErrores.log
done <$rutaTemporales/lanzados_a_pendientes.log
cat $rutaTemporales/erroresPendientes.log | sed 's/  / /g' | sed '/No se ha podido/d' |
    cut -f 7-14 -d " " | sort -u > $rutaTemporales/erroresPendientes_Final.log;

finally find, grep, sed, cut, etc. commands may be simplified.

4
  • You missed removing a cat. I don't think you've answered the primary question - if you have, you need to point out where your answer saves time. You can do if grep... Jul 18, 2012 at 10:24
  • it solves the second part of the problem : logsErrores that is larger than 20k Jul 18, 2012 at 10:30
  • and maybe first part too : for 100 000 reps echo | cut, is very inefficient Jul 18, 2012 at 10:36
  • Tomorrow I will try it and post the result Jul 18, 2012 at 15:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.