46

I have a file called domain which contains some domains. For example:

google.com
facebook.com
...
yahoo.com

And I have another file called site which contains some sites URLs and numbers. For example:

image.google.com   10
map.google.com     8
...
photo.facebook.com  22
game.facebook.com   15
..

Now I'm going to count the url number each domain has. For example: google.com has 10+8. So I wrote an awk script like this:

BEGIN{
  while(getline dom < "./domain" > 0) {
    domain[dom]=0;
  }
  for(dom in domain) {
    while(getline < "./site" > 0) {
      if($1 ~/$dom$)   #if $1 end with $dom {
        domain[dom]+=$2;
      }
    }
  }
}

But the code if($1 ~/$dom$) doesn't run like I want. Because the variable $dom in the regular expression was explained literally. So, the first question is:

Is there any way to use variable $dom in a regular expression?

Then, as I'm new to writing script

Is there any better way to solve the problem I have?

5 Answers 5

58

awk can match against a variable if you don't use the // regex markers.

if ( $0 ~ regex ){ print $0; }

In this case, build up the required regex as a string

regex = dom"$"

Then match against the regex variable

if ( $1 ~ regex ) {
  domain[dom]+=$2;
}
26

First of all, the variable is dom not $dom -- consider $ as an operator to extract the value of the column number stored in the variable dom

Secondly, awk will not interpolate what's between // -- that is just a string in there.

You want the match() function where the 2nd argument can be a string that is treated as the regular expression:

if (match($1, dom "$")) {...}

I would code a solution like:

awk '
  FNR == NR {domain[$1] = 0; next}
  {
    for (dom in domain) {
      if (match($1, dom "$")) {
        domain[dom] += $2
        break
      }
    }
  }
  END {for (dom in domain) {print dom, domain[dom]}}
' domain site 
1
  • 1
    Apropos all those talking about how variables are not prefixed with $, it's better explained (IMO) as $ means field in awk, so $dom would mean field whatever the value of dom is. Variables in awk are used unquoted and without $. It's not shell! Commented Jan 4, 2018 at 9:47
2

One way using an awk script:

BEGIN {
    FS = "[. ]"
    OFS = "."
}

FNR == NR {
    domain[$1] = $0
    next
}

FNR < NR {
    if ($2 in domain) {
        for ( i = 2; i < NF; i++ ) {
            if ($i != "") {
                line = (line ? line OFS : "") $i
            }
        }
        total[line] += $NF
        line = ""
    }
}

END {
    for (i in total) {
        printf "%s\t%s\n", i, total[i]
    }
}

Run like:

awk -f script.awk domain.txt site.txt

Results:

facebook.com    37
google.com  18
2
  • This approach is not going to work if you get a domain like "first.second.example.com" in the site file. Commented Jul 18, 2012 at 16:00
  • @glennjackman, yes you are correct. I did not consider that :-(
    – Steve
    Commented Jul 18, 2012 at 22:59
1

You clearly want to read the site file once, not once per entry in domain. Fixing that, though, is trivial.

Equally, variables in awk (other than fields $0 .. $9, etc) are not prefixed with $. In particular, $dom is the field number identified by the variable dom (typically, that's going to be 0 since domain strings don't convert to any other number).

I think you need to find a way to get the domain from the data read from the site file. I'm not sure if you need to deal with sites with country domains such as bbc.co.uk as well as sites in the GTLDs (google.com etc). Assuming you are not dealing with country domains, you can use this:

BEGIN {
    while (getline dom < "./domain" > 0) domain[dom] = 0
    FS = "[ .]+"
    while (getline  < "./site" > 0)
    {
        topdom = $(NF-2) "." $(NF-1)
        domain[topdom] += $NF          
    }
    for (dom in domain) print dom "  " domain[dom]
}

In the second while loop, there are NF fields; $NF contains the count, and $1 .. $(NF-1) contain components of the domain. So, topdom ends up containing the top domain name, which is then used to index into the array initialized in the first loop.

Given the data in the question (minus the lines of dots), the output is:

yahoo.com  0
facebook.com  37
google.com  18
0

The problem of the answers above is that you cannot use the "metacharacters" (e.g. \< for a word boundary at the beginning of a word) if you use a string instead of a regular expression /.../. If you had a domain xyz.com and two sites ab.xyz.com and cd.prefix_xyz.com, the numbers of the two site entries would be added to xyz.com

Here's a solution using awk's pipe and the sed command: ...

for(dom in domain) {
    while(getline < "./site" > 0) {
        # let sed replaces occurence of the domain at the end of the site
        cmd = "echo '" $1 "' | sed 's/\\<'" dom "'$/NO_VALID_DOM/'"
        cmd | getline x
        close(cmd)
        if (match(x, "NO_VALID_DOM")) { 
          domain[dom]+=$2;
        }
    }
    close("./site") # this misses in original code
}

...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.