Reminder: the formula for entropy is
H(S)=-sum[ P(Xi) * log2 P(Xi) ]
, where
S
is a content you want to calculates it's entropy,
Xi
is i-th
character in the document, and
P(Xi)
is a probability of seeing the character Xi
in the content.
The first problem here is to estimate correctly P(Xi)
. To do it correctly you need to download as many diverse pages as you can. At very least 100, several thousands would be better. This is important, because you need to have a real pages that represent well your domain.
Now, you have to reconstruct HTTP layer from the packets. It is not easy task in real life, because some pages will be split across several packets, and their order of arrival may be not be as you expect, and some packets might be lost and retransmitted. I recommend you to read this blog, to get the grip on the subj.
Also, I suggest you calculate entropy for headers and body of HTTP requests separately. This is because I expect that distribution of characters in the header and the body to be different.
Now, when you have an access to the desired content you just count frequencies of each character. Something like following (doc_collection
might a list composed of content of all HTTP headers you have extracted from your PCAPs.):
def estimate_probabilities(doc_collection):
freq = Counter()
for doc in doc_collection:
freq.update(Counter(doc))
total = 1.0*sum(freq.values())
P = { k : freq[k]/total for k in freq.keys() }
return P
Now that you have the probabilities of the characters, calculating entropy is simple:
import numpy as np
def entropy(s, P):
epsilon = 1e-8
sum = 0
for k,v in Counter(s).iteritems():
sum -= v*P[k]*np.log2(P[k] + epsilon)
return sum
If you like, you can even speed it up using map
:
import numpy as np
def entropy(s, P):
epsilon = 1e-8
return -sum(map(lambda a: a[1] * P[a[0]] * np.log2(P[a[0]] + epsilon), Counter(s).items()))
epsilon
is needed to prevent from logarithm to go to minus infinity, if the probability of a symbol is close to zero.
Now, if you want to calculate entropy excluding some characters ("\r" and "\n" in your case) just zero their probabilities, e.g. P['\n'] = 0
That will remove all those characters from the count.
-- updated to answer the comment:
If you want to sum the entropy depending on existence of the substring, your program will look like:
....
P = estimate_probabilities(all_HTTP_headers_list)
....
count_with, count_without = 0, 0
H = entropy(s, P)
if '\r\n\r\n' in s:
count_with += H
else:
count_without += H
all_HTTP_headers_list
is a concatenation of all the headers you have, s
is the specific header.
-- update2: how to read HTTP headers
pyshark
is not the best solution for packet manipulation, because it drops the payload, but it is ok to just get the headers.
pkts = pyshark.FileCapture('dump.pcap')
headers = []
for pk in pkts:
if pk.highest_layer == 'HTTP':
raw = pk.tcp.payload.split(':')
headers.append( ''.join([ chr(int(ch, 16)) for ch in raw ]) )
Here you check that your packet actually has HTTP layer, get its payload (from the TCP layer as ':' separated string), then do some string manipulations and at the end receive all HTTP headers from the PCAP as a list.
pyshark
but if you can capture the HTTP requests, you can just do a Regular Expression and have two counters:count_with
andcount_without
. If "\r\n\r\n" is in the string, incrementcount_with
; else incrementcount_without
.Ti
in the formula is the probability of seeing a certain character. And you can not calculate entropy of number of packets. But you can calculate entropy of each specific page. First, you have to find the probabilities of each symbol. In order to do that, you need to estimate those probabilities. That means downloading as many pages as you can (several thousands would be enough, to represent well the statistics), and running your code on all the data.