3
"DF","00000000@11111.COM","FLTINT1000130394756","26JUL2010","B2C","6799.2"
"Rail","00000.POO@GMAIL.COM","NR251764697478","24JUN2011","B2C","2025"
"DF","0000650000@YAHOO.COM","NF2513521438550","01JAN2013","B2C","6792"
"Bus","00009.GAURAV@GMAIL.COM","NU27012932319739","26JAN2013","B2C","800"
"Rail","0000.ANU@GMAIL.COM","NR251764697526","24JUN2011","B2C","595"
"Rail","0000MANNU@GMAIL.COM","NR251277005737","29OCT2011","B2C","957"
"Rail","0000PRANNOY0000@GMAIL.COM","NR251297862893","21NOV2011","B2C","212"
"DF","0000PRANNOY0000@YAHOO.CO.IN","NF251327485543","26JUN2011","B2C","17080"
"Rail","0000RAHUL@GMAIL.COM","NR2512012069809","25OCT2012","B2C","5731"
"DF","0000SS0@GMAIL.COM","NF251355775967","10MAY2011","B2C","2000"
"DF","0001HARISH@GMAIL.COM","NF251352240086","22DEC2010","B2C","4006"
"DF","0001HARISH@GMAIL.COM","NF251742087846","12DEC2010","B2C","1000"
"DF","0001HARISH@GMAIL.COM","NF252022031180","09DEC2010","B2C","3439"
"Rail","000AYUSH@GMAIL.COM","NR2151120122283","25JAN2013","B2C","136"
"Rail","000AYUSH@GMAIL.COM","NR2151213260036","28NOV2012","B2C","41"
"Rail","000AYUSH@GMAIL.COM","NR2151313264432","29NOV2012","B2C","96"
"Rail","000AYUSH@GMAIL.COM","NR2151413266728","29NOV2012","B2C","96"
"Rail","000AYUSH@GMAIL.COM","NR2512912359037","08DEC2012","B2C","96"
"Rail","000AYUSH@GMAIL.COM","NR2517612385569","12DEC2012","B2C","96"

Above is the sample data. Data is sorted according to email addresses and the file is very large around 1.5Gb

I want output in another csv file something like this

"DF","00000000@11111.COM","FLTINT1000130394756","26JUL2010","B2C","6799.2",1,0 days
"Rail","00000.POO@GMAIL.COM","NR251764697478","24JUN2011","B2C","2025",1,0 days
"DF","0000650000@YAHOO.COM","NF2513521438550","01JAN2013","B2C","6792",1,0 days
"Bus","00009.GAURAV@GMAIL.COM","NU27012932319739","26JAN2013","B2C","800",1,0 days
"Rail","0000.ANU@GMAIL.COM","NR251764697526","24JUN2011","B2C","595",1,0 days
"Rail","0000MANNU@GMAIL.COM","NR251277005737","29OCT2011","B2C","957",1,0 days
"Rail","0000PRANNOY0000@GMAIL.COM","NR251297862893","21NOV2011","B2C","212",1,0 days
"DF","0000PRANNOY0000@YAHOO.CO.IN","NF251327485543","26JUN2011","B2C","17080",1,0 days
"Rail","0000RAHUL@GMAIL.COM","NR2512012069809","25OCT2012","B2C","5731",1,0 days
"DF","0000SS0@GMAIL.COM","NF251355775967","10MAY2011","B2C","2000",1,0 days
"DF","0001HARISH@GMAIL.COM","NF251352240086","09DEC2010","B2C","4006",1,0 days
"DF","0001HARISH@GMAIL.COM","NF251742087846","12DEC2010","B2C","1000",2,3 days
"DF","0001HARISH@GMAIL.COM","NF252022031180","22DEC2010","B2C","3439",3,10 days
"Rail","000AYUSH@GMAIL.COM","NR2151213260036","28NOV2012","B2C","41",1,0 days
"Rail","000AYUSH@GMAIL.COM","NR2151313264432","29NOV2012","B2C","96",2,1 days
"Rail","000AYUSH@GMAIL.COM","NR2151413266728","29NOV2012","B2C","96",3,0 days
"Rail","000AYUSH@GMAIL.COM","NR2512912359037","08DEC2012","B2C","96",4,9 days
"Rail","000AYUSH@GMAIL.COM","NR2512912359037","08DEC2012","B2C","96",5,0 days
"Rail","000AYUSH@GMAIL.COM","NR2517612385569","12DEC2012","B2C","96",6,4 days
"Rail","000AYUSH@GMAIL.COM","NR2517612385569","12DEC2012","B2C","96",7,0 days
"Rail","000AYUSH@GMAIL.COM","NR2151120122283","25JAN2013","B2C","136",8,44 days
"Rail","000AYUSH@GMAIL.COM","NR2151120122283","25JAN2013","B2C","136",9,0 days

i.e if entry occurs 1st time i need to append 1 if it occurs 2nd time i need to append 2 and likewise i mean i need to count no of occurences of an email address in the file and if an email exists twice or more i want difference among dates and remember dates are not sorted so we have to sort them also against a particular email address and i am looking for a solution in python using numpy or pandas library or any other library that can handle this type of huge data without giving out of bound memory exception i have dual core processor with centos 6.3 and having ram of 4GB

2
  • 7
    Put them in a database. Sort by name, then by date. Apr 19, 2013 at 17:26
  • it sounds like something to do following the Map-Reduce approach
    – Manolo
    Apr 19, 2013 at 17:46

3 Answers 3

8

make sure you have 0.11, read these docs: http://pandas.pydata.org/pandas-docs/dev/io.html#hdf5-pytables, and these recipes: http://pandas.pydata.org/pandas-docs/dev/cookbook.html#hdfstore (esp the 'merging on millions of rows'

Here is a solution that seems to work. Here is the workflow:

  1. read data from your csv by chunks and appending to an hdfstore
  2. iterate over the store, which creates another store that does the combiner

Essentially we are taking a chunk from the table and combining with a chunk from every other part of the file. The combiner function does not reduce, but instead calculates your function (the diff in days) between all elements in that chunk, eliminating duplicates as you go, and taking the latest data after each loop. Kind of like a recursive reduce almost.

This should be O(num_of_chunks**2) memory and calculation time chunksize could be say 1m (or more) in your case

processing [0] [datastore.h5]
processing [1] [datastore_0.h5]
    count                date  diff                        email
4       1 2011-06-24 00:00:00     0           0000.ANU@GMAIL.COM
1       1 2011-06-24 00:00:00     0          00000.POO@GMAIL.COM
0       1 2010-07-26 00:00:00     0           00000000@11111.COM
2       1 2013-01-01 00:00:00     0         0000650000@YAHOO.COM
3       1 2013-01-26 00:00:00     0       00009.GAURAV@GMAIL.COM
5       1 2011-10-29 00:00:00     0          0000MANNU@GMAIL.COM
6       1 2011-11-21 00:00:00     0    0000PRANNOY0000@GMAIL.COM
7       1 2011-06-26 00:00:00     0  0000PRANNOY0000@YAHOO.CO.IN
8       1 2012-10-25 00:00:00     0          0000RAHUL@GMAIL.COM
9       1 2011-05-10 00:00:00     0            0000SS0@GMAIL.COM
12      1 2010-12-09 00:00:00     0         0001HARISH@GMAIL.COM
11      2 2010-12-12 00:00:00     3         0001HARISH@GMAIL.COM
10      3 2010-12-22 00:00:00    13         0001HARISH@GMAIL.COM
14      1 2012-11-28 00:00:00     0           000AYUSH@GMAIL.COM
15      2 2012-11-29 00:00:00     1           000AYUSH@GMAIL.COM
17      3 2012-12-08 00:00:00    10           000AYUSH@GMAIL.COM
18      4 2012-12-12 00:00:00    14           000AYUSH@GMAIL.COM
13      5 2013-01-25 00:00:00    58           000AYUSH@GMAIL.COM
import pandas as pd
import StringIO
import numpy as np
from time import strptime
from datetime import datetime

# your data
data = """
"DF","00000000@11111.COM","FLTINT1000130394756","26JUL2010","B2C","6799.2"
"Rail","00000.POO@GMAIL.COM","NR251764697478","24JUN2011","B2C","2025"
"DF","0000650000@YAHOO.COM","NF2513521438550","01JAN2013","B2C","6792"
"Bus","00009.GAURAV@GMAIL.COM","NU27012932319739","26JAN2013","B2C","800"
"Rail","0000.ANU@GMAIL.COM","NR251764697526","24JUN2011","B2C","595"
"Rail","0000MANNU@GMAIL.COM","NR251277005737","29OCT2011","B2C","957"
"Rail","0000PRANNOY0000@GMAIL.COM","NR251297862893","21NOV2011","B2C","212"
"DF","0000PRANNOY0000@YAHOO.CO.IN","NF251327485543","26JUN2011","B2C","17080"
"Rail","0000RAHUL@GMAIL.COM","NR2512012069809","25OCT2012","B2C","5731"
"DF","0000SS0@GMAIL.COM","NF251355775967","10MAY2011","B2C","2000"
"DF","0001HARISH@GMAIL.COM","NF251352240086","22DEC2010","B2C","4006"
"DF","0001HARISH@GMAIL.COM","NF251742087846","12DEC2010","B2C","1000"
"DF","0001HARISH@GMAIL.COM","NF252022031180","09DEC2010","B2C","3439"
"Rail","000AYUSH@GMAIL.COM","NR2151120122283","25JAN2013","B2C","136"
"Rail","000AYUSH@GMAIL.COM","NR2151213260036","28NOV2012","B2C","41"
"Rail","000AYUSH@GMAIL.COM","NR2151313264432","29NOV2012","B2C","96"
"Rail","000AYUSH@GMAIL.COM","NR2151413266728","29NOV2012","B2C","96"
"Rail","000AYUSH@GMAIL.COM","NR2512912359037","08DEC2012","B2C","96"
"Rail","000AYUSH@GMAIL.COM","NR2517612385569","12DEC2012","B2C","96"
"""


# read in and create the store
data_store_file = 'datastore.h5'
store = pd.HDFStore(data_store_file,'w')

def dp(x, **kwargs):
    return [ datetime(*strptime(v,'%d%b%Y')[0:3]) for v in x ]

chunksize=5
reader = pd.read_csv(StringIO.StringIO(data),names=['x1','email','x2','date','x3','x4'],
                     header=0,usecols=['email','date'],parse_dates=['date'],
                     date_parser=dp, chunksize=chunksize)

for i, chunk in enumerate(reader):
    chunk['indexer'] = chunk.index + i*chunksize

    # create the global index, and keep it in the frame too
    df = chunk.set_index('indexer')

    # need to set a minimum size for the email column
    store.append('data',df,min_itemsize={'email' : 100})

store.close()

# define the combiner function
def combiner(x):

    # given a group of emails (the same), return a combination
    # with the new data

    # sort by the date
    y = x.sort('date')

    # calc the diff in days (an integer)
    y['diff'] = (y['date']-y['date'].iloc[0]).apply(lambda d: float(d.item().days))
    y['count'] = pd.Series(range(1,len(y)+1),index=y.index,dtype='float64')  
    
    return y

# reduce the store (and create a new one by chunks)
in_store_file = data_store_file
in_store1 = pd.HDFStore(in_store_file)

# iter on the store 1
for chunki, df1 in enumerate(in_store1.select('data',chunksize=2*chunksize)):
    print "processing [%s] [%s]" % (chunki,in_store_file)

    out_store_file = 'datastore_%s.h5' % chunki
    out_store = pd.HDFStore(out_store_file,'w')

    # iter on store 2
    in_store2 = pd.HDFStore(in_store_file)
    for df2 in in_store2.select('data',chunksize=chunksize):

        # concat & drop dups
        df = pd.concat([df1,df2]).drop_duplicates(['email','date'])

        # group and combine
        result = df.groupby('email').apply(combiner)
            
        # remove the mi (that we created in the groupby)
        result = result.reset_index('email',drop=True)
            
        # only store those rows which are in df2!
        result = result.reindex(index=df2.index).dropna()

        # store to the out_store
        out_store.append('data',result,min_itemsize={'email' : 100})
    in_store2.close()
    out_store.close()
    in_store_file = out_store_file

in_store1.close()

# show the reduced store
print pd.read_hdf(out_store_file,'data').sort(['email','diff'])
2
  • My first thought was also to put data in database but it wont do the trick i need to track 1st occurence of each email address also and count too and what about getting the diff of dates whats the soln for that?? @Jeff My file has more than 20million rows and unique values are turned out to be more than 5million if i start comparing those i will got complexity of 5million*20million which can take months to solve the problem how can i reduce comlexity from n*n so that i can handle such large amounts of volume
    – Geek
    Apr 20, 2013 at 4:37
  • so the first time an email appears its email is then the reference date for subsequent appearances? for the 2nd email it's easy, count is 1 and days is diff of days, what about 3rd email. does days get updated to be the diff between 3rd date and 1st date or is the number of days somehow involved (maybe the 3rd days is max of current and 3rd date - reference date?)
    – Jeff
    Apr 20, 2013 at 11:37
7

Use the built-in sqlite3 database: you can insert the data, sort and group as necessary, and there's no problem using a file which is larger than available RAM.

4

Another possible (system-admin) way, avoiding database and SQL queries plus a whole lot of requirements in runtime processes and hardware resources.

Update 20/04 Added more code and simplified approach:-

  1. Convert the timestamp to seconds (from Epoch) and use UNIX sort, using email and this new field (that is: sort -k2 -k4 -n -t, < converted_input_file > output_file)
  2. Initialize 3 variable, EMAIL, PREV_TIME and COUNT
  3. Interate over each line, if new email is encountered, add "1,0 day". Update PREV_TIME=timestamp, COUNT=1, EMAIL=new_email
  4. Next line: 3 possible scenario
    • a) if same email, different timestamp: calculate days, increment COUNT=1, update PREV_TIME, add "Count, Difference_in_days"
    • b) If same email, same timestamp: increment COUNT, add "COUNT, 0 day"
    • c) If new email, start from 3.

Alternative to 1. is to add a new field TIMESTAMP and remove it upon printing out the line.

Note: If 1.5GB is too huge to sort at a go, split it into smaller chuck, using email as the split point. You can run these chunks in parallel on different machine

/usr/bin/gawk -F'","' ' { 
    split("JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC", month, " "); 
    for (i=1; i<=12; i++) mdigit[month[i]]=i; 
    print $0 "," mktime(substr($4,6,4) " " mdigit[substr($4,3,3)] " " substr($4,1,2) " 00 00 00"
)}' < input.txt |  /usr/bin/sort -k2 -k7 -n -t, > output_file.txt

output_file.txt:

"DF","00000000@11111.COM","FLTINT1000130394756","26JUL2010","B2C","6799.2",1280102400 "DF","0001HARISH@GMAIL.COM","NF252022031180","09DEC2010","B2C","3439",1291852800 "DF","0001HARISH@GMAIL.COM","NF251742087846","12DEC2010","B2C","1000",1292112000 "DF","0001HARISH@GMAIL.COM","NF251352240086","22DEC2010","B2C","4006",1292976000
...

You pipe the output to Perl, Python or AWK script to process step 2. through 4.

5
  • This gawk script performs the COUNT and DAYS calculation on my sorted output
    – Alvin K.
    Apr 20, 2013 at 23:08
  • It worked out thanks man :)! I have applied split according to , only [root@amanka Desktop]# gawk 'BEGIN{ OFS = ","; COUNT = 0; PREV_TIME=0; EMAIL=0; while(( getline line<"aven.csv") > 0 ) { split(line, a , ",") if (EMAIL != a[2]) { EMAIL = a[2]; COUNT = 1; PREV_TIME = a[7]; print line, "1,0 day" } else { if (PREV_TIME == a[7]) { COUNT = COUNT + 1; print line, COUNT, "0 day"; } else { DAYS = ((a[7] - PREV_TIME)/(60*60*24)); PREV_TIME = a[7]; COUNT = COUNT + 1; print line, COUNT, DAYS " days"; } } } }'
    – Geek
    Apr 21, 2013 at 16:52
  • You're welcome. I'd be curious to know 1) How much memory did gawk+sort use? and 2) How much time it took on the 1.5gb file?
    – Alvin K.
    Apr 21, 2013 at 21:39
  • I am not sure about memory but it realy took very less time around 10-12 min to give the output sorting takes around 15 mins and it was unbelievably faster than any other language solution i am thinking of Even single time parsing of file O(n) in python takes around 35 mins Which is reduced to half using shell script
    – Geek
    Apr 24, 2013 at 8:40
  • and i have more problem regarding the same file it would be great if you can help me out in that you can chk question at stackoverflow.com/questions/16186224/…
    – Geek
    Apr 24, 2013 at 8:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.