7

I have created a loop that generates some values. I want to store those values in a data frame. For example, completed one loop, append to the first row.

def calculate (allFiles):

    result = pd.DataFrame(columns = ['Date','Mid Ebb Total','Mid Flood Total','Mid Ebb Control','Mid Flood Control'])

    total_Mid_Ebb = 0
    total_Mid_Flood = 0
    total_Mid_EbbControl = 0
    total_Mid_FloodControl = 0

    for file_ in allFiles:
        xls = pd.ExcelFile(file_)
        df = xls.parse('General Impact')
        Mid_Ebb = df[df['Tidal Mode'] == "Mid-Ebb"] #filter 
        Mid_Ebb_control = df[df['Station'].isin(['C1','C2','C3'])] #filter control
        Mid_Flood = df[df['Tidal Mode'] == "Mid-Flood"] #filter
        Mid_Flood_control = df[df['Station'].isin(['C1','C2','C3', 'SR2'])] #filter control
        total_Mid_Ebb += Mid_Ebb.Station.nunique() #count unique stations = sample number
        total_Mid_Flood += Mid_Flood.Station.nunique()
        total_Mid_EbbControl += Mid_Ebb_control.Station.nunique()
        total_Mid_FloodControl += Mid_Flood_control.Station.nunique()

    Mid_Ebb_withoutControl = total_Mid_Ebb - total_Mid_EbbControl
    Mid_Flood_withoutControl = total_Mid_Flood - total_Mid_FloodControl

    print('Ebb Tide: The total number of sample is {}. Number of sample without control station is {}. Number of sample in control station is {}'.format(total_Mid_Ebb, Mid_Ebb_withoutControl, total_Mid_EbbControl))
    print('Flood Tide: The total number of sample is {}. Number of sample without control station is {}. Number of sample in control station is {}'.format(total_Mid_Flood, Mid_Flood_withoutControl, total_Mid_FloodControl))

The dataframe result contains 4 columns. The date is fixed. I would like to put total_Mid_Ebb, Mid_Ebb_withoutControl, total_Mid_EbbControl to the dataframe.

3 Answers 3

8

I believe you need append scalars in loop to list of tuples and then use DataFrame constructor. Last count differences in result DataFrame:

def calculate (allFiles):

    data = []
    for file_ in allFiles:
        xls = pd.ExcelFile(file_)
        df = xls.parse('General Impact')
        Mid_Ebb = df[df['Tidal Mode'] == "Mid-Ebb"] #filter 
        Mid_Ebb_control = df[df['Station'].isin(['C1','C2','C3'])] #filter control
        Mid_Flood = df[df['Tidal Mode'] == "Mid-Flood"] #filter
        Mid_Flood_control = df[df['Station'].isin(['C1','C2','C3', 'SR2'])] #filter control
        total_Mid_Ebb = Mid_Ebb.Station.nunique() #count unique stations = sample number
        total_Mid_Flood = Mid_Flood.Station.nunique()
        total_Mid_EbbControl = Mid_Ebb_control.Station.nunique()
        total_Mid_FloodControl = Mid_Flood_control.Station.nunique()
        data.append((total_Mid_Ebb, 
                     total_Mid_Flood, 
                     total_Mid_EbbControl, 
                     total_Mid_FloodControl))

    cols=['total_Mid_Ebb','total_Mid_Flood','total_Mid_EbbControl','total_Mid_FloodControl']

    result = pd.DataFrame(data, columns=cols)
    result['Mid_Ebb_withoutControl'] = result.total_Mid_Ebb - result.total_Mid_EbbControl
    result['Mid_Flood_withoutControl']=result.total_Mid_Flood-result.total_Mid_FloodControl

    #if want check all totals
    total = result.sum()
    print (total)


    return result
6

Here is an example of loading data per column in a dataframe after each iteration of a loop. While this is not THE only method, it's one that helps understand concept better.

Necessary imports

import pandas as pd
from random import randint

First define an empty data-frame of 5 columns to match your problem

df = pd.DataFrame(columns=['A','B','C','D','E'])

Next we iterate through for loop and generate value using randint() and add one value at a time to each column Staring with 'A' all the way to 'E',

for i in range(5): #add 5 rows of data
    df.loc[i, ['A']] = randint(0,99)
    df.loc[i, ['B']] = randint(0,99)
    df.loc[i, ['C']] = randint(0,99)
    df.loc[i, ['D']] = randint(0,99)
    df.loc[i, ['E']] = randint(0,99)

We get a DF whose 5 rows are populated.

>>> df
    A   B   C   D   E
0   4  74  71  37  90
1  41  80  77  81   8
2  14  16  82  98  89
3   1  77   3  56  91
4  34   9  85  44  19

Hope above helps and you are able to tailor to your needs.

0

Note this does not produce a row per file as requested, but it more of a comment about general use of Pandas for problems like this - it is often easier to read all the data then process using the pandas files than to write your own loops over different cases.

I think you are not using pandas in the idiomatic way here. I think you will save a lot of code and get a more understandable result if you do it this way:

controlstations = ['C1', 'C2', 'C3', 'SR2']
df = pd.concat(pd.read_excel(file_, sheetname='General Impact') for file_ in files)
df['Control'] = df.Station.isin(controlstations)
counts = df.groupby(['Control', 'Tidal Mode']).Station.agg('nunique')

So here you are reading all the excel files into a single dataframe first, then adding a column to indicate if that is a control station or not, then using groupby to count the different combinations.

counts is a series with a two-dimensional index (for some made up data):

Control  Tidal Mode
False    Mid-Ebb       2
         Mid-Flood     2
True     Mid-Ebb       2
         Mid-Flood     2

You can access the values you have in your function like this:

total_Mid_Ebb = counts['Mid-Ebb'].sum()
total_Mid_Ebb_Control = counts['Mid-Ebb', True]
total_Mid_Flood = counts['Mid-Flood'].sum()
total_Mid_Flood_Control = counts['Mid-Flood', True]

After which you can easily add them to a DataFrame:

import datetime
today = datetime.datetime.today()
totals = [total_Mid_Ebb, total_Mid_Flood, total_Mid_Ebb_Control, total_Mid_Flood_Control]
result = pd.DataFrame(data=[totals], columns=['Mid Ebb Total', 'Mid Flood Total', 'Mid Ebb Control', 'Mid Flood Control'],
                       index=[today])
4
  • 1
    Hmmm, I am not sure if your solution is good here, because first creates one big df with all data, then filter and last groupby and aggregate. But OP need for each file some filtering and counts, so better is loop each file, count and create row in DataFrame for each loop=for each file. And your solution dont do it. What do you think about it?
    – jezrael
    Jan 18, 2018 at 7:32
  • Unless you have so much data you can't fit it in memory, it makes sense to use the memory, doesn't it? I misunderstood the requirement to have a row for every file, but I think even then I would perhaps rather use an auxiliary column and groupby. Jan 18, 2018 at 7:39
  • Yes, but memory better is then my solution, because output for each file is only one tuple, so no big DataFrame is necessary. In my opinion generaly, if all data re necessary is better first concat and then processes like your solution, but here not - it is only overcomplicated, because need distinguish between each file and first concat and then split back and apply some counts for each file.
    – jezrael
    Jan 18, 2018 at 7:44
  • 1
    I agree your solution is more memory efficient. My issue was more with the manual filtering. I suppose if you need a row for every file my solution is not great, but I'm not sure that's really the big problem. I'll add a note to the solution to explain that it doesn't do a row per file. Jan 18, 2018 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.