1
import pandas as pd 
import numpy as np
data=[]
columns = ['A', 'B', 'C']
data = [[0, 10, 5], [0, 12, 5], [2, 34, 13], [2, 3, 13], [4, 5, 8], [2, 4, 8], [1, 2, 4], [1, 3, 4], [3, 8, 12],[4,10,12],[6,7,12]]

df = pd.DataFrame(data, columns=columns)
print(df)

#     A   B   C
# 0   0  10   5
# 1   0  12   5
# 2   2  34  13
# 3   2   3  13
# 4   4   5   8
# 5   2   4   8
# 6   1   2   4
# 7   1   3   4
# 8   3   8  12
# 9   4  10  12
# 10  6   7  12

Now I want to create two data frames df_train and df_test such that no two numbers of column 'C' are in the same set. eg. in column C the element 5 should be either in the training set or testing set .So, the rows [0, 10, 5], [0, 12, 5], [2, 34, 13] will either go in training set or testing set but not in both.This choosing of elements of column C should be done randomly.

I am stuck on this step and cannot proceed.

2 Answers 2

1

First sample your df , then groupby C get the cumcount distinct the duplicated value within the same group.

s=df.sample(len(df)).groupby('C').cumcount()
s
Out[481]: 
5     0
7     0
2     0
1     0
0     1
6     1
10    0
4     1
3     1
8     1
9     2
dtype: int64
test=df.loc[s[s==1].index]
train=df.loc[s[s==0].index]
test
Out[483]: 
   A   B   C
0  0  10   5
6  1   2   4
4  4   5   8
3  2   3  13
8  3   8  12
train
Out[484]: 
    A   B   C
5   2   4   8
7   1   3   4
2   2  34  13
1   0  12   5
10  6   7  12
0

The question is not so clear of what the expected output of the two train and test set dataframe should looks like.

Anyway, I will try to answer.

I think you can first sort the dataframe values:

df_sorted = df.sort_values(['C'], ascending=[True])
print(df_sorted)

Out[1]:
    A   B   C
6   1   2   4
7   1   3   4
0   0  10   5
1   0  12   5
4   4   5   8
5   2   4   8
8   3   8  12
9   4  10  12
10  6   7  12
2   2  34  13
3   2   3  13

Then split the sorted dataframe:

uniqe_c = df_sorted['C'].unique().tolist()
print(uniqe_c)

Out[2]:
[4, 5, 8, 12, 13]

train_set = df[df['C'] <= uniqe_c[2]]
val_set = df[df['C'] > uniqe_c[2]]
print(train_set)

# Train set dataframe

Out[3]:
   A   B  C
0  0  10  5
1  0  12  5
4  4   5  8
5  2   4  8
6  1   2  4
7  1   3  4

print(val_set)

# Test set dataframe

Out[4]:
    A   B   C
2   2  34  13
3   2   3  13
8   3   8  12
9   4  10  12
10  6   7  12

From 11 samples, after the split, 6 samples go to the train set and 5 samples go to the validation set. So, checked and no missing samples in the total combined two dataframes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.