I hope someone with more knowledge than I can provide some wisdom before I pull all my hair out.
I have a dataframe which looks like this
Date Unit Length AM/PM unit_new
5 Monday\r13 January 12345H\rEngineering - Unit 1: Engineering Principles\r23456H\rHealth and Social Care - Unit 2: Working in Health\rand Social Care 2h 00m\r1h 30m morning
6 Tuesday\r14 January 34567H\rBusiness/Enterprise and Entrepreneurship -\rUnit 3: Personal and Business Finance\r12345L\rApplied Human Biology - Unit 1: Principles of\rHuman Biology\r23456K\rConstruction and the Built Environment -\rUnit 1: Construction Principles 2h 00m\r1h 30m\r1h 30m morning
7 Wednesday\r15 January 34567H/1C\rApplied Science/Forensic and Criminal Investigation\r- Unit 1: Principles and Applications of Science I -\rChemistry\r12345H\rSport and Exercise Science - Unit 1: Sport and Exercise\rPhysiology 0h 40m\r1h 30m morning
Now the problem I have it that the 'Unit' column has multiple records worth of data on each row, but it is not consistent in the number of records on a row. The 'length' column has the same setup as the 'Unit' column. The 'Date' and the 'AM/PM' columns have a single entry.
This image better explains the problem. Row 5 has two records one for Enginering and one for HSC, the length column follows suit. The 'Date' and 'AM/PM' are the same for both records. Line 6 has three records and line 7 has two.
Now what I am trying to do is split each record onto its own row. In trying to do this I have tried a number of different methods and not made much ground.
Method idea one My first thought was to try and add new row(s) under the relevant row and extract the data from 'Unit' and 'Length' columns while copying the data from 'Date' and 'AM/PM' columns. This proved to be trick as inseting into the middle of the df it more complicated.
Method idea two Next I thought to append rows to the bottom of the df and sort later.
So I wrote a function that did a count of the number of records on each row and output to a series.
def code_count_func():
code_count = df.Unit.str.count('\d{5}\w').subtract(+1)
# drop na's to stop error
code_count.dropna(inplace = True)
# converting to int
code_count = code_count.iloc[0:].astype(int)
The code below is what I am trying at the moment it splits into a list of strings in a new column called 'unit_new' but the regex is not quite capturing as per my colourful image.
for index, row in code_count_func().iteritems():
df['unit_new'] = df.Unit.str.split('(\d{5}\w)')
The second issue is that I also am not sure how to finsh the program. I was thinking using the DataFrame.explode
method but I am unsure how to use this on the 'Unit' and 'Length' columns but just copy from 'Date' and 'AM/PM' columns.
Can someone please give me some guidance on how to use use the explode method or similar. Also if anyone is able to help with my regex, please.
A little more on the regex issues. So the one thing that is consistent with the pattern in the 'Unit' column is the five digit number and one letter e.g. 12345K which is used at the start of each new record. So looking at row 5 I want to get this:-
12345H\rEngineering - Unit 1: Engineering Principles\r 23456H\rHealth and Social Care - Unit 2: Working in Health\rand Social Care
I have tried a number of patterns but no luck.
Required output