2

I'm trying to achieve with pandas something that seems to be simple, but I'm stuck after several unglorious tests.

Here's the thing. I've got one Dataframe (let's call it streets) with only two series : streets by name and a gender related to them :

     name                             gender
0    Abraham Lincoln Avenue           undefined
1    Donald Trump Dead End            undefined
2    Hillary Clinton Street           undefined
...
1754 Ziggy Marley Boulevard           undefined

On the other hand, I've got an other Dataframe (let's call it fnames), very very huge. It has four series :

       gender   gender_detail  main_gender      first_name
0      F        Female         Female           Aaf
1      F        Female         Female           Aafke
2      F        Female         Female           Aafkea
3      M        Male           Male             Aafko
...
40211  F        Female         Female           Zyta

So like you've certainly guessed, I would to use 'first_name' serie of fnames to check if one of the first names appears or not in 'name' serie of streets.

If the first name is found, I update 'gender' serie in streets with related value of fnames' serie called 'gender'. If not, I let 'undefined'.

Obviously, I can't use two for loops because of Dataframes' size... Is there any quick solution to achieve that ?

For example, should I create a dictionnary with only first name as key and gender as value to be more efficient ?

PS : I don't know if it can simplify the issue but my two Dataframes are sorted by alphabetical order !

1 Answer 1

2

Yes, I think you can use dict with map of splitted column name by split by whitespace and selected first value by str[0], last replace NaN by fillna:

print (df1)
                        name     gender
0     Abraham Lincoln Avenue  undefined
1      Donald Trump Dead End  undefined
2     Hillary Clinton Street  undefined
3                 Aaf Street  undefined
1754  Ziggy Marley Boulevard  undefined

print (df2)
      gender gender_detail main_gender first_name
0          F        Female      Female        Aaf
1          F        Female      Female      Aafke
2          F        Female      Female     Aafkea
3          F        Female      Female      Aafko
40211      F        Female      Female       Zyta
d = df2.set_index('first_name')['gender'].to_dict()
print (d)
{'Zyta': 'F', 'Aaf': 'F', 'Aafkea': 'F', 'Aafke': 'F', 'Aafko': 'F'}

print (df1['name'].str.split().str[0])
0       Abraham
1        Donald
2       Hillary
3           Aaf
1754      Ziggy
Name: name, dtype: object

df1['gender'] = df1['name'].str.split().str[0].map(d).fillna('undefined')
print (df1)
                        name     gender
0     Abraham Lincoln Avenue  undefined
1      Donald Trump Dead End  undefined
2     Hillary Clinton Street  undefined
3                 Aaf Street          F
1754  Ziggy Marley Boulevard  undefined
3
  • Fantastic jezrael, just tested that and it worked perfectly fine ! Many thanks to you ! Feb 19, 2017 at 17:28
  • Hi @jezrael ! Two questions if you have a minute. I noticed that if the df1['name'] has only one part ("Mainstreet", for example) I've got automatically the first entry of d (and not 'undefined'). The second point concerns names in several part (like 'de Gaulle'). Do you think there is any method to update df1['name'] with these two special cases without erasing the previous work ? Thanks in advance, and have a nice week-end ;-) ! Feb 25, 2017 at 16:29
  • You can filter out all rows with one word by df1=df1[df1.name.str.split().str.len()!=1] how does it work? Second question is a bit problematic, solution is split by second whitespace with parameter n and replace by combine first df1['gender'] = df1['name'].str.split().str[0].map(d) twowordsname = df1['name'].str.split(n=1).str[0].map(d) df1['gender'] = df1['gender'].combine_first(twowordsname).fillna('undefined'). Now I am only on phone, so untested, please check it and if some problem let me know. Nice weekend.
    – jezrael
    Feb 25, 2017 at 19:23

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.