1

my data has a column of cities where a same city has been written differently. For example the city 'Bangalore' (correct spelling) has been spelled as 'bangalore','bengaluru','banglore' etc. I want to make all these different spelling into one correct spelling. Till now I have been doing it manually searching for the different spellings and replacing it with the correct.

Can anyone help me to get this process done faster and less of manual work using R?

Thank You

1
  • Using a variety of passes might help you. Even something as simple as removing vowels can be helpful for grouping: gsub("[aeiouy]", "", c("bangalore", "bengaluru", "banglore")) for instance gives "bnglr" for all 3 variations. Commented Mar 7, 2018 at 5:51

2 Answers 2

4

The phonics library, which has soundex capabilities, might be helpful here. Assuming that Bangalore is the correct spelling, you could compare other city names, possibly misspelled, against this reference:

library(phonics)
x <- 'Bangalore'
y <- 'Banglore'

if (soundex(x) == soundex(y) & x != y) {
    # handle misspelling for Bangalore
}

The above logic just says that if Bangalore and Banglore sound the same, but yet are spelled differently, then flag the pair for inspection.

1
  • 1
    Just to add, there's a whole bunch of phonetic coders in the phonics package - including nysiis() and friends, which might work well. Commented Mar 7, 2018 at 5:47
3

Below an example using stringdist which can calculate similarity between strings.

library(dplyr)
library(stringdist)

Generate some sample data:

cities <- data.frame(city = c('bangalore','bengaluru','banglore',
  'bangalore', 'bangalore', 'bangalore', 'new york', 'newyork', 
  'nyork', 'new york', 'new york'))

Determine unique values and calculate how often these occur (the most frequent one will probably be the correct one)

dta <- cities %>% group_by(city) %>% count() %>% 
  ungroup() %>% mutate(i = row_number())

For each combination of city names calculate a simularity, put most similar ones on top

pairs <- expand.grid(x = seq_len(nrow(dta)), y = seq_len(nrow(dta))) %>% 
  # Only need to compare i to  all records j, with j > i 
  filter(y > x) %>%
  left_join(dta, by = c(x = 'i')) %>% rename(cityx = city, nx = n) %>%
  left_join(dta, by = c(y = 'i')) %>% rename(cityy = city, ny = n) %>%
  mutate(similarity = stringsim(cityx, cityy, method = "jw")) %>% 
  arrange(desc(similarity))

You can then investigate top records:

> pairs
#    x y     cityx nx     cityy ny similarity
# 1  1 2 bangalore  4  banglore  1  0.9629630
# 2  4 5  new york  3   newyork  1  0.9583333
# 3  5 6   newyork  1     nyork  1  0.9047619
# 4  4 6  new york  3     nyork  1  0.8750000
# 5  2 3  banglore  1 bengaluru  1  0.7222222
# 6  1 3 bangalore  4 bengaluru  1  0.6944444
# 7  2 6  banglore  1     nyork  1  0.6583333
# 8  2 5  banglore  1   newyork  1  0.6011905
# 9  1 5 bangalore  4   newyork  1  0.5873016
# 10 2 4  banglore  1  new york  3  0.5833333
# 11 1 4 bangalore  4  new york  3  0.5694444
# 12 3 5 bengaluru  1   newyork  1  0.4761905
# 13 3 4 bengaluru  1  new york  3  0.4583333
# 14 1 6 bangalore  4     nyork  1  0.4370370
# 15 3 6 bengaluru  1     nyork  1  0.4370370

You could try different methods in stringsim. Jaro-Winkler has the advantage that it is relatively fast for large data sets.

Phonetic coders are sometimes too rough (quiet different strings can have the same phonetic encoding; especially soundex) and are usually developed specific for (American) last names.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.