Skip to main content
Filter by
Sorted by
Tagged with
-1 votes
0 answers
38 views

Optimizing Data Processing for Product Comparison Script (Python) [closed]

I have a Python script that scrapes product data from multiple websites and performs various processing steps before saving the results as a comparison table (CSV and Excel). However, the data ...
eran's user avatar
  • 9
1 vote
0 answers
49 views

Trying to fix names in my database with fuzzywuzzy

What I'm trying to do is find and correct similar names in my database, like 'Patrick Maxwell' and 'Patrick Maxwel.' However, the issue I'm facing is that the best match for each name is often itself, ...
Kauan Randall Oliveira Ferreir's user avatar
0 votes
0 answers
39 views

Fuzzy Match 2 Large Pandas Dataframes

I have 2 pandas dataframes that both contain company names. I want to left join df1(~10k rows) with df2(~1.6m rows) on company names using a fuzzy match. My current function takes too long to run, so ...
L H's user avatar
  • 27
0 votes
1 answer
50 views

PySpark FuzzyWuzzy UDF Causing Timeout Errors on Small Dataset / TimeoutError when filtering columns with Fuzzy Wuzzy similarity scores in PySpark

I'm working on a PySpark script to calculate similarity scores between columns using FuzzyWuzzy. I've defined a UDF for this purpose and used a for loop to iterate over columns specified in a metadata ...
Vanith C's user avatar
1 vote
1 answer
41 views

Why doesn't fuzzywuzzy's process.extractBests give a 100% score when the tested string 100% contains the query string?

I'm testing fuzzywuzzy's process.extractBests() as follows: from fuzzywuzzy import process # Define the query string query = "Apple" # Define the list of choices choices = ["Apple&...
Franck Dernoncourt's user avatar
0 votes
0 answers
44 views

Is there a way to use fuzzy selection to narrow the choices in a wx.ComboBox (wxWidgets)

I want to populate a wx.ComboBox with a large number of items but rather than having to scroll through them all, I'd like to type a string into the ComboBox and narrow the items in the ComboBox using ...
wef's user avatar
  • 361
0 votes
1 answer
145 views

How to do fuzzy merge with 2 large pandas dataframes?

I have 2 pandas dataframes that both contain company names. I want to merge these 2 dataframes on company names using a fuzzy match. But the problem is 1 dataframe contains 5m rows and the other 1 ...
L H's user avatar
  • 27
0 votes
0 answers
192 views

Fuzzy match on large dataset- speed up and keep similar matches

I have two beer datasets; one is ~3 million entries, and another is 175 thousand entries. Doing a fuzzy match on these two will take way too long. I've run a few tests on the same 1000 random sample ...
nick kalra's user avatar
0 votes
2 answers
103 views

Fuzzy Logic to match the records in a dataframe

I have a huge datasets of 2 Million and i want to match the records based on the fuzzy logic I have my original dataframe like +---------+---------------+ | name| address| +---------+-------...
arbind Sah's user avatar
0 votes
0 answers
59 views

Optimum way to get a single best unique match of two pandas dataframe columns, based on a set of parameters present in both dataframes

I would like to fuzzy match two Pandas dataframes based on a number n parameters in columns, all present in both dataframes. The output should be a pair of unique filenames. The parameter sets might ...
Sebastian Tschauner's user avatar
0 votes
0 answers
77 views

Is there a way to use the fuzzywuzzy python package to exactly match one part of a string and partially match the other?

I'm trying to determine if one specific business exists in two different datasets. There are a few things that are making this more complicated than simply doing a join in SQL to only return identical ...
Mandi Moon's user avatar
0 votes
0 answers
51 views

How to store the substring that produces the closest match instead of the entire string

I am fuzzy matching a list of names from a list of strings. I am then creating a dictionary that holds the matched names and the strings from which the match came from (see below). I used the ...
LennyAngola's user avatar
0 votes
0 answers
26 views

Compare 2 files and use fuzzy wuzzy to return a table with the best match

I have the following two datasets: Raw Data Lookup Table and I am trying to use fuzzy wuzzy to create the following extract via a dataframe. Expected file table I was able to create a loop to find ...
junkxboxx's user avatar
0 votes
1 answer
153 views

How to use FuzzyWuzzy to match and merge two datasets in Python [duplicate]

I have three excel datasets that I am trying to merge. These three datasets all have names in column A, however across all three datasets there are punctuation, different spelling, different spaces, ...
Jason Jiang's user avatar
0 votes
1 answer
266 views

How to get a substring using fuzzywuzzy from a single long string (without spaces)

I am trying to perform a fuzzy search on a single long "word", using a target word using the process.extract from thefuzz (fuzzywuzzy). However, I am getting single letters/numbers, as ...
Bas Jansen's user avatar
  • 3,343
0 votes
0 answers
20 views

Overwrite Initial automated mapping

I am trying to map column names in a dataset to names in a standard, the standard names are in a yaml file so you can change them if you need to. I was able to do an initial automated mapping, but I ...
Johan VS's user avatar
2 votes
1 answer
134 views

Python merge two dataframes (fuzzy matching, some columns exactly match one does not)

I have two dataframes: df1 district place year votes candidate 1 1 2000 25 bond james 1 1 2000 30 smith john peter 1 1 2000 10 caprio leonardo di 1 1 ...
MG Fern's user avatar
  • 119
0 votes
1 answer
150 views

Probem with implementing my search bar(fuzzywuzzy) flutter

I'm using the package fuzzywuzzy from flutter and I don't know why but it says my result variable is null. Can you guys help me and explain why? I wanted the value of result I tried putting it in an ...
Roger Antony's user avatar
0 votes
0 answers
383 views

fuzzy matching millions of records

I am trying do a fuzzy matching between two columns in which col="co_zip22" will iterate through all the rows in col="co_zip23" and will find a match with a match score So ...
user22600865's user avatar
0 votes
0 answers
47 views

Standardizing Slightly Different Values Across 4 Dataframes

I am looking for an elegant (computationally less expensive solution) to this: I have 4 scraped dataframes containing football club names. Sometimes 1 or 2 out of 4 frames contain the name slightly ...
EscadeSupremo's user avatar
0 votes
0 answers
81 views

Jupyter notebook to compare xml

I have some data in xml that are provided via rss. I want to write a simple script to be able to point the code in the right direction and compare only the tags that i want to check. The idea is that ...
signorz's user avatar
0 votes
1 answer
21 views

fuzzy search in two datasets issue

import pandas as pd from fuzzywuzzy import fuzz from recordlinkage.preprocessing import clean from concurrent.futures import ThreadPoolExecutor column1_name = 'Sold To Customer Name' column2_name = '...
med jalel's user avatar
0 votes
1 answer
415 views

Deduplication with FuzzyWuzzy creating new columns

I've been trying to find duplicates for all cells in a column called "text" that at least match 90% and only keep the first row in case of duplicates found (and remove the rest of the ...
jms's user avatar
  • 225
0 votes
1 answer
65 views

How can I fix a incorrect coloring of floats using openpyxl?

I have problem with my code. The code is supposed to run a different script that scrapes two match betting websites and then writes the matches and rates in xlsx file. The code I have problem with is ...
Kryštof Bochníček's user avatar
2 votes
1 answer
3k views

TheFuzz Library in Python - Ratio Function

I'm trying to understand how the ratio is calculated in this function. I've been searching all over the internet and even asking ChatGPT about it, but I can't find a single answer. The issue is that, ...
Iñaki Baglivo's user avatar
1 vote
1 answer
65 views

How to avoid cyclic matches in fuzzywuzzy

Here is my dataframe: df = pd.DataFrame( dict(Name=['Emma Howard', 'Emma Ward', 'Emma Warner', 'Emma Wayden'], Age=[33, 34, 43, 44], Score=[90, 95, 93, 92]) ) list2 = df['Name'].tolist() ...
Kingston X's user avatar
-3 votes
1 answer
52 views

Find All Variation Of A Word In A Column

How to identify all the variation of a word in a column_one, and then fill a value in other column, , columns_two, whenever a variation of that word is found? E.g. Fill column value with P, whenever a ...
fast_crawler's user avatar
2 votes
2 answers
2k views

Fuzzy Matching Optimization in PySpark

I am trying to perform some fuzzy matching on some data through PySpark. To accomplish this I am using the fuzzywuzzy package and running it on Databricks. My dataset is very simple. It is stored in a ...
Minura Punchihewa's user avatar
1 vote
1 answer
708 views

Grouping fuzzy matches within same column in a data frame

I am trying to group the similar names of companies basis fuzzy matching ( within same column ). But neither they are grouping correctly nor do I have the same number of rows in the resulting dataset. ...
ss_0708's user avatar
  • 193
0 votes
1 answer
174 views

Clean column values using string matching

I have a pandas column with thousand of company name values, a name can have many variations, but it ultimately refers to the same company name. I was thinking of doing some string matching using ...
Alejandro L's user avatar
3 votes
3 answers
165 views

Python Speeding Up a Loop for text comparisons

I have a loop which is comparing street addresses. It then uses fuzzy matching to tokenise the addresses and compare the addresses. I have tried this both with fuzzywuzzy and rapidfuzz. It ...
John Smith's user avatar
  • 2,826
0 votes
0 answers
48 views

Join data frames after applying fuzzywuzzy library

I have two data frames df1 = pd.DataFrame({'Key':['Apple Souce', 'Banana', 'Orange', 'Strawberry', 'John tabel']}) df2 = pd.DataFrame({'Key2':['Aple suce','apple','app','orange', 'Mango', 'Orag','Jon ...
hyeri's user avatar
  • 683
0 votes
1 answer
869 views

How to apply entity name matching (rapidfuzz) at scale

I have one dataframe of user inputted car names which I need to match against another database of car names (millions of records) and extract the unique identifier. What's the best way of improving ...
RoyalPotatoe's user avatar
0 votes
0 answers
108 views

How to get string similarity between elements from 2 dataframes

I have 2 excels files. In the first one there is a catalog of items and the category they belong, in the second one, it is a massive recollection of data items. The problems starts in my excel with ...
Fernando Castilla's user avatar
0 votes
0 answers
112 views

Python Fuzzywuzzy does not give a similarity of 100 for identical terms

Why is Fuzzywuzzy library not giving me back proper results for terms which are identical? I would assume a similarity of 100, or at least 95 if 100 is not working. Am I missing something? Maybe there'...
smartini's user avatar
  • 445
0 votes
0 answers
69 views

why does Fuzzywuzzy python script take forever to generate results?

To give an idea, I have an excel file(.xlsx format) within which I am working with 2 sheets at a time. I am interested in 'entity name' from sheet a and 'name' from sheet b. Sheet b has 'name' column ...
technophile_3's user avatar
3 votes
3 answers
1k views

How to find the longest common substring in a list of strings (>2 strings)? Trying FuzzyWuzzy and Sequence matcher

So I am trying to find a common identifier for journals using dois. For example, I have a list of dois for a journal: ['10.1001/jamacardio.2016.5501', '10.1001/jamacardio.2017.3145', '10.1001/...
msci's user avatar
  • 59
1 vote
0 answers
30 views

How can I detect similarity of names in the same columns

Guys I have a dataset like this: ` df = pd.DataFrame(data = ['John','gal britt','mona','diana','molly','merry','mony','molla','johnathon','dina'],\ columns = ['Name']) df ` it gives ...
Ismail Awad's user avatar
0 votes
1 answer
149 views

Adjusting nested apply() functions for large datasets

I have two dataframes that I'm trying to compare, and am facing a volume issue. I am passing one row of a new item description through a 4.5 million row inventory list and calculating similarity. I ...
Zachqwerty's user avatar
1 vote
1 answer
867 views

Fuzzy Matching 2 DataFrames on multiple columns which includes one column with Float Values

I have 2 DataFrames namely 'Master_data_df' & 'My_records_df'. I am required to find out records which are missed out from 'Master_data_df' by comparing with 'My_records_df'. Here the column '...
sagardbhangale's user avatar
0 votes
0 answers
85 views

difflib.get_close_matches is not giving any output when I compared 2 columns in pandas dataframe

I have 2 pandas dataframes, one with clean city names(df2) and another with unclean city names(df1). sample values: df2.city_name: bangalore df1.city_name: bongolor I tried to use the below code to ...
Anisha A's user avatar
0 votes
1 answer
86 views

for Loop over a list fuzzy match printing out match score

I have a question about a fuzzy match. Here is the function I am trying to write: def fuzz(x, keys): for i in keys: a = fuzz.ratio(x, keys) return dataset['match'] = dataset.col1....
james's user avatar
  • 5
0 votes
2 answers
94 views

How to extract a string within a parentheses that contains the string, a comma, and an integer?

I am trying to pull a string name that is within a parentheses that contains the strings follow by comma and an integer. My current dataframe output is this: print df1: name ...
atom's user avatar
  • 1
0 votes
1 answer
1k views

Parallel processing with rapidfuzz function

I have a dataset of 100 000 records. My problem is many to many type where i need to calculate the fuzzy score of name column in each row with 100k rows. I am using for loop to iterate each row and ...
Alpha Green's user avatar
6 votes
4 answers
775 views

How to vectorize and speed-up double for-loop for pandas dataframe when doing text similarity scoring

I have the following dataframe: d_test = { 'name' : ['South Beach', 'Dog', 'Bird', 'Ant', 'Big Dog', 'Beach', 'Dear', 'Cat'], 'cluster_number' : [1, 2, 3, 3, 2, 1, 4, 2] } df_test = pd....
illuminato's user avatar
  • 1,184
1 vote
2 answers
1k views

Fuzzywuzzy to compare two lists of strings of unequal length and save multiple similarity metrics

I'm trying to compare two lists of strings and produce similarity metrics between both lists. The lists are of unequal length, one is roughly 50,000 the other is about 3,000. But here are two MWE data ...
user2205916's user avatar
  • 3,416
0 votes
0 answers
54 views

python fuzz clusters with labels

wondering if anyone knows how to go about doing this in python. Is there a way to fuzzy match the first column(Fruit) against itself and build an output column like the 2nd column(Cluster)? thank you!
learn2learn's user avatar
0 votes
1 answer
322 views

Apply fuzzy string matching of two columns in two Pandas dataframes while preserving a similarity score and output a Pandas DataFrame

I have two data frames that I'm trying to merge, based on a primary & foreign key of company name. One data set has ~50,000 unique company names, the other one has about 5,000. Duplicate company ...
user2205916's user avatar
  • 3,416
1 vote
1 answer
2k views

How to apply fuzzy matching across a dataframe column with multiple lists and save results in a new column

I have a similar problem to the links provided in the following references with minor differences but want the same results: Apply fuzzy matching across a dataframe column and save results in a new ...
lordgriffith's user avatar
0 votes
1 answer
92 views

apply defined function to column pandas and fuzzywuzzy

I am using the fuzzywuzzy library to match strings in a reference list using Levenshtein Distance. I want to apply this function to a series, matching each value of the series to a value in a ...
Alejandro L's user avatar

1
2 3 4 5
11