517
questions
-1
votes
0
answers
38
views
Optimizing Data Processing for Product Comparison Script (Python) [closed]
I have a Python script that scrapes product data from multiple websites and performs various processing steps before saving the results as a comparison table (CSV and Excel).
However, the data ...
1
vote
0
answers
49
views
Trying to fix names in my database with fuzzywuzzy
What I'm trying to do is find and correct similar names in my database, like 'Patrick Maxwell' and 'Patrick Maxwel.' However, the issue I'm facing is that the best match for each name is often itself, ...
0
votes
0
answers
39
views
Fuzzy Match 2 Large Pandas Dataframes
I have 2 pandas dataframes that both contain company names. I want to left join df1(~10k rows) with df2(~1.6m rows) on company names using a fuzzy match. My current function takes too long to run, so ...
0
votes
1
answer
50
views
PySpark FuzzyWuzzy UDF Causing Timeout Errors on Small Dataset / TimeoutError when filtering columns with Fuzzy Wuzzy similarity scores in PySpark
I'm working on a PySpark script to calculate similarity scores between columns using FuzzyWuzzy. I've defined a UDF for this purpose and used a for loop to iterate over columns specified in a metadata ...
1
vote
1
answer
41
views
Why doesn't fuzzywuzzy's process.extractBests give a 100% score when the tested string 100% contains the query string?
I'm testing fuzzywuzzy's process.extractBests() as follows:
from fuzzywuzzy import process
# Define the query string
query = "Apple"
# Define the list of choices
choices = ["Apple&...
0
votes
0
answers
44
views
Is there a way to use fuzzy selection to narrow the choices in a wx.ComboBox (wxWidgets)
I want to populate a wx.ComboBox with a large number of items but rather than having to scroll through them all, I'd like to type a string into the ComboBox and narrow the items in the ComboBox using ...
0
votes
1
answer
145
views
How to do fuzzy merge with 2 large pandas dataframes?
I have 2 pandas dataframes that both contain company names. I want to merge these 2 dataframes on company names using a fuzzy match. But the problem is 1 dataframe contains 5m rows and the other 1 ...
0
votes
0
answers
192
views
Fuzzy match on large dataset- speed up and keep similar matches
I have two beer datasets; one is ~3 million entries, and another is 175 thousand entries. Doing a fuzzy match on these two will take way too long. I've run a few tests on the same 1000 random sample ...
0
votes
2
answers
103
views
Fuzzy Logic to match the records in a dataframe
I have a huge datasets of 2 Million and i want to match the records based on the fuzzy logic I have my original dataframe like
+---------+---------------+
| name| address|
+---------+-------...
0
votes
0
answers
59
views
Optimum way to get a single best unique match of two pandas dataframe columns, based on a set of parameters present in both dataframes
I would like to fuzzy match two Pandas dataframes based on a number n parameters in columns, all present in both dataframes. The output should be a pair of unique filenames. The parameter sets might ...
0
votes
0
answers
77
views
Is there a way to use the fuzzywuzzy python package to exactly match one part of a string and partially match the other?
I'm trying to determine if one specific business exists in two different datasets. There are a few things that are making this more complicated than simply doing a join in SQL to only return identical ...
0
votes
0
answers
51
views
How to store the substring that produces the closest match instead of the entire string
I am fuzzy matching a list of names from a list of strings. I am then creating a dictionary that holds the matched names and the strings from which the match came from (see below). I used the ...
0
votes
0
answers
26
views
Compare 2 files and use fuzzy wuzzy to return a table with the best match
I have the following two datasets:
Raw Data
Lookup Table
and I am trying to use fuzzy wuzzy to create the following extract via a dataframe.
Expected file table
I was able to create a loop to find ...
0
votes
1
answer
153
views
How to use FuzzyWuzzy to match and merge two datasets in Python [duplicate]
I have three excel datasets that I am trying to merge. These three datasets all have names in column A, however across all three datasets there are punctuation, different spelling, different spaces, ...
0
votes
1
answer
266
views
How to get a substring using fuzzywuzzy from a single long string (without spaces)
I am trying to perform a fuzzy search on a single long "word", using a target word using the process.extract from thefuzz (fuzzywuzzy). However, I am getting single letters/numbers, as ...
0
votes
0
answers
20
views
Overwrite Initial automated mapping
I am trying to map column names in a dataset to names in a standard, the standard names are in a yaml file so you can change them if you need to. I was able to do an initial automated mapping, but I ...
2
votes
1
answer
134
views
Python merge two dataframes (fuzzy matching, some columns exactly match one does not)
I have two dataframes:
df1
district place year votes candidate
1 1 2000 25 bond james
1 1 2000 30 smith john peter
1 1 2000 10 caprio leonardo di
1 1 ...
0
votes
1
answer
150
views
Probem with implementing my search bar(fuzzywuzzy) flutter
I'm using the package fuzzywuzzy from flutter and I don't know why but it says my result variable is null.
Can you guys help me and explain why?
I wanted the value of result I tried putting it in an ...
0
votes
0
answers
383
views
fuzzy matching millions of records
I am trying do a fuzzy matching between two columns in which col="co_zip22" will iterate through all the rows in col="co_zip23" and will find a match with a match score
So ...
0
votes
0
answers
47
views
Standardizing Slightly Different Values Across 4 Dataframes
I am looking for an elegant (computationally less expensive solution) to this:
I have 4 scraped dataframes containing football club names. Sometimes 1 or 2 out of 4 frames contain the name slightly ...
0
votes
0
answers
81
views
Jupyter notebook to compare xml
I have some data in xml that are provided via rss. I want to write a simple script to be able to point the code in the right direction and compare only the tags that i want to check.
The idea is that ...
0
votes
1
answer
21
views
fuzzy search in two datasets issue
import pandas as pd
from fuzzywuzzy import fuzz
from recordlinkage.preprocessing import clean
from concurrent.futures import ThreadPoolExecutor
column1_name = 'Sold To Customer Name'
column2_name = '...
0
votes
1
answer
415
views
Deduplication with FuzzyWuzzy creating new columns
I've been trying to find duplicates for all cells in a column called "text" that at least match 90% and only keep the first row in case of duplicates found (and remove the rest of the ...
0
votes
1
answer
65
views
How can I fix a incorrect coloring of floats using openpyxl?
I have problem with my code. The code is supposed to run a different script that scrapes two match betting websites and then writes the matches and rates in xlsx file. The code I have problem with is ...
2
votes
1
answer
3k
views
TheFuzz Library in Python - Ratio Function
I'm trying to understand how the ratio is calculated in this function. I've been searching all over the internet and even asking ChatGPT about it, but I can't find a single answer.
The issue is that, ...
1
vote
1
answer
65
views
How to avoid cyclic matches in fuzzywuzzy
Here is my dataframe:
df = pd.DataFrame(
dict(Name=['Emma Howard', 'Emma Ward', 'Emma Warner', 'Emma Wayden'],
Age=[33, 34, 43, 44], Score=[90, 95, 93, 92])
)
list2 = df['Name'].tolist()
...
-3
votes
1
answer
52
views
Find All Variation Of A Word In A Column
How to identify all the variation of a word in a column_one, and then fill a value in other column, , columns_two, whenever a variation of that word is found?
E.g. Fill column value with P, whenever a ...
2
votes
2
answers
2k
views
Fuzzy Matching Optimization in PySpark
I am trying to perform some fuzzy matching on some data through PySpark. To accomplish this I am using the fuzzywuzzy package and running it on Databricks.
My dataset is very simple. It is stored in a ...
1
vote
1
answer
708
views
Grouping fuzzy matches within same column in a data frame
I am trying to group the similar names of companies basis fuzzy matching ( within same column ).
But neither they are grouping correctly nor do I have the same number of rows in the resulting dataset. ...
0
votes
1
answer
174
views
Clean column values using string matching
I have a pandas column with thousand of company name values, a name can have many variations, but it ultimately refers to the same company name. I was thinking of doing some string matching using ...
3
votes
3
answers
165
views
Python Speeding Up a Loop for text comparisons
I have a loop which is comparing street addresses.
It then uses fuzzy matching to tokenise the addresses and compare the addresses. I have tried this both with fuzzywuzzy and rapidfuzz.
It ...
0
votes
0
answers
48
views
Join data frames after applying fuzzywuzzy library
I have two data frames
df1 = pd.DataFrame({'Key':['Apple Souce', 'Banana', 'Orange', 'Strawberry', 'John tabel']})
df2 = pd.DataFrame({'Key2':['Aple suce','apple','app','orange', 'Mango', 'Orag','Jon ...
0
votes
1
answer
869
views
How to apply entity name matching (rapidfuzz) at scale
I have one dataframe of user inputted car names which I need to match against another database of car names (millions of records) and extract the unique identifier. What's the best way of improving ...
0
votes
0
answers
108
views
How to get string similarity between elements from 2 dataframes
I have 2 excels files. In the first one there is a catalog of items and the category they belong, in the second one, it is a massive recollection of data items.
The problems starts in my excel with ...
0
votes
0
answers
112
views
Python Fuzzywuzzy does not give a similarity of 100 for identical terms
Why is Fuzzywuzzy library not giving me back proper results for terms which are identical?
I would assume a similarity of 100, or at least 95 if 100 is not working. Am I missing something?
Maybe there'...
0
votes
0
answers
69
views
why does Fuzzywuzzy python script take forever to generate results?
To give an idea, I have an excel file(.xlsx format) within which I am working with 2 sheets at a time.
I am interested in 'entity name' from sheet a and 'name' from sheet b.
Sheet b has 'name' column ...
3
votes
3
answers
1k
views
How to find the longest common substring in a list of strings (>2 strings)? Trying FuzzyWuzzy and Sequence matcher
So I am trying to find a common identifier for journals using dois. For example, I have a list of dois for a journal:
['10.1001/jamacardio.2016.5501',
'10.1001/jamacardio.2017.3145',
'10.1001/...
1
vote
0
answers
30
views
How can I detect similarity of names in the same columns
Guys I have a dataset like this:
`
df = pd.DataFrame(data = ['John','gal britt','mona','diana','molly','merry','mony','molla','johnathon','dina'],\
columns = ['Name'])
df
`
it gives ...
0
votes
1
answer
149
views
Adjusting nested apply() functions for large datasets
I have two dataframes that I'm trying to compare, and am facing a volume issue.
I am passing one row of a new item description through a 4.5 million row inventory list and calculating similarity. I ...
1
vote
1
answer
867
views
Fuzzy Matching 2 DataFrames on multiple columns which includes one column with Float Values
I have 2 DataFrames namely 'Master_data_df' & 'My_records_df'.
I am required to find out records which are missed out from 'Master_data_df' by comparing with 'My_records_df'.
Here the column '...
0
votes
0
answers
85
views
difflib.get_close_matches is not giving any output when I compared 2 columns in pandas dataframe
I have 2 pandas dataframes, one with clean city names(df2) and another with unclean city names(df1).
sample values:
df2.city_name: bangalore
df1.city_name: bongolor
I tried to use the below code to ...
0
votes
1
answer
86
views
for Loop over a list fuzzy match printing out match score
I have a question about a fuzzy match.
Here is the function I am trying to write:
def fuzz(x, keys):
for i in keys:
a = fuzz.ratio(x, keys)
return
dataset['match'] = dataset.col1....
0
votes
2
answers
94
views
How to extract a string within a parentheses that contains the string, a comma, and an integer?
I am trying to pull a string name that is within a parentheses that contains the strings follow by comma and an integer.
My current dataframe output is this:
print df1:
name ...
0
votes
1
answer
1k
views
Parallel processing with rapidfuzz function
I have a dataset of 100 000 records. My problem is many to many type where i need to calculate the fuzzy score of name column in each row with 100k rows. I am using for loop to iterate each row and ...
6
votes
4
answers
775
views
How to vectorize and speed-up double for-loop for pandas dataframe when doing text similarity scoring
I have the following dataframe:
d_test = {
'name' : ['South Beach', 'Dog', 'Bird', 'Ant', 'Big Dog', 'Beach', 'Dear', 'Cat'],
'cluster_number' : [1, 2, 3, 3, 2, 1, 4, 2]
}
df_test = pd....
1
vote
2
answers
1k
views
Fuzzywuzzy to compare two lists of strings of unequal length and save multiple similarity metrics
I'm trying to compare two lists of strings and produce similarity metrics between both lists. The lists are of unequal length, one is roughly 50,000 the other is about 3,000.
But here are two MWE data ...
0
votes
0
answers
54
views
python fuzz clusters with labels
wondering if anyone knows how to go about doing this in python.
Is there a way to fuzzy match the first column(Fruit) against itself and build an output column like the 2nd column(Cluster)?
thank you!
0
votes
1
answer
322
views
Apply fuzzy string matching of two columns in two Pandas dataframes while preserving a similarity score and output a Pandas DataFrame
I have two data frames that I'm trying to merge, based on a primary & foreign key of company name. One data set has ~50,000 unique company names, the other one has about 5,000. Duplicate company ...
1
vote
1
answer
2k
views
How to apply fuzzy matching across a dataframe column with multiple lists and save results in a new column
I have a similar problem to the links provided in the following references with minor differences but want the same results:
Apply fuzzy matching across a dataframe column and save results in a new ...
0
votes
1
answer
92
views
apply defined function to column pandas and fuzzywuzzy
I am using the fuzzywuzzy library to match strings in a reference list using Levenshtein Distance. I want to apply this function to a series, matching each value of the series to a value in a ...