Questions tagged [similarity]

Similarity measures quantify how much alike objects (e.g. documents, feature vectors) are.

Filter by
Sorted by
Tagged with
0 votes
2 answers
24 views

Calculate Cosine Similarity Sentences ValueError: Expected 2D array, got 1D array instead

So I'm doing a cosine similarity calculation on a list of sentences. I've got the embedding of the calculations done. Here's the embedding The shape of embedding (11, 3072) [[-0.02179624 -0.17235152 -...
  • 3
0 votes
1 answer
15 views

Matlab - find out to which of three matrices another one is the most similar

so I have three pre-defined 1 row matrices for example: a = [0.4 0.25 0.5 -0.6 0.2] b = [-0.8 0 0 0.7 -0.25] c = [0.15 -0.4 -0.8 0.3 -0.05] and the I have another matrix with multiple rows, such as: ...
0 votes
0 answers
30 views

Jaccard similarity for array of shape (411, 13) gives an error: ValueError: operands could not be broadcast together with shapes (411,) (13,)

I want to write a code of calculating jaccard similarity with the use of these formulas. My original dataset is in shape (411, 13) However, it works like this: # Read in the data from CSV file data = ...
0 votes
1 answer
44 views

Find and compare all methods recursively

This python code is working correctly and it returns the similarity score of 91% import textdistance string_a = 'Hello World!' string_b = 'Hello Word!' algs = textdistance.algorithms algs....
  • 31.1k
0 votes
1 answer
61 views

Please tell me how to fill an empty matrix with elements using for loop

I am a beginner who is studying bioinformatics with scanpy these days. I am trying to improve so any help is very welcome, thanks lot! ##This lists contains gene names. Angio=['ADAM17','AXIN1','AXIN2',...
  • 15
0 votes
0 answers
5 views

spatial similarity of two 2D variables

I want to asssess the spatial similarity of two 2D variables. For instance, temperature and precipitation in a lat-lon grid, for each instant. Is there a way to do this with python? Thanks in advance
  • 1
1 vote
1 answer
28 views

Calibrating one timeseries with another

I have two timeseries in two different dataframes. What is the best way to find the calibration coefficient between them? I was thinking to substrack one dataframe from another and divide with ...
  • 85
1 vote
1 answer
48 views

Map (1:1) N input sentences to N given sentences by similarity

I want to map N_a input sentences to N_b given sentences so that the mapping is one-to-one. That is, Every N_a is assigned No N_a appears more than once Unfortunately the inputs vary slightly over ...
  • 1,843
0 votes
0 answers
33 views

How to get string similarity between elements from 2 dataframes

I have 2 excels files. In the first one there is a catalog of items and the category they belong, in the second one, it is a massive recollection of data items. The problems starts in my excel with ...
0 votes
0 answers
27 views

Scoring issues while using BM25Similarity

While tinkering solr scores in a custom BM25Similarity class using solr params, I find that the scores are logged correctly but score of docs for the same query does not change after the calculation ...
1 vote
0 answers
31 views

How to improve text similarity/classification performance when classes are semantically similar?

I have an NLP classification problem whereby I want to match an input string (a question) to the most suitable string from a list of reference strings (FAQs), or abstain if confidence in a ...
0 votes
0 answers
6 views

How to calculate the Jaccard index of co-occurence matrix without diagonal values?

I transformed a 2-mode matrix into a 1-mode co-occurence matrix and canceled all the values on the diagonal (because I thought it was meaningless). I wanted to use the jaccard method to calculate the ...
1 vote
1 answer
29 views

Spacy similarity score for sweet M&M fails

python 3 spacy seems to have a problem with sweets such as M&Ms. import spacy nlp = spacy.load("en_core_web_lg" ) query = nlp( "M&M" ) query2 = nlp("M&M ...
  • 1,823
0 votes
0 answers
48 views

Build doc2vec model and find textually similar reviews using gensim

The dataset is Amazon's review dataset in gz file. # A function to read the zipped data at a specfic path # # How to use: # PATH = "/path/to/file" # for line in parse(PATH): # do something ...
  • 81
0 votes
0 answers
5 views

Reducing effect of a tag that most of the items have

This is more of a general data question. Suppose you had a few items each with an id and a list of tags item0 = { id: 0, tags: [t0,t1,t2] } item1 = { id: 1, tags: [t0,t2,t3,t4] } item2 = { ...
  • 28
1 vote
0 answers
39 views

Efficiently calculate a similarity matrix of a sequence of string

I have a large sequence of strings (~ 20,000 strings) and a custom string similarity function. I want to see which strings are similar to which strings. One way to do this is to calculate a similarity ...
  • 145
-1 votes
1 answer
22 views

How to evaluate how similar 5 datapoints are to 1 target datapoint

I am struggling to wrap my head around a problem I need to resolve. Say that we have a cars dataset (1) with many different cars that have different features (id, age, mileage, color, model,...). On ...
-1 votes
0 answers
10 views

How to weigh features or determine feature importance in unsupervised learning

I have two sets each with 15-20 attributes. I am using similarity/distance metrics like Jaccard or Hamming to find the similarity/distance between the two sets. I am looking at an option to weigh the ...
  • 45
0 votes
2 answers
84 views

Compare each string with all other strings in a dataframe

I have this dataframe: mylist = [ "₹67.00 to Rupam Sweets using Bank Account XXXXXXXX5343<br>11 Feb 2023, 20:42:25", "₹66.00 to Rupam Sweets using Bank Account ...
  • 31.1k
0 votes
0 answers
49 views

How to modify the code to only use one of the folders with the similar first word in their name using javascript?

The following code is used to generate NFTs by combining layers which are located inside folders. What if I wanted to use the entire collection of folders provided in props.folderNames but also force ...
  • 1
-1 votes
1 answer
32 views

Measure similarity of objects over a period of time

I've got a dataset that has monthly metrics for different stores. Each store has three monthly (Total sales, customers and transaction count), my task is over a year I need to find the store that most ...
0 votes
1 answer
47 views

How to correct this sas function in order to have the jaccard distance?

I created a SAS function using fcmp to calculate the jaccard distance between two strings. I do not want to use macros, as I'm going to use it through a large dataset for multiples variables. the ...
0 votes
0 answers
13 views

Using Proxy package Jaccard distance to calculate similarity between rows in data set

I have a very large data set (over 60K rows), a subset of which for a particular data type is around 32K rows). Sample data using dput is given below (first 200 rows of Water). My question is, how can ...
0 votes
1 answer
34 views

Displaying multiple results based on similarity Python

I would like to know if anyone could help me with this problem. Im a beginner at Python, and im trying to create a program. The problem is that i want to search for a person in a stats database, and ...
0 votes
0 answers
6 views

Can Cohen’s be used to detect change point in similarity?

I have two timeseries with 40 rows each. Typically, I can calculate Cohen’s d beginning with at least 2 rows. Is it statistically acceptable to plot the cohen’s d of the two timeseries to determine ...
  • 121
2 votes
1 answer
65 views

Normalize text with Excel

I need a function to normalize data. For example: in column A there are the reference data (written correctly) and in column B the data to correct. Is there a function that can find the most similar ...
0 votes
0 answers
19 views

Excel Fuzzy Lookup - Wrong Similarity value

The plugin Excel Fuzzy Lookup is returning a Similarity value of 1.0 for the case below, they are almost identical but not 100% identical, so I was expecting a value of slightly less than zero. ...
0 votes
1 answer
29 views

FuzzyWuzzy on 2 col from different DataFrames

I have a very easy but not simple(to me at least!) question I have 2 DFs: df1: Account_Name samsung tesla microsoft df2: Company_name samsung electronics samsung Ltd tesla motors Microsoft ...
0 votes
0 answers
42 views

Comparison of two long lists for every combination

I have two lists of 170000 elements each say list1 and list2 containing strings. The elements of the 2 are the same. I would like to confront the elements of list1 with all the elements of list2 and ...
  • 595
0 votes
1 answer
43 views

Finding similarity in a pandas variable

I have a dataset with company names as follows: {0: 'SEEO INC', 1: 'BOSCH GMBH ROBERT', 2: 'SAMSUNG SDI CO LTD', 12: 'NAGAI TAKAYUKI', 21: 'WESTPORT POWER INC', 26: 'SAMSUNG ELECTRONICS CO LTD', ...
  • 595
1 vote
1 answer
39 views

How would I make clusters from a Levenshtein similarity matrix?

I have a similarity matrix of words and would like to apply an algorithm that can put the words in clusters. Here's the example I have so far: from Levenshtein import distance import numpy as np ...
  • 357
1 vote
1 answer
85 views

How can I check similarity in meaning and not just having same words between two texts with spacy

I'm trying to compare two different texts—one coming from a Curriculum Vitae (CV) and the other from a job announcement. After cleaning the texts, I'm trying to compare them to detect if a job ...
0 votes
0 answers
20 views

Mantel test (geographic distance - diversity)

I would like to test if/how i) geographic distance and/or ii) environmental parameter effect species diversity. I tried code below which works but with numerous errors making me unsure whether ...
  • 1
1 vote
1 answer
32 views

'float' object is not iterable", 'occurred at index 1'

i have for the following dataset company_name_Ignite Mate Bence Raul Marina 01 TELECOM LTD NaN 01 Telecom, Ltd. 01 Telecom, ...
2 votes
0 answers
18 views

Subset Text Similarity Score - How to detect a piece of small text that is very similar to subset of a much bigger text

There are many ways to detect similarity between 2 texts like Jaccard Index, TFIDF cosine similarity or sentence embedding. However all of them are refering to use case of 2 texts which are fully to ...
0 votes
0 answers
30 views

Calculating Jaccard similarity between tweets

I have a problem about calculating semantic similarity or Jaccard similarity between tweets, I know that how Jaccard similarity works, but I think it can be devastating to compare all tweets with each ...
1 vote
1 answer
45 views

How to find best string match out of multiple possibilities in a dataframe?

I have a DF that looks like this: Row Master Option1 Option2 1 00150042 plc WAGON PLC wegin llp 2 01 ...
1 vote
2 answers
34 views

Iterating over 2 columns and comparing similarities in Python

I have a DF that looks like this: Row Account_Name_HGI company_name_Ignite 1 00150042 plc WAGON PLC 2 01 telecom, ltd. 01 TELECOM LTD 3 0404 ...
0 votes
0 answers
43 views

Why the structural similarity index measure decreases with decreasing image resolution?

I computed the structural similarity index (SSIM) value between a ground truth mask and its corresponding predicted mask in an image segmentation task using a UNet model. Both the ground truth and ...
  • 1,157
0 votes
1 answer
78 views

pandas: group near similar string data

I am trying to use groupby on a column with str type of data that has near similar values and get a count of it: for example: col A col year col C abc 2009 no plan today abc2 2009 ...
  • 17
0 votes
1 answer
96 views

how to find percentage of similarity between two arrays

I have two data arrays x and y: x = array([ 0., 0., 84., 80., 59., 22., 0., 0., 0., 0., 52., 122., 117., 1., 10., 0., 0., 0., 0., 0., 0., 92., 90., 74., ...
0 votes
1 answer
23 views

ElasticSearch: more_like_this query

I have an index = "es_demo" , where I need to find similar documents to given "_id", I don't think it is working as the returned results have same "_id" as mentioned in ...
0 votes
0 answers
56 views

Return an item in a list that has the highest similarity with a compared value (using SpaCy)

I have an assignment where I am building a function that will take in a movie description and will compare it to a list containing 10 other movie descriptions. Using spacy, I have gotten as far as ...
0 votes
0 answers
34 views

scipy's pdist(X, metric='dice') VS scipy.spatial.distance.dice() produce different results

can anyone explain why these produce different results? I cant find anything in the docs that suggest why they would be different. doc for pdist from scipy.spatial.distance import pdist X = [[1, 0, 0]...
1 vote
0 answers
48 views

Similarity score with penalty if it doesn't match a specific part

I have been working with KNN, Fuzzy Matching, and Neural networks to do similarity scoring and have run into an interesting problem. We are using it to match drug formulations and I have found that ...
  • 732
1 vote
1 answer
25 views

Cypher - match nodes with similar relations rank result based on number of identical relations

I have a graph where node/relations look like: (n:Entity)-[r:HasAttribute]->(a:Attribute) All Entity- and Attibute-nodes have a property called id_obj to identify each node Lets say I have an ...
1 vote
0 answers
22 views

How can I detect similarity of names in the same columns

Guys I have a dataset like this: ` df = pd.DataFrame(data = ['John','gal britt','mona','diana','molly','merry','mony','molla','johnathon','dina'],\ columns = ['Name']) df ` it gives ...
0 votes
0 answers
61 views

Color similarity for multiple colored images

I am trying to compare color between multiple images. My target is to find similar colored images sorted by the highest similarity score. For example: The input image is: Current output: I have ...
0 votes
1 answer
31 views

Can I apply similarity models to .tif files in R?

Basically, I have two groups of .tif files (some for Spain and some other for California), and I want to statistically compare the climatic variables (max, min and mean T, and Monthly rainfall) in ...
  • 3
0 votes
0 answers
29 views

ValueError: Expected 2D array, got 1D array instead: array=[4. 4. 3. ... 3. 3. 3.]

I tried to find cosine_similarity that compare pivot table and object using .loc, and the result says ValueError: Expected 2D array, got 1D array instead: array=[4. 4. 3. ... 3. 3. 3.]. Reshape your ...
  • 1

1
2 3 4 5
37