Questions tagged [gensim]
Gensim is a free Python framework designed to automatically extract semantic topics from documents, as efficiently (computer-wise) and painlessly (human-wise) as possible.
2,331
questions
0
votes
0
answers
11
views
How to make custom scorer for Gensim lda to get alpha and eta values?
I am trying to get alpha and eta values when my gensim LDA model is set to 'asymmetric' mode. It can easily be get through applying lda_model.alpha . This returns an array of alpha values .
I am ...
0
votes
1
answer
26
views
FastText: Can't see the representation of words that starts with @ or @
I am working in an NLP project using FastText. I have some texts which contains words like @.poisonjamak, @aminagabread, @iamquak123 and I want to see their FastText representation. I want to mention ...
-1
votes
1
answer
42
views
TypeError: and how do I pre process my data?
https://pastecord.com/yfabaficup.sql
Im getting this type error preprocessing my data.
tried searching the whole google and stack overflow
cannot find answer please help. Im using pandas and gensim ...
0
votes
0
answers
35
views
AttributeError: 'Series' object has no attribute 'apply' is observed on a cudf created dataframe column
To run the data frames on GPU, I am using cudf from rapidsai. When I am trying to read a text column to generate preprocessed tokens, I am getting the below error:
Code:
import gensim
import cudf
...
0
votes
1
answer
17
views
Use gensim with existing embedding table
I have generated a bunch of word embeddings using a fine-tuned transformer model from the huggingface transformers library.
Now would like do some quick evaluation, whether the results are any good. I ...
1
vote
1
answer
49
views
Is there a required size of data set for LDA to work in python?
I am trying to apply LDA on Stack overflow posts for a project. My corpus has about 650 post of interest and I was wondering if the size of the corpus would hinder the functioning of the LDA later on ...
1
vote
1
answer
51
views
How to convert small dataset into word embeddings instead of one-hot encoding?
I have a dataset of 33 words that are a mix of verbs and nouns, for eg. father, sing, etc. I have tried converting them to 1-hot encoding but for my use case, it has been suggested to look into ...
1
vote
1
answer
44
views
Gensim- KeyError: 'word not in vocabulary'
I am trying to achieve something similar in calculating product similarity used in this example. how-to-build-recommendation-system-word2vec-python/
I have a dictionary where the key is the item_id ...
0
votes
0
answers
24
views
softcosinesimilarity doesn't work while using gensim
i have a set of documents and i try to use soft cosine similarity to detect the similar documents and print the similarity matrix but when i run the code i face many errors:
code :
enter image ...
0
votes
1
answer
67
views
Creating a pairwise similarity matrix of a list of words using Gensim Fasttext in python
I have a list of words, and I need to create a pairwise similarity matrix using the Fasttext word embedding. This is what I am currently doing:
from gensim.models import fasttext as ft
from sklearn....
0
votes
1
answer
14
views
GoogleNews-vectors-negative300.bin cannot be loaded in gensim models MemoryError
I have been trying to load GoogleNews Vector file to gensim models. The program never finished loading and I keep getting MemoryError. A few days ago, I didn't have that problem. I don't know why I ...
-1
votes
1
answer
69
views
How to handle KeyError(f"Key '{key}' not present") wor2vec with gensim
I have build a model with gensim library and am trying to get the vector of word that not present in the vocabulary but i have an error, and i want to handle this error with the best i way.
If i can ...
0
votes
1
answer
54
views
I get an (AttributeError: The vocab attribute was removed from KeyedVector in Gensim 4.0.0) when i try to load google news vector embeddings
I have a class Featurizer that checks for the existance of an embedding_file which has word embeddings from Google news vectors and loads it when called. However when i use the class Feauturizer to ...
0
votes
1
answer
42
views
how to find similarity for multi-word tokens? ex: ("united states of america", "united kingdom")?
I am using Gensim to compute the cosine similarity between two words. I am using following Stanford GLoVE 42B data set: https://nlp.stanford.edu/data/glove.42B.300d.zip
when i do this
glove_model....
0
votes
1
answer
29
views
EOFError when converting gensim word2vec to binary format
I have a pretrained embeddings with word2vec format in txt. I loaded it and then saved it to .bin. But I cannot load this embeddings as an EOFError: unexpected end of input; is count incorrect or file ...
0
votes
1
answer
53
views
How to create word embedding using Word2Vec on Python?
I have seen many tutorials online on how to use Word2Vec (gensim).
Most tutorials are showing on how to find the .most_similar word or similarity between two words.
But, how if I have text data X and ...
1
vote
1
answer
30
views
Page similarity calculation with Gensim
there are some questions about this which I've studied, but I'm still not sure about a couple of things.I want to split a page stream (concatenated PDFs) in to singular documents. So the trick is to ...
0
votes
1
answer
46
views
How to speed up Word2Vec's initial vocabulary scan for massive data?
I now have a massive data corpus dataset, it has about 11 billion sentences, each sentence has about 10 words, divided into more than 12,000 files ending with .txt.gz. I wish to Skip--Gram it with ...
0
votes
0
answers
17
views
Error installing gensim similarities package
I'm running into an error installing some of the sub-packages from Gensim in a both a local jupyter notebook and Google Colab. I've looked at other tutorials, but the solutions aren't working for me. ...
0
votes
0
answers
37
views
Runtime Error when running a LDA model of gensim on Windows
Okay, so i know this is a Windows error and not a Gensim error. Based on previous examples on the internet and other comments/solutions from Stack Overflow I came up with the code below. However, the ...
0
votes
1
answer
40
views
What is the data type of X in pca.fit_transform(X)?
I got a word2vec model abuse_model trained by Gensim. I want to apply PCA and make a plot on CERTAIN words that I only care about (vs. all words in the model). Therefore, I created a dict d whose keys ...
1
vote
1
answer
83
views
KeyError: word not in vocabulary" in word2vec
I wanted to convert some Japanese word to vector so that I can train the model for prediction. For that I downloaded pretrained models from Here.
import gensim
from sklearn.feature_extraction.text ...
1
vote
0
answers
19
views
Sharing updated gensim model file between gunicorn workers
I am trying to create an API using Gunicorn and Flask for calling a gensim model.
One of the endpoints would allow users to input new products into the model without any downtime. We achieve this ...
0
votes
0
answers
32
views
Python Error in NLP Function: "String indices must be integers"
I have individual CSV's and I am hoping to apply a topic model to each of them.
A CSV dataframe looks like:
<OUT>
PageNumber English_tags_only
59 people, trees, lego, water
The ...
0
votes
1
answer
11
views
Infer "shapes", or infer analogous relations in Word2Vec
Gensim Word2Vec offers a system for inferring analogous relationships, that is, with the "same shape" as those already found?
Es:
Starting from King, Queen
I would like to get other couples ...
0
votes
0
answers
22
views
LDA Topic Modelling Extract Words of Certain Probabilities to CSV - Python
I have this function to apply a topic model to every document. However, I want to extract words which are over e.g. 70% probability of contributing to each topic and print these into a CSV dataframe. ...
0
votes
1
answer
49
views
Apply Lambda to Function Not Working? - Python
I have a df called central_edi_grouped_posts like:
df = pd.DataFrame({'PageNumber': [175, 162, 576], 'new_tags': [['flower architecture people'], ['hair red bobbles'], ['sweets chocolate shop']})
<...
1
vote
0
answers
50
views
How to measure the coherence in sklearn LDA model (Non-gensim LDA model)?
I have tried using two techniques, but I am getting different results. I just want to be sure about which one to go with.
Method 1:
I tried using
from tmtoolkit.topicmod.evaluate import ...
0
votes
1
answer
31
views
Custom word-embeddings in gensim
I have a word embedding matrix (say M) obtained of order V x N where V is the size of the vocabulary and N is the size of each word vector. I want the word2vec model of gensim to initialise its word ...
0
votes
2
answers
28
views
Can I create a histogram/bar graph from a list in python? typeError: expected str, found List
I am using matplotlib, pandas and gensim. I am trying to create a histogram based on frequent words by extracting text directly from a website. I am receiving a typeError in this instance:
text = ','....
0
votes
1
answer
39
views
How do I convert a text corpus into a string? gettign error code: expected str instance, list found
Code
Getting the error code: word_frequency = Counter(" ".join(description_list).split()).most_common(10)
TypeError: sequence item 0: expected str instance, list found
What am I doing wrong? ...
0
votes
1
answer
74
views
Topic model for each row in dataframe
I have a subset of a dataframe that looks like (note, the new_tags are not exhaustively illustrated here):
df = pd.DataFrame({'PageNumber': [175, 162, 576], 'new_tags': [['flower architecture people'],...
2
votes
0
answers
15
views
what is the result of nlp function of spicy libarary in pesian text(is it a word vector?)
I use nlp function of spicy lib for creating word vector of persian text,
but the result of this function is not a word vector,and warning is said word vector is empty
i wanna know what is the result ...
0
votes
1
answer
40
views
Unable to produce visualisations to calculate topic frequency for LSI model
I am trying to create a graph which shows the frequency of the topics for LSI. I was able to do this for my LDA model using the same code.
When I try to visualise my LSI topics I get error messages as ...
0
votes
0
answers
24
views
Getting bus error when importing gensim package
I am getting:
"Bus error"
"Command run output for: python3 home/public/python/preprocess_eng.py
python3 home/public/python/preprocess_eng.py exited with no output and return code 10&...
0
votes
2
answers
44
views
Cannot see DEBUG logs for “number of documents converged“ info when running Gensim's LDA suggested for choosing iterations and passes
In official Gensim tutorial there is a mention about how to set number of iterations and passes:
I suggest the following way to choose iterations and passes. First, enable logging (as described in ...
0
votes
1
answer
44
views
Save gensim doc2vec model in google drive
I want to save my Gensim doc2vec model into my notebooks folder which is in my google drive but when I run the model.save('doc2vec_model') method it is saving the model somewhere else.
fname = ...
0
votes
0
answers
44
views
ValueError: could not broadcast input array from shape (50,) into shape (1,)
why i am getting error when i run the below code?
ValueError: could not broadcast input array from shape (50,) into shape (1,)
from gensim.test.utils import datapath, get_tmpfile
from gensim.models ...
1
vote
0
answers
175
views
ERROR: Could not build wheels for numexpr, which is required to install pyproject.toml-based projects
I'm trying to install some requirements via pip install -r requirements.txt on my raspberry pi4(ARMv7)
this is requirements.txt
...
kss==2.5.1
numpy==1.20.2
konlpy==0.5.2
scikit-learn==0.24.2
gensim==...
1
vote
1
answer
39
views
Generating Trigrams with Gensim's Phraser Package in Python
I have the following code snippet which I created with the help of this tutorial for unsupervised sentiment analysis purposes:
sent = [row for row in file_model.message]
phrases = Phrases(sent, ...
0
votes
0
answers
26
views
ImportError: cannot import name 'emd' while using gensim wmdistance()
Below is the code I am trying to execute and getting import name emd not found. Whereas I have installed pyemd package.
import gensim.downloader as api
import json
from gensim.models.word2vec import ...
1
vote
0
answers
16
views
whats the behind algorithm of gensim similarity package
I use from gensim similarity Package for calculate similarity between my news sentences.
i want to know how this package works?
what is the algorithm behind this similarity Process?
can anyone help me?...
0
votes
1
answer
49
views
How to get negative word samples in Gensim Word2Vec Model?
I am using gensim Word2Vec model to train word embeddings. My code is:
w2v_model = Word2Vec(min_count=20,
window=2,
vector_size=50,
...
0
votes
2
answers
168
views
word2vec/gensim — RuntimeError: you must first build vocabulary before training the model
I am having trouble training my own word2vec model on the .txt files.
The code:
import gensim
import json
import pandas as pd
import glob
import gensim.downloader as api
import matplotlib.pyplot as ...
0
votes
1
answer
46
views
Understanding output of gensim LDA topic modeling API
I was trying to understand gensim mallet wrapper for topic modeling as explained in this notebook.
In point 11, it prepares corpus which if of format Term Document frequency:
>>> print(corpus[...
0
votes
1
answer
15
views
how to convert a matrix to BoW format?
I am trying to convert a matrix to the type that can be received by gensim. AuthorTopic Model, which means I should convert a matrix to a sparse vector. I have already tried several functions in ...
0
votes
1
answer
59
views
How do I use gensim to vectorize these words in my dataframe so I can perform clustering on them?
I am trying to do a clustering analysis (preferably k-means) of poetry words on a pandas dataframe. I am firstly trying to vectorize the words by using the word-to-vector feature in the gensim package....
0
votes
1
answer
72
views
Topic weight in gensim lda model
do we have any function in gensim package to calculate the weights of topics ?
I need this weight to compare them with 0.05 (p_value) to understand which one is mean full.
0
votes
1
answer
46
views
Extract meaningful words from spaceless texts
I have not done much NLP, but have a need. For example for the string 'australiafreedomrally', I need to automatically extract meaningful words i.e., 'australia', 'freedom' and 'rally'.
Is there any ...
-1
votes
1
answer
36
views
Is there a way to check which topic a word would be in?
I have used Gensim's LDA topic modeling to create 6 topics. But now I would like to give the model a word and see which topic that would fall under. Is this possible? If so through which method?
Ex.
...