Questions tagged [corpus]
A corpus most commonly refers to a collection of structured text. Please consider asking your question on https://opendata.stackexchange.com, if your question is not closely related to programming or you are just looking for a freely available corpus for any purpose.
689
questions
1
vote
1
answer
22
views
Database that allows full text search in O(1)
I have a database of documents where searching quickly for keywords and patterns would be very useful to have.
I know of "Burrows–Wheeler transform"/FM-index. I wonder if there are any ...
0
votes
1
answer
19
views
Clean file line by line and write lines into a new file
I am new to Python and I want to clean a big file line by line and write clean lines into a new file. I think I have been able to read the file but for some reason, when I try to write the file, only ...
2
votes
1
answer
28
views
How to solve missing words in nltk.corpus.words.words()?
I have tried to remove non-English words from a text. Problem many other words are absent from the NLTK words corpus.
My code:
import pandas as pd
lst = ['I have equipped my house with a new [xxx]...
0
votes
2
answers
48
views
How do I create two subsets out of a corpus based on multiple keywords?
I am working with a large body of political speeches in quanteda and would like to create two subsets.
The first one should contain one or more from a list of specific keywords(e.g. "migrant*&...
0
votes
0
answers
12
views
How to read tagged and lemmatized corpus with NLTK?
I'm creating a corpus for my personal conlang, and I've been tagging and lemmatizing everything manually. The format is token/tag/lemma. For example, je/c/je nu/bd/nu jet/vasp3s/imi ambrós/nmns/ambrós ...
0
votes
1
answer
12
views
how to convert a matrix to BoW format?
I am trying to convert a matrix to the type that can be received by gensim. AuthorTopic Model, which means I should convert a matrix to a sparse vector. I have already tried several functions in ...
0
votes
1
answer
32
views
Best way of Splitting up and then putting back a huge dataframe?
Basically I have a dataframe with 300 000 rows, which I need to do sentiment analysis on. The dataframe contains reddit comments, so I need to transform them into corpus, but the function I have only ...
1
vote
2
answers
95
views
Relationship between vocab size and complexity
I have 2 corpuses, if one has a larger vocabulary size than the other, does it mean its language is more complex?
Apart from complexity of the language, what else can effect the size of the vocabulary ...
0
votes
0
answers
24
views
filtered content of corpus by custom function with R
I want to analysis filtered texts by custom function (function with parameters) using R.
I used readlines function to extract my text and I get large list with 258 lists. Then, using VCorpus(...
0
votes
0
answers
32
views
How can I get a list of sentences from a corpus using R (and maybe the koRpus package)?
I am new to R and I am trying to use its functionalities for corpus analysis in Portuguese texts. I am using the koRpus package. I am writing a Shiny app that shows all the sentences in which a search ...
0
votes
1
answer
53
views
do "for loop" for table division and frequency word calculation
I would need to divide this dataframe of 1038319 rows into smaller tables of 25k each, then on each table I have to do the following operation, at the beginning I was doing manually table by table, ...
1
vote
1
answer
127
views
R: How to count the total number of tokens in a corpus?
I have created a Quanteda corpus called readtext_corpus with 190 types of text. I would like to count the total number of tokens or words in the corpus. I tried the function ntoken which gives a ...
0
votes
0
answers
28
views
How can i use Corpus Convokit with chatterbot?
I am using chatterbot and want to train bot with convokit datasets, but script fails with this error:
File "main.py", line 22, in <module>
trainer.train(corpus_movie)
File "C:\...
0
votes
2
answers
32
views
Problem with creating dictionary with gensim for LDA
I have a problem running gensim to create a Dictionary and the Doc Term Matrix.
When I run:
from gensim import corpora, models
import gensim
clean = ['door', 'cat', 'mom']
dictionary = corpora....
0
votes
0
answers
109
views
55 Hour Long Single Epoch Colab GPU Training Time for a Small RoBERTa-like Transformer
I am following this guide: https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb
To train a transformer for Pali. A dead language but has applicability ...
0
votes
0
answers
19
views
How do I import a dataset already in xml available on a server to R and create a data frame out of it
I am new at coding and even more new with R.
This is the dataset corpus I am supposed to use: ftp://205.193.86.89/Consolidation_Regs_1.2.0/EN
There's an English and French version, I only need the ...
0
votes
2
answers
85
views
Is it possible that python takes very long to assign a name to a huge list (~10 gb)?
I am working with a very long list of lists (I'm talking 10 gb if you put it in a file), a cleaned corpus. In my script, I assign it a name, and then use it in another function that has to do with ...
0
votes
1
answer
50
views
Compress PDF files in R to make it faster to create corpus
I am doing a structural topic model of a large number of documents. The corpus is created from over 2500 pdf-files, and naturally, it takes a very long time to process these. However, I can see, that ...
0
votes
1
answer
26
views
TermDocumentMatrix Error after Cleaning Corpus
My problem is that I want to pass my corpus to the tm function termdocumentmatrix() and it fails with the error: Error in UseMethod("meta", x): no applicable method for meta' applied to an ...
0
votes
0
answers
17
views
quanteda does not recognize it's own corpus_group
I need help with corpus_group.
I am working with twitter data. Everything seems to be going fine. However, when I try to use corpus_group, I get an error message: could not find function "...
1
vote
0
answers
22
views
Multiprocessing in NLTK
I am trying to figure out a way to process a huge personal corpora in the following way and it's taking well beyond hours to process. I have a good enough CPU to speed it up manifold but I can't ...
-1
votes
1
answer
16
views
Python NLTK Lazy
I am attempting to analyze a text in Python using NLTK but when I run the program I get an error that says TypeError: 'LazyCorpusLoader' object is not callable. I'm not exactly sure why I am getting ...
0
votes
1
answer
18
views
Create a program using NLTK that asks for a word and checks whether it is more frequent as a Noun or a Verb in Brown corpus
import nltk
from nltk.corpus import brown
user=input("Enter a word:")
words=brown.tagged_words()
for word in words:
if
I have started like this but I can"t go on
0
votes
1
answer
27
views
Text Mining: Cluster Analysis phrases. ERROR: cannot take a sample larger than the population
I'm working on a dataset of thousands of sentences. The dataset is structured by a column and k rows.
I've to find some similarities between them e i'm doing a cluster Analysis. I created a corpus and ...
0
votes
1
answer
37
views
How to extract big docs from NLTK corpus
I have downloaded the corpus Reuters from the NLTK library and want to store 10 random documents with more than 50 elements in a new variable.
I have already downloaded the corpus and have written the ...
0
votes
0
answers
22
views
How to create a corpus with multiple columns using python?
I need to create a corpus of news articles with columns as dates. Dates - full 2021 year.
Is it a good idea to create a simple .csv file with articles as values? There might be several articles on any ...
0
votes
1
answer
78
views
Quanteda: How do I create a corpus and plot dispersion of words?
I have some data which looks like this:
date signs horoscope newspaper
<chr> <chr> <chr> ...
0
votes
1
answer
100
views
Cleaning Webscrape Text for FastText in Python
BLUF: How can I clean up large corpora with issues of words being lumped together (like "quarterbackpa" or "passquarterback")?
Background:
I am working on my Capstone project for ...
0
votes
0
answers
11
views
How to identify expanded text of short words using python, For Ex: "mgmt" for management, "svcs" for services using python
I am working on a project where there are lot of short form of words like "Mgmt" or "Mgmnt" for "Managament", "Svcs" for services. I would want to identify ...
0
votes
1
answer
67
views
Detokenize a Quanteda tokens object
I have a quanteda tokens object that I created using the "window" option (see code below). I'm interested in doing this on a series of words in order to inform the creation of a custom ...
0
votes
1
answer
86
views
how to list all downloaded datset from nltk
I downloaded some of the datasets from nltk using
import nltk
import nltk.corpus
nltk.download()
Now I want to list all the downloaded dataset
I don't know how.
0
votes
2
answers
26
views
How can I count unique 2 word phrases that are seperated by a comma within a cell in R?
I have a dataframe of different locations (Location) along with the species of animals (Spp) found at each location. The species of animals are coded using their unique Genus species names. I would ...
0
votes
0
answers
19
views
How to evaluate the quality of a text corpus?
I have collected text data from various domains and online sources to create a monolingual text corpus for my research purposes. However, I am confused about how to evaluate the quality and acceptance ...
0
votes
1
answer
37
views
save properly a persian text with english keyword inside, on my local computer
I want to save a list of sentences in English on my local computer, The problem is that there are some English names inside and when I save that the order of letter is changed.
e.g
['بله، آن را آزار ...
0
votes
0
answers
20
views
Keeping documents in a corpus that contain a string?
I am new to R, and I apologize if the solution is obvious! I have a simple corpus TM object, and I need to ONLY keep DOCUMENTS that contain specific string values in the "content" meta-field ...
0
votes
1
answer
61
views
quanteda: subsetting the corpus based on key terms in title column
I have a corpus of approx. 8,000 news articles related to migration, migrants, asylum, and refugees. However, I would like to subset this corpus to articles that only include these terms in the title, ...
0
votes
1
answer
86
views
the repetitions in Gensim Word2Vec training corpus
I am using Gensim to train a Word2Vec embedding on different corpora, pertaining to different years, to compare the embedding vectors.
My question is: if I repeat the documents of a specific year ...
0
votes
1
answer
95
views
Creating Corpus from wiki dump file using Jupyter notebook
I'm trying to follow this page to create a wiki corpus, but I'm using Jupiter notebook https://www.kdnuggets.com/2017/11/building-wikipedia-text-corpus-nlp.html
this is my code:
import sys
from gensim....
0
votes
1
answer
35
views
Structure of Gensim Word Embedding corpus
I want to train a word2vec model using Gensim. I preprocessed my corpus, which is made of hundreds of thousands of articles from a specific newspaper. I preprocessed them (lower casing, lemmatizing, ...
0
votes
0
answers
30
views
How is data parsed in flair classification model training?
So I have prepared a dataset to train a flair classification model. It is formatted in the correct way (label<class_x> ). However, when I went to train my model I believe it jumbles up the data. ...
0
votes
0
answers
37
views
How do I automatically assign XML tags to blocks of text with R?
I have text that looks as follows:
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud ...
1
vote
2
answers
117
views
Convert Corpus from quanteda to tm
My data mycorpus is in a quanteda-corpus (corpus-function from quanteda) which I need to convert to a corpus under the tm package. I know about quanteda's convert-function. This, though, only converts ...
0
votes
0
answers
17
views
Which documents is in filtered metadata
I have about 300 metadata files, I did a filter based on particular term "diseas" which exceeding a threshold where I have 10 files. I used the following code and it runs perfectly fine:
#5 ...
0
votes
1
answer
44
views
For loop KeyError: 4675 when making corpus from Pandas dataframe
I've tried to make corpus from Pandas dataframe (with shape (14454, 9)).
However, whenever the range exceeds 10k, the for loop return KeyError: 4675 , yet works well for 10k below.
# getting the ...
0
votes
1
answer
21
views
R TM package produces strange results with Inspect command
I'm having a little trouble with the inspect function from the tm package in R.
I have a sample 2-row data.table as defined below:
dt <- data.table(doc_id = c(1, 2), text = c('the driver of the 1st ...
2
votes
1
answer
23
views
combining words in tm R is not achieving desired result
I am trying to combine a few words so that they count as one.
In this example I want val and valuatin to be counted as valuation.
The code I have been using to try and do this is below:
#load in ...
0
votes
1
answer
100
views
Remove 2 stopwords lists with Quanteda package R
I'm working with quanteda package on a corpus dataframe, and here is the basic code i use :
library(quanteda)
fmsi_des <- dfm(corpus_des, remove=stopwords("spanish"), verbose=TRUE,
...
0
votes
1
answer
54
views
Memory problems when using lapply for corpus creation
My eventual goal is to transform thousands of pdfs into a corpus / document term matrix to conduct some topic modeling. I am using the pdftools package to import my pdfs and work with the tm package ...
1
vote
0
answers
97
views
Cannot access a downloaded corpus in NLTK
I have downloaded the Russian NLTK corpus via nltk.download, but I cannot actually use it (Python 3.6.9, NLTK 3.5):
>>> nltk.download('averaged_perceptron_tagger_ru')
[nltk_data] Downloading ...
1
vote
0
answers
26
views
missing gold corpus in a qa system
I know that in order to evaluate a QA(question answering) system you need to compare the output(answer) against a gold standard. My question is, in absence of a gold standard, how can I evaluate a QA ...