Questions tagged [corpus]

A corpus most commonly refers to a collection of structured text. Please consider asking your question on https://opendata.stackexchange.com, if your question is not closely related to programming or you are just looking for a freely available corpus for any purpose.

Filter by
Sorted by
Tagged with
2 votes
0 answers
84 views

Unable to edit metadata in corpus

I have the following corpus: library(jsonlite) library(tm) query = "https://www.ebi.ac.uk/ebisearch/ws/rest/pride?query=submitter_country:Norway&size=1000&fields=submitter_keywords&...
Illimar Rekand's user avatar
0 votes
1 answer
40 views

Searching for specific words in Corpus with R (tm package)

I have a Corpus (tm package), containing a collection of 1.300 different text documents [Content: documents: 1.300]. My goal is now to search the frequency of a specific wordlist in each of those ...
Li4991's user avatar
  • 71
0 votes
0 answers
18 views

Recommended way to extract "the representative" (not necessarily most frequent) 4-grams in a corpus? TF-IDF or

I have a corpus of 500 research articles and I want to extract the top 4-grams NOT simply based on the highest frequency but relevance to the research article genre in general (the 4-grams ...
Vahid's user avatar
  • 65
0 votes
0 answers
34 views

Docvarsfrom = filenames error message in Quanteda in R: "Filename elements are not equal in length"

I am putting together a corpus in quanteda in R and seem to be stuck at the first step. I want to specify that docvars should be taken from the filenames of my corpus. I had already named all files in ...
David Threlfall's user avatar
0 votes
0 answers
18 views

URLError: [WinError 10060] | When trying to install wordnet through Anaconda Jupyter (python)

I am getting this error - URLError: <urlopen error [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established ...
Kshitish Sood's user avatar
0 votes
1 answer
26 views

Why is the text in the files I am concatenating in Powershell coming out altered?

Sorry if this doesn't make much sense, I'm not much of a programmer. I am using PowerShell to concatenate all of the files within a folder into a single larger file, however when I do this, the text ...
Gabriel Wallace's user avatar
0 votes
0 answers
12 views

Does it make sense to have less than 30 documents and more than 10000 words in Latent Dirichlet Allocation?

We are trying find out topics more covered by political parties in italy using LDA and we have about 30 parties and lots of words (8000). Does it makes sense? Since the words are taken from daily ...
Andrea Fratini's user avatar
0 votes
0 answers
26 views

I am trying to create a corpus using pdf documents

When I am writing the code, I am getting the following error: PDF error: Unknown Metadata type: 'XMP' PDF error: Unknown Metadata type: 'XMP' corp <- Corpus(URISource(files), ...
Sanjay Fuloria's user avatar
0 votes
0 answers
35 views

Converting a dataframe with metadata columns to a corpus using tm in R

Issue with converting data frame to corpus Hello, I have a df with 4 columns (1 is the text and the other three are some additional info). I would like to convert the df into a corpus so that text ...
Rastko M.'s user avatar
0 votes
1 answer
43 views

How to expand a .RDS text corpus using R and Readr package

i am attempting to expand a text corpus that was made available to me. The file itself is a .RDS file, and i need to expand it using the text from 20 different PDF documents, where 1 PDF file is its ...
Roald Andre Kvarv's user avatar
1 vote
0 answers
35 views

Sentence tokenization splits sentences where there is dialogue followed by 'he/she said', how can I fix this?

I am extracting sentences from pdfs into an excel spreadsheet. It handles most cases fine, but when there is a dialogue which ends in ? or ! and is followed by something like 'he said.' it splits the ...
Paige Cox's user avatar
0 votes
0 answers
21 views

Deleting all text up to a specific string in Notepadqq

I'm trying to remove every character up to and including a specific string of text on every line of a text file in Notepadqq (Not Notepad++), and am looking for a way to automate ðis task. I have ...
IndigoGollum's user avatar
3 votes
0 answers
68 views

Stylometry with R-Studio: Error with command oppose from package "stylo" - why is my corpus not usable?

I run a stylometric analysis of poetry in R Studio using the package "stylo". I want to compare two sets of texts via function oppose. I created subdirectories in my working directory for ...
xusha666's user avatar
0 votes
0 answers
61 views

Calculate Term Frequency and Inverse Document Frequency for a very large data in Rstudio

I am doing a code trying to calculate the Term Frequency and Inverse Document Frequency (TF-IDF) for a very large file. The root file is indie_mar_23_0 which is a "character" class file I ...
Anh Duy's user avatar
  • 23
1 vote
1 answer
162 views

How to select most efficient text similarity/distance algorithm to detect word relatedness in natural languages? [closed]

I've done some research on different Python libraries and on algorithms used to measure text distance/similarities: Levenshtein distance, Jaro-Wrinkler, Hamming, etc. None of them seems to work well ...
jfontana's user avatar
  • 143
0 votes
0 answers
7 views

couldn't create dictionary of corpus frequency

corpus = [] pstem = PorterStemmer() for i in range(df['cleaned_tweets'].shape[0]): tweet= df['cleaned_tweets'] corpus.append(tweet) print(corpus) print('corpus created sucessfully') rawTexdata ...
Sara's user avatar
  • 1
0 votes
0 answers
21 views

R text mining example fail on iconv

I'm using R4.2.2 on Window and I'm trying to work through the example on text mining here: https://medium.com/@SAPCAI/text-clustering-with-r-an-introduction-for-data-scientists-c406e7454e76 It fails ...
elbillaf's user avatar
  • 1,974
0 votes
1 answer
79 views

How to develop a corpus(corpus analysis)

I am goingt to build a linguistic corpus, but i don't understand which technologies should i use for it. Is it true, that for developing a courpus for any language i necessarily have to use IMS Corpus ...
Murad Mammadzada's user avatar
0 votes
0 answers
28 views

With R's tm package, VectorSource produces a list instead of corpus

The following code ... library(tm) vectorOfText <- c("twas brillig and the slithey toves", "did gyre and gimble in the wabes") names(vectorOfText) <- c(&...
mnr's user avatar
  • 456
0 votes
1 answer
31 views

How can you exclude certain words before periods from being used as sentence breaks in quanteda's corpus_reshape?

In some cases, certain periods are mistakenly used as sentence breaks when using corpus_reshape. I have a corpus from the pharmaceutical industry and in many cases "Dr." is mistakenly used ...
paul's user avatar
  • 13
1 vote
2 answers
1k views

Error when working on a corpus: missing value where TRUE/FALSE needed

My plan is to work on a corpus of data in R, using the Quanteda package. I'm using the poliblogs.csv database accessible from here, with the script below. In the past, the script worked smoothly, but ...
A.G.'s user avatar
  • 11
1 vote
2 answers
146 views

Grouping text data in a corpus by a data frame variable

I have a data frame in R with a column that I need to do basic text analysis on. I am able to do this modifying the code as needed from this source. However, I now need to do this same analysis but ...
new2data's user avatar
  • 117
0 votes
1 answer
108 views

Merging words in a wordcloud made in R

I have created a word cloud with the following frequency of terms: interesting interesting 21 economics economics 12 learning learning 9 learn learn 6 taxes ...
Jess's user avatar
  • 1
0 votes
0 answers
19 views

Does corpus take into account of the word frequency in each document, or just the word frequency in all documents?

As stated in the title, I have some understanding issues regarding corpus, as far as what I've read from articles and guides, it seems that the corpus only contains the frequency of words in all ...
Qaaqq11282211's user avatar
0 votes
1 answer
34 views

Tool for detecting differences between text passages from two different groups

I have text data from two different groups. In total I have around 4000 text passages with around 300 words. I am searching for a tool that allows me to analyze the difference between these two groups....
Irazall's user avatar
  • 117
0 votes
0 answers
106 views

TFIDF model created by TfidfVectorizer contains words which are not in the corpus it was trained on

I have trained a TF-IDF model on a specific corpus. This corpus is a set of string, which has been cleaned. I have gotten rid of stopwords, numbers, done some stemming, etc. The TF-IDF is trained on ...
Dino's user avatar
  • 1
1 vote
0 answers
38 views

World Cloud with two-word terms?

I am trying to create a wordcloud for my internship and I'm struggling with it, i would appreciate any help! I have a repository of 200 pdf documents, and I have to build a wordcloud of the terms that ...
marcosmm's user avatar
-1 votes
2 answers
46 views

not be able to remove stopword

i have stopword list but somehow the program cannot remove the stopword in the corpus the code i use stop_factory = StopWordRemoverFactory() more_stopword = ['selamat','halo','hallo','hi'] dok_word = ...
abbym's user avatar
  • 29
0 votes
0 answers
17 views

Dataframe meta-data not attaching with TextResuseCorpus command in R

I have a dataframe (twitterdata1) with two columns that looks something like this: full_name text John I love my wife, kids, and billiards club Larry Best damn veterinarian in the midwest Tom ...
lwe's user avatar
  • 349
0 votes
0 answers
26 views

Build a Corpus with tokens corpuses pfd texts as variable in a development Quanteda function

I am a beginner r user and I would need some help with my project. I want to build a quanteda corpus for PDF text analysis. I have developed a function that gonna help me rebuild the corpus by putting ...
Mad Max's user avatar
1 vote
1 answer
45 views

Finding training data for tweets with labels: positive, negative, neutral

Are there any training data of tweets with labels: positive, negative, neutral as following: "tweet 1" positive "tweet 2" positive "tweet 3" neutral "tweet 4" ...
TheRi's user avatar
  • 67
0 votes
1 answer
66 views

output issues with NLTK CHILDES Corpus Reader in Python

I am working on a child language project and would like to use the CHILDES Corpus Reader package to analyze children's language data. However, the methods do not output anything. I am trying with the ...
Arthur_Kan's user avatar
0 votes
1 answer
124 views

Computing frequency of words for each document in a corpus/DFM for R

I want to replicate a measure of common words from a Paper in R. They describe their procedure as follows: "To construct Common words,..., we first determine the relative frequency of all words ...
yasiko41's user avatar
1 vote
2 answers
109 views

How to extract the date from PDF file names to a data set?

I am trying to extract the date from multiple PDF's to create a date column in a dataset. I have a folder holding all the pdf's and am trying to do a topic modelling over a time period, hence I need ...
blablabla 's user avatar
0 votes
1 answer
101 views

Data from data frame to corpus in R

I need to get data from a data frame that I have to a corpus, but I get an error "Error in VCorpus(DataframeSource(df)) : could not find function "VCorpus"" Do you have an idea why?...
user1986315's user avatar
1 vote
0 answers
28 views

Orange3 txt files don't seem to work with logistic regression

I just started working with Orange3 doing text analytics. When I create a corpus made up of .txt files I can get the workflow to function where I can generate a word cloud and I can view the corpus ...
crv.hagiograph's user avatar
0 votes
1 answer
39 views

R Text Mining: find correlations among words within a certain distance of a keyword

I have been following this example and was wondering if it is possible to draw the figure 4.4 for combinations of words that were within 10 words of the keyword instead of words that are right next to ...
user3390169's user avatar
  • 1,025
0 votes
0 answers
105 views

How to read a text file folder to convert to corpus for word analysis?

I'm using Rstudio. And I have a folder ("/datahome/mario/datasets/robson/CLAY_Arquivos_PDF/TXT") with 5012 articles in ".txt", which need to be transformed into a corpus, into a ...
Robson Costa's user avatar
0 votes
0 answers
107 views

I am unable to read my database from the PDF file

I'm doing sentiment analysis on articles. I am transforming my PDF file into corpus; these files are in a Folder. #reference: https://data.library.virginia.edu/reading-pdf-files-into-r-for-text-mining/...
Robson Costa's user avatar
1 vote
2 answers
139 views

R text analysis: Counting occurences of any combinations of words from two different keyword lists with a given distance of each other

Thanks for reading. For a reserach project, I'm doing some text analysis. We are analyzing large texts (company reports) and I'm looking to count keyword frequencies within that text. However, I have ...
dmort's user avatar
  • 181
2 votes
1 answer
471 views

Most frequent phrases from text data in R

Does anyone here have experience in identifying the most common phrases (3 ~ 7 consecutive words)? Understand that most analysis on frequency focuses on the most frequent/common word (along with ...
Luther_Proton's user avatar
1 vote
0 answers
55 views

Function Corpus in Quanteda doesn't work because of a kwic objects

First of all, I'm working on a big data project which consists in analyze some press URLs to detect the most popular topics. My topic is about football (Mbappe contract) and I collected 180 URLs from ...
Cristina's user avatar
0 votes
1 answer
49 views

Text file classification in r from KNN to SVM

my problem is that i don't understand how to go to SVM, actually i have 20% mistakes in KNN so i want to improve this stat, i work on html files that i put in a VCorpus, clean, put in a DTM, figure ...
Thechrome37's user avatar
1 vote
0 answers
70 views

Making a biagram table from Corpus

i am trying to make a biagram table from a corpus with a text file which contains 100 sentences but i am unable to do it. right now i only have code with showing how many sentences that text file has. ...
John Will's user avatar
1 vote
1 answer
122 views

Database that allows full text search in O(1)

I have a database of documents where searching quickly for keywords and patterns would be very useful to have. I know of "Burrows–Wheeler transform"/FM-index. I wonder if there are any ...
Yorai Levi's user avatar
3 votes
1 answer
459 views

How to solve missing words in nltk.corpus.words.words()?

I have tried to remove non-English words from a text. Problem many other words are absent from the NLTK words corpus. My code: import pandas as pd lst = ['I have equipped my house with a new [xxx]...
Lud's user avatar
  • 57
0 votes
2 answers
137 views

How do I create two subsets out of a corpus based on multiple keywords?

I am working with a large body of political speeches in quanteda and would like to create two subsets. The first one should contain one or more from a list of specific keywords(e.g. "migrant*&...
Luke's user avatar
  • 5
0 votes
1 answer
84 views

how to convert a matrix to BoW format?

I am trying to convert a matrix to the type that can be received by gensim. AuthorTopic Model, which means I should convert a matrix to a sparse vector. I have already tried several functions in ...
Yu Fu's user avatar
  • 1
0 votes
1 answer
56 views

Best way of Splitting up and then putting back a huge dataframe?

Basically I have a dataframe with 300 000 rows, which I need to do sentiment analysis on. The dataframe contains reddit comments, so I need to transform them into corpus, but the function I have only ...
JoshBob's user avatar
  • 81
1 vote
2 answers
574 views

Relationship between vocab size and complexity

I have 2 corpuses, if one has a larger vocabulary size than the other, does it mean its language is more complex? Apart from complexity of the language, what else can effect the size of the vocabulary ...
daniel's user avatar
  • 61

1
2 3 4 5
15