Questions tagged [quanteda]
The `quanteda` package provides a fast, flexible toolset for the management, processing, and quantitative analysis of textual data in R.
402
questions
6
votes
1answer
393 views
Stem completion in R replaces names, not data
My team is doing some topic modeling on medium-sized chunks of text (tens of thousands of words), using the Quanteda package in R. I'd like to reduce words to word stems before the topic modeling ...
5
votes
1answer
4k views
Quanteda: how to remove my own list of words
Since there is no ready implementation of stopwords for Polish in quanteda, I would like to use my own list. I have it in a text file as a list separated by spaces. If need be, I can also prepare a ...
5
votes
1answer
3k views
what is the best way to remove non-ASCII characters from a text Corpus when using Quanteda in R?
I am in dire need. I have a corpus that I have converted into a common language, but some of the words were not properly converted into English. Therefore, my corpus has non-ASCII characters such as U+...
5
votes
1answer
364 views
R: removal of regex from Quanteda DFM, Sparse Document-Feature Matrix, object?
Quanteda package provides the sparse document-feature matrix DFM and its methods contain removeFeatures. I have tried dfm(x, removeFeatures="\\b[a-z]{1-3}\\b") to remove too short words as well as dfm(...
5
votes
1answer
2k views
Feature selection in document-feature matrix by using chi-squared test
I am doing texting mining using natural language processing. I used quanteda package to generate a document-feature matrix (dfm). Now I want to do feature selection using a chi-square test.
I know ...
4
votes
1answer
2k views
tidytext, quanteda, and tm returning different tf-idf scores
I am trying to work on tf-idf weighted corpus (where I expect tf to be a proportion by document rather than simple count). I would expect the same values to be returned by all the classic text mining ...
4
votes
2answers
537 views
Naive Bayes in Quanteda vs caret: wildly different results
I'm trying to use the packages quanteda and caret together to classify text based on a trained sample. As a test run, I wanted to compare the build-in naive bayes classifier of quanteda with the ones ...
4
votes
2answers
3k views
Convert dfmSparse from Quanteda package to Data Frame or Data Table in R
I have a dfmSparse object (large, with 2.1GB) which is tokenized and with ngrams (unigrams, bigrams, trigrams and fourgrams), and I want to convert it to a data frame or a data table object with the ...
4
votes
2answers
6k views
Generating all word unigrams through trigrams in R
I am trying to generate a list of all unigrams through trigrams in R to, eventually, make a document-phrase matrix with columns including all single words, bigrams, and trigrams.
I expected to find ...
4
votes
1answer
73 views
How to do fuzzy pattern matching with quanteda and kwic?
I have texts written by doctors and I want to be able to highlight specific words in their context (5 words before and 5 words after the word I search for in their text). Say I want to search for the ...
4
votes
1answer
5k views
How do you use a LIWC-formatted dictionary with the R package Quanteda?
As LIWC software and dictionaries are proprietary, I was pleased to see they seemed to play well with the still-in-development but excellent R package Quanteda.
The documentation for the R package ...
3
votes
1answer
60 views
Merge two dataframe by rows using common words [duplicate]
df1 <- data.frame(freetext = c("open until monday night", "one more time to insert your coin"), numid = c(291,312))
df2 <- data.frame(freetext = c("open until night"...
3
votes
1answer
7k views
Create dfm step by step with quanteda
I want to analyze a big (n=500,000) corpus of documents. I am using quanteda in the expectation that will be faster than tm_map() from tm. I want to proceed step by step instead of using the ...
3
votes
2answers
180 views
Logical combinations in quanteda dictionaries
I'm using the quanteda dictionary lookup. I'll try to formulate entries where i can lookup logical combinations of words.
For example:
Teddybear = (fluffy AND adorable AND soft)
Is this possible? ...
3
votes
1answer
234 views
How to remove stopwords in multiple languages?
I have a corpus with two languages (the language information is saved in the docvar lang) and want to remove stopwords depending on the docvar value.
I am using a substantively nonsensical example to ...