Questions tagged [quanteda]

The `quanteda` package provides a fast, flexible toolset for the management, processing, and quantitative analysis of textual data in R.

Filter by
Sorted by
Tagged with
6
votes
1answer
393 views

Stem completion in R replaces names, not data

My team is doing some topic modeling on medium-sized chunks of text (tens of thousands of words), using the Quanteda package in R. I'd like to reduce words to word stems before the topic modeling ...
5
votes
1answer
4k views

Quanteda: how to remove my own list of words

Since there is no ready implementation of stopwords for Polish in quanteda, I would like to use my own list. I have it in a text file as a list separated by spaces. If need be, I can also prepare a ...
5
votes
1answer
3k views

what is the best way to remove non-ASCII characters from a text Corpus when using Quanteda in R?

I am in dire need. I have a corpus that I have converted into a common language, but some of the words were not properly converted into English. Therefore, my corpus has non-ASCII characters such as U+...
5
votes
1answer
364 views

R: removal of regex from Quanteda DFM, Sparse Document-Feature Matrix, object?

Quanteda package provides the sparse document-feature matrix DFM and its methods contain removeFeatures. I have tried dfm(x, removeFeatures="\\b[a-z]{1-3}\\b") to remove too short words as well as dfm(...
5
votes
1answer
2k views

Feature selection in document-feature matrix by using chi-squared test

I am doing texting mining using natural language processing. I used quanteda package to generate a document-feature matrix (dfm). Now I want to do feature selection using a chi-square test. I know ...
4
votes
1answer
2k views

tidytext, quanteda, and tm returning different tf-idf scores

I am trying to work on tf-idf weighted corpus (where I expect tf to be a proportion by document rather than simple count). I would expect the same values to be returned by all the classic text mining ...
4
votes
2answers
537 views

Naive Bayes in Quanteda vs caret: wildly different results

I'm trying to use the packages quanteda and caret together to classify text based on a trained sample. As a test run, I wanted to compare the build-in naive bayes classifier of quanteda with the ones ...
4
votes
2answers
3k views

Convert dfmSparse from Quanteda package to Data Frame or Data Table in R

I have a dfmSparse object (large, with 2.1GB) which is tokenized and with ngrams (unigrams, bigrams, trigrams and fourgrams), and I want to convert it to a data frame or a data table object with the ...
4
votes
2answers
6k views

Generating all word unigrams through trigrams in R

I am trying to generate a list of all unigrams through trigrams in R to, eventually, make a document-phrase matrix with columns including all single words, bigrams, and trigrams. I expected to find ...
4
votes
1answer
73 views

How to do fuzzy pattern matching with quanteda and kwic?

I have texts written by doctors and I want to be able to highlight specific words in their context (5 words before and 5 words after the word I search for in their text). Say I want to search for the ...
4
votes
1answer
5k views

How do you use a LIWC-formatted dictionary with the R package Quanteda?

As LIWC software and dictionaries are proprietary, I was pleased to see they seemed to play well with the still-in-development but excellent R package Quanteda. The documentation for the R package ...
3
votes
1answer
60 views

Merge two dataframe by rows using common words [duplicate]

df1 <- data.frame(freetext = c("open until monday night", "one more time to insert your coin"), numid = c(291,312)) df2 <- data.frame(freetext = c("open until night"...
3
votes
1answer
7k views

Create dfm step by step with quanteda

I want to analyze a big (n=500,000) corpus of documents. I am using quanteda in the expectation that will be faster than tm_map() from tm. I want to proceed step by step instead of using the ...
3
votes
2answers
180 views

Logical combinations in quanteda dictionaries

I'm using the quanteda dictionary lookup. I'll try to formulate entries where i can lookup logical combinations of words. For example: Teddybear = (fluffy AND adorable AND soft) Is this possible? ...
3
votes
1answer
234 views

How to remove stopwords in multiple languages?

I have a corpus with two languages (the language information is saved in the docvar lang) and want to remove stopwords depending on the docvar value. I am using a substantively nonsensical example to ...

15 30 50 per page
1
2 3 4 5
…
27