Questions tagged [tokenize]
Tokenizing is the act of splitting a string into discrete elements called tokens.
622
questions with no upvoted or accepted answers
19
votes
0answers
910 views
How do I implement a custom UITextInputTokenizer?
I have a UITextView and am using its tokenizer to check which words the user taps on.
My goal is to change what the tokenizer thinks of as a word. Currently it seems to define words as consecutive ...
6
votes
0answers
1k views
what should I choose? ngram filter,ngram tokenizer or fuzzy match query?
I am a little confused about usage of filter, tokenizer vs query. I can select ngram filter or tokenizer during indexing (through an analyzer) I can also use multi_field to store different variation ...
5
votes
1answer
485 views
Solr(Lucene) is indexing only the first document after adding a custom TokenFilter
I created a custom token filter which concatenates all the tokens in the stream. This is my incrementToken() function
public boolean incrementToken() throws IOException {
...
5
votes
1answer
1k views
Catching errors thrown by token_get_all (Tokenizer)
PHPs token_get_all function (which allows converting a PHP source code into tokens) can throw two errors: One if an unterminated multiline comment is encountered, the other if an unexpected char is ...
4
votes
1answer
2k views
Token indices sequence length error when using encode_plus method
I got a strange error when trying to encode question-answer pairs for BERT using the encode_plus method provided in the Transformers library.
I am using data from this Kaggle competition. Given a ...
4
votes
1answer
1k views
Why do wc -w and Python's len(text.split()) give a different result?
In which circumstances would the Unix command line utility 'wc' and Python's len(text.split()) give a different result?
A bit of context, although it shouldn't be relevant because the only thing we ...
4
votes
0answers
163 views
Is there a way to highlight function calls using Pygments (or another library)?
I was quite disappointed to discover that functions calls were not highlighted using Pygments.
See it online (I tested it with all available styles)
Builtin functions are highlighted but not mine.
...
4
votes
0answers
637 views
Python ast and tokenizer
In python how would you use the tokenize module in conjunction with the ast module? Both use different node types how would you correlate the two together? Is there a way? Possibly the parser module?
4
votes
1answer
903 views
Custom tokenizer solr only is invoked at the first
I created a custom tokenizer, it seem work fine by checking with admin/analysis.jsp and with system.out log. However when I perform querying in the field which use this custom tokenizer, I saw that ...
4
votes
0answers
1k views
Matching all tokens in a Multivalued Field in Solr
I'm trying an approach to resolve a search requirement by tokenizing a serial string into individual terms inside of a multivalued field.
basically, each document may contain multiple values for ...
3
votes
0answers
185 views
Time Efficient Frequency Count (top 500 words) of a Large Directory in Python?
I wish to calculate the top 500 words (in terms of occurrence) for all the files (a maximum of 15,000 files) of a directory. I am experimenting the same on small sized directories containing 1000 ...
3
votes
1answer
476 views
How to prepare training data for OpenNLP to Tokenize the token that contains more than one word?
In some language (for example: Vietnamese), some vocabulary consists of multiple words. So that some tokens which contain more than one word can be tokenized not just using the white space.
I have ...
3
votes
0answers
49 views
Microsoft Natural Language List: is there an equivalent to “no language” or “raw unicode” or “Language neutral”?
Is there something equivalent to "non-language" or "raw" in the Microsoft Natural Language List, which would have the effect of causing the word-breaking (i.e. tokenizing) algorithms to use only the ...
3
votes
0answers
270 views
Getting “expected string or bytes-like object” for a python for loop
This is the code i am trying to run.
src_doc = open("set1_neg.txt", 'rb')
words = []
for line in src_doc.readlines():
words.append(line)
user_input= word_tokenize(words)
print(words)
But, i keep ...
3
votes
1answer
534 views
How to test a Lucene Analyzer?
I'm not getting the expected results from my Analyzer and would like to test the tokenization process.
The answer this question: How to use a Lucene Analyzer to tokenize a String?
List<String> ...
3
votes
0answers
153 views
How do i create a new codemirror mode?
So i recently finished my own functional programming language for a project and I want to create a codemirror mode for the language as it transcompiles to javascript. I have written a tokenizer for it ...
3
votes
1answer
552 views
How to create a Tokenizing Control for UWP as known from Outlook when using To, Cc and Bcc
There is a great article about how to write a Tokenizing Control for WPF here: Tokenizing control ā convert text to tokens
But how is this accomplished in an UWP App? The Windows 10 UWP Mail client ...
3
votes
1answer
671 views
Auto-complete select multiple tags in asp.net
Anyone can tell me how can i use tokenizing in auto-complete for multiple selection, I am make you sure that, i want only with asp.net web from web service
My Code:
$(function () {
// Web ...
3
votes
0answers
150 views
Associating Eclipse's JDT tokenizer's token identifiers with its JDT parser's AST node identifiers
I'm trying to find the common ground between Eclipse's JDT tokenizer and its JDT parser.
Details of my code setup are as follows. For tokenizing, I'm using an object of type PublicScanner in package ...
3
votes
2answers
42 views
Divide authorship string into authors
I am trying to devise a smart way to divide a string of authors, the problem is that the string uses commas not only for the division between authors, but also for the division between their first and ...
3
votes
1answer
2k views
WPF control for tokenizing
I didn't found any wpf library that contains control for tokenizing (tagging) text input. Perhaps I can't find the library, because I incorrectly call this component. How it is called or where there ...
3
votes
0answers
1k views
tokenize error in python
I have this sample program from python-docx library example-extracttext.py to extract text from a docx file.
#!/usr/bin/env python
"""
This file opens a docx (Office 2007) file and dumps the text.
...
3
votes
2answers
718 views
Regular expression for a fake programming language
I have an assignment to create a lexical analyzer that that translates the language into a series of tokens. I'm using java.util.regex to go through a string finding the different tokens and i put ...
3
votes
2answers
834 views
Good java tokenizers?
I am curious about any good tokenizers out there for java OTHER THAN those included in the standard libs such as StreamTokenizer and StringTokenizer, which in my opinion are not very good.
The only ...
3
votes
3answers
834 views
OpenNLP Tokenizer does not detect words that belong together?
I am new to NLP and I came across OpenNLP. From my understanding tokenization means segmenting text into words and sentences. Words are often separated by white spaces but not all white spaces are ...
2
votes
0answers
29 views
WordNet Lemmatizer don't it lemmatize all inflections of a word in CSV file
I want to lemmatize all words in sentences(CSV file, and this file contains tweets) for our goal to reduce the frequency of words. By using wordnet lemmatizer. I know defult for lemmatizer is NOUN but ...
2
votes
0answers
39 views
How to save Tensorflow tokenized text on Mlflow for predictions?
Did anyone had saved tf tokenzied data on Mlflow and loaded it for prediction in production code? I know there is a method to write as json or pickle but I have no idea how to send it to Mlflow and ...
2
votes
0answers
36 views
Tweet Tokenizing A column in Dataframe ; Getting Error
I have a dataset with comment_texts. I want to tokenize these using code below:
from nltk.tokenize import TweetTokenizer
tt = TweetTokenizer()
df['tokenized_text'] = df["comment_text"].apply(tt....
2
votes
0answers
68 views
Wordpiece Tokenization Model
Can somebody tell me how exactly the wordpiece model work ? I am having some hard time trying to understand how exactly the wordpiece model is working. I understand the BPE that it is based on merging ...
2
votes
0answers
85 views
Using Tokenizer in TextLineDataset#map
I try to use TextLineDataset and Tokenizer together but I run into problems when using it with #map. I get an error that it is not possible to iterate over a Tensor. I think I understand, that #...
2
votes
0answers
57 views
Tokenizing mathematical expression - how to know how many implicit multiplications were used?
I have an interesting problem. I'm working on a program that will tokenize a mathematical expression, like in the following example:
'(3+4)*sin(45)' -> ['(', '3', '+', '4', ')', '*', 'sin', '(', '45',...
2
votes
0answers
34 views
TensorFlow Tokenize by Unique CSV Row Rather than Unique Word in Row
I'm trying to use tokenizer to classify free text into select categories.
For feature, I'm using this:
x_tokenizer = Tokenizer()
x_tokenizer.fit_on_texts(x)
x_train = x_tokenizer.texts_to_matrix(...
2
votes
0answers
518 views
LookupError: Attempted to load tokenizers/punkt/english.pickle
I am using pycharm IDE. I dowloaded nltk using file-> settings-> Project Interpreter -> nltk.
After running my code below I have received an Lookup error
My code
# My Code
def get_tags(s):
...
2
votes
1answer
249 views
How to tokenize all the files from all the subfolders
I have a dataset which consists of 50 subfolders and each of these subfolders has 20-30 files without extension. What I wanted to do is tokenizing the texts in the files for each subfolders, and write ...
2
votes
0answers
111 views
Adding rare languages to Apache Solr
We would like to add support for Filipino, Lao, Malaysian and Vietnamese to our Solr Index (Solr v4 and Solr v6) and would be delighted to get hints on what Tokenizer to use.
For Lao I found that the ...
2
votes
1answer
931 views
How do I tokenize array of json objects?
Hi I need to tokenize an array of json objects but I'm not sure how to go about doing that.
Currently, I have this snippet:
StringTokenizer tokenizer = new StringTokenizer(request, "{}:,\"");
Map&...
2
votes
1answer
1k views
How to add an tokenizer exception for whitespaces in Spacy language models
The following is my code where I take an user input.
import en_core_web_sm
nlp = en_core_web_sm.load()
text = input("please enter your text or words here")
doc = nlp(text)
print([t.text for t in doc]...
2
votes
1answer
144 views
what is ambiguity in alphabet in automata theory?
I am just new in automata field. I have read many articles, and seen many video. I stuck in some first topics. It can be easy for others. but after spending a lot of time,i am still unable to ...
2
votes
0answers
135 views
span_tokenize gives generator object as output
I have written a simple piece of code to see exactly how the span_tokenize function works. Documentation for this can be found here: http://www.nltk.org/api/nltk.tokenize.html
Here is my piece of code
...
2
votes
1answer
1k views
Flesch-Kincaid readability test in python
I need help with this problem I'm having. I need to write a function that returns a FRES (Flesch reading-ease test) from a text. Given the formula:
In other words my task is to turn this formula into ...
2
votes
2answers
5k views
Python : How to tokenize from file?
I'm a newbie in python. I would like to know how to tokenize twitter data from file.
My code is :
with codecs.open('example.csv', 'r',"utf-8") as f:
for line in f:
tweet = f.readlines()
...
2
votes
0answers
117 views
How to measure the response time in Spark using Scala
I am using Tokenizer and StopWordsRemover in spark-shell (with Scala) following these links:
https://spark.apache.org/docs/latest/ml-features.html#tokenizer
https://spark.apache.org/docs/latest/ml-...
2
votes
0answers
745 views
check if a string contains non-english words in java
I would like to know if there is a way to check if a given string may contain English word and returns the non-English substrings.
What I've tried so far is to load an English dictionary to a file and ...
2
votes
0answers
196 views
How to dump Parse tree and token info from GCC?
I want to dump token information (list of all tokens for a c file) and parse tree from GCC compiler. I went through the gcc documentation but it was not clear on how to do it.
How can I get this ...
2
votes
0answers
1k views
Extracting all possible ngrams in R tm document term matrix
I am using the "tm" package in R to create a term document matrix. Then I use "RWeka" to extract trigrams as specified in the code below
myCorpus <- VCorpus(VectorSource(reddata$Tweet))
#create ...
2
votes
0answers
200 views
Parsing conditions in a custom DSL using a tokenizer
I am trying to parse a "rules string" provided over an API in PHP.
Given that I have an array of items such as
$items = [41, 42, 51, 54, 65, 12];
I wish to evaluate this expression
({41} AND {51})...
2
votes
1answer
956 views
How to tokenize Spanish text with NLTK or Pattern-es
Basically the issue I am having is with separating object and direct object pronouns from verbs.
Ie 'aprenderlo' should ideally be tokenized as two separate entities, 'dimelo' should be tokenized as ...
2
votes
0answers
469 views
Tokenizing a Hashtag containing underscores with Elasticsearch
We have been tokenizing twitter style hashtags with a word_delimiter filter. For a hashtag '#SomeHashtag' we expect users might search for #SomeHashtag or SomeHashtag.
For #SomeHashtag our analyzer ...
2
votes
1answer
4k views
Keras - Text preprocessing
The goal:
To generate text from an authors style.
Input: an authors work to train on, a seed for a prediction
Output: generated text from that seed
Question about the embedding layer in keras:
I ...
2
votes
0answers
118 views
Stanford POS tagger won't lemmatize pre-tokenized text
I'd like to lemmatize and POS tag text according to the FAQ. This command provided in the FAQ works correctly:
$ java -cp "*:lib/*" edu.stanford.nlp.tagger.maxent.MaxentTagger \
-model models/...