Questions tagged [tokenize]
Tokenizing is the act of splitting a string into discrete elements called tokens.
2,520
questions
0
votes
1answer
13 views
Is there a way to save my processed Dataframes in Jupyter Notebooks
I'm new to Jupyter Notebooks and I was wondering if there is a way to save my (Preprocessed Data) instead of re running the file each time. I'm working on a NLP project where steps like (Lemmatization ...
0
votes
1answer
43 views
Python based tokenizers for non English languages
We currently build text based models using scikit learn library in Python. Scikit learn has default support of tokenization for English laungauge. We want to add support for non-english languages as ...
0
votes
0answers
96 views
Is there any way to get the subword counts dictionary from a tensorflow_datasets SubwordTextEncoder object?
I am using the TensorFlow SubwordTextEncoder class to get subwords from a corpus. Besides the subwords (tokens) I need to have the individual count (case sensitive) for each subword token found in the ...
8
votes
2answers
7k views
Text tokenization with Stanford NLP : Filter unrequired words and characters
I use Stanford NLP for string tokenization in my classification tool. I want to get only meaningful words, but I get non-word tokens (like ---, >, . etc.) and not important words like am, is, to (...
2
votes
1answer
32 views
How to ignore double slashes while making a search query in elastic search?
For example, in the elastic search document, the string is stored as \\(\\log_4(3x^2+11x)=1\\)
The search query what I want to make it work is (log_4(3x^2+11x)=1).
What is the best way to make this ...
25
votes
10answers
54k views
Python - RegEx for splitting text into sentences (sentence-tokenizing) [duplicate]
I want to make a list of sentences from a string and then print them out. I don't want to use NLTK to do this. So it needs to split on a period at the end of the sentence and not at decimals or ...
0
votes
1answer
56 views
Spacy - modify tokenizer for numeric patterns
I have seen some ways to create a custom tokenizer, but I am a little confused. What I am doing is using the Phrase Matcher to match patterns. However, it would match a 4-digit number pattern, say ...
1
vote
1answer
86 views
spaCy SPECIAL-1 token overriding suffix rule causing annotation misalignment
My text is like
'Laboratories, Inc.'
Which gets tokenized like
Laboratories TOKEN
, SUFFIX
Inc. SPECIAL-1
However annotations usually don't include suffix characters like '.'
So I ...
0
votes
0answers
18 views
Does spark has multi-language tokenization support?
Does anyone have experience in multi-language tokenizer in spark? I searched spark NLP, spark ML-lib and cannot find any support for multi-language tokenization. If you happen to know some great tool, ...
0
votes
2answers
88 views
Split string with regex by new lines, symbols and withspaces in python
I'm new to regex library, and I'm trying to make from a text like this
"""constructor SquareGame new(){
let square=square;
}"""
This outputs a list:
['constructor', '...
0
votes
2answers
97 views
“asciifolding” with tokenizer “pattern” in elasticsearch
can anyone tell me why "asciifolding" doesn't work on the "pattern" tokenizer in my mapping below?
I need to use the "pattern" tokenizer but I also need to not ...
0
votes
1answer
103 views
Keras Tokenizer sequence to text changes word order
I am training a model on DUC2004 and Giga word corpus, for which I am using Tokenizer() from keras as follows:
tokenizer = Tokenizer(num_of_words)
tokenizer.fit_on_texts(list(x_train))
#convert text ...
0
votes
0answers
71 views
AttributeError: 'list' object has no attribute 'map' in python while counting number of tokens in a list
this is my list vendorRDD
I am trying to count the totatal number of tokens (in this case this should return 10)
and, this is my code:
count=vendorRDD.map(lambda x:len(x[1])).sum()
However, i am ...
1
vote
1answer
60 views
How to tokenize word with hyphen in Spacy
I want to tokenize bs-it to ["bs","it"] using spacy, as I am using it with rasa. The output which I get from is ["bs-it"]. Can somebody help me with that?
0
votes
0answers
582 views
UITextView Possible to get line number?
Is there an easy way ie convenience method to get the line number in a textview from the position perhaps using UITokenizer?
Something like:
UITextView *textView = (UITextView *)recognizer.view;
...
1
vote
0answers
496 views
How to slice string depending on length of tokens
When I use (with a long test_text and short question):
from transformers import BertTokenizer
import torch
from transformers import BertForQuestionAnswering
tokenizer = BertTokenizer.from_pretrained('...
-1
votes
1answer
19 views
loop string and get value xslt
I have html input file with that i need to process style information and design XSLT
<div style="width: 3%; min-width: 10px; padding: 0px; border-color: black; vertical-align: top; word-wrap: ...
1
vote
1answer
76 views
Spark equivalent to Keras Tokenizer?
So far, I pre-process text data using numpy and build-in fuctions (such as keras tokenizer class, tf.keras.preprocessing.text.Tokenizer: https://keras.io/api/preprocessing/text/).
And there is were I ...
1
vote
1answer
24 views
Regex split at all punctuations and English character sequences and keep delimiters with zhuyin in Python
How do I tokenise a string with a fixed set of symbols (zhuyin), punctuations, and English characters into zhuyin sequences (space delimited but sometimes joined by punctuations or English characters),...
0
votes
0answers
43 views
Tokenize in Python
I am trying to build a function that python that allows me to tokenize a character string. I have performed the following function:
def tokenize(string):
words = nltk.word_tokenize(string)
return ...
-1
votes
1answer
38 views
How does spacy split “'s”?
Spacy gives name's as 2 tokens -> name, 's. How could I combine those two tokens? Which rule define the splitting of "'s", infix, or others?
3
votes
3answers
122 views
NLTK word tokenize all words except words with a dash, e.g ('hi-there', 'me-you')
I am not sure how can I use the nltk.word_tokenize method if I want to tokenize everything except the words with a dash (i.e excludes all words that have a dash in between). example:
'hi-there', 'me-...
0
votes
0answers
41 views
In spacy, how to consider “-” as a prefix if it is followed by a date?
I'm trying to improve spacy ner for some cases. I had for example the case : Failing date recognition where the date "2019" is not recognized because it is preceded by a "-".
I checked and spacy ...
2
votes
1answer
959 views
spacy tokenize apostrophe
I am trying to properly split words to fit my corpus. I'm already using this approach which fixes hyphenated words, what I can't seem to figure out is how to keep words with apostrophes for ...
4
votes
5answers
3k views
How could spacy tokenize hashtag as a whole?
In a sentence containing hashtags, such as a tweet, spacy's tokenizer splits hashtags into two tokens:
import spacy
nlp = spacy.load('en')
doc = nlp(u'This is a #sentence.')
[t for t in doc]
output:
...
1
vote
1answer
751 views
Is it possible to change the token split rules for a Spacy tokenizer?
The (German) spacy tokenizer does not split on slashes, underscores, or asterisks by default, which is just what I need (so "der/die" results in a single token).
However it does split on parentheses ...
5
votes
2answers
2k views
How to apply tokenization to a TensorFlow Dataset?
I am working with the cnn_dailymail dataset which is part of the TensorFlow Datasets. My goal is to tokenize the dataset after applying some text preprocessing steps to it.
I access and preprocess ...
1
vote
1answer
47 views
Can I apply custom token rules to tokens split by prefixes in spaCy?
I customized a spaCy Tokenizer with additional rules and prefixes to treat w/ and f/ as with and for, respectively. The prefixes are correctly splitting them off, but the custom rules for lemmas and ...
8
votes
1answer
5k views
Spacy custom tokenizer to include only hyphen words as tokens using Infix regex
I want to include hyphenated words for example: long-term, self-esteem, etc. as a single token in Spacy. After looking at some similar posts on StackOverflow, Github, its documentation and elsewhere, ...
1
vote
0answers
80 views
Spark Scala: SQL rlike vs Custom UDF
I've a scenario where 10K+ regular expressions are stored in a table along with various other columns and this needs to be joined against an incoming dataset. Initially I was using "spark sql rlike" ...
1
vote
0answers
31 views
What is the best way to save keras models and tokenizers for NLP?
I have been using the following snippets for many time:
For saving:
# saving model and tokenizer
with open('./static_models/tokenizer.pickle', 'wb') as handle:
pickle.dump(tokenizer, handle, ...
3
votes
1answer
1k views
Tokens to Words mapping in the tokenizer decode step huggingface?
Is there a way to know the mapping from the tokens back to the original words in the tokenizer.decode() function?
For example:
from transformers.tokenization_roberta import RobertaTokenizer
...
1
vote
0answers
331 views
How to reduce vocabulary using tf.keras.preprocessing.text.Tokenizer?
I'm using tf.keras.preprocessing.text.Tokenizer to build a vocabulary for my corpus (5 million documents). The tokenizer finds 145K tokens. The issue is that the embedding layer has far too many ...
55
votes
11answers
218k views
Is there a function to split a string in PL/SQL?
I need to write a procedure to normalize a record that have multiple tokens concatenated by one char. I need to obtain these tokens splitting the string and insert each one as a new record in a table. ...
0
votes
0answers
18 views
Lexer token conflict between “GreaterThan” and “BinaryShift” “>>”
Imagine that TypeScript example:
let test = 1 >> 6
let arr: Array<Array<String>> = [["Hello", "World"], ["snipped"]]
How does the lexer differ the token "GreaterThan" and "...
0
votes
0answers
37 views
How to prioritize exact match using nGram tokenizer?
I am wondering, how to rank an exact match higher than the ngram matches.
For instance:
If I search for asus
It scores higher casual than asus or if I search for app it gives me first laptop and than ...
0
votes
1answer
588 views
How to tokenize a list of lists in python
After scraped online, I got the children's books in text format from Gutenberg.com.
Now I would like to analyze the words. But I failed to do the tokenization since the content turned out to be list ...
0
votes
2answers
160 views
How to tokenize python code using the Tokenize module?
Consider that I have a string that contains the python code.
input = "import nltk
from nltk.stem import PorterStemmer
porter_stemmer=PorterStemmer()
words=["connect","connected","connection","...
1
vote
0answers
34 views
elasticsearch combining muti-word token to single token
Basically, Let's say I have a list phrases in vocabulary
- University of Texas Dallas
- University of Tokyo
- University of Toronto
Let's say I have 3 documents
- doc1: I study at University of ...
0
votes
1answer
546 views
Bert Tokenizer is not working despite importing all packages. Is there a new syntax change to this?
Trying to run the tokenizer for Bert but I keep getting errors. Can anyone help where I am going wrong.
FullTokenizer = bert.bert_tokenization.FullTokenizer
bert_layer = hub.KerasLayer("https://tfhub....
1
vote
2answers
592 views
How to stop BERT from breaking apart specific words into word-piece
I am using a pre-trained BERT model to tokenize a text into meaningful tokens. However, the text has many specific words and I don't want BERT model to break them into word-pieces. Is there any ...
19
votes
9answers
79k views
Split string by a character? [duplicate]
How can I split a string such as "102:330:3133:76531:451:000:12:44412 by the ":" character, and put all of the numbers into an int array (number sequence will always be 8 elements long)? Preferably ...
0
votes
0answers
11 views
Is there any pre-fitted Keras tokenizer?
In Text data preprocessing in keras, we use Tokenizer class. But if we fit the training data in the Tokenizer, then any new word in the testing set does not get identified when generating sequences. ...
0
votes
0answers
164 views
Faster ways to POS tag using Python
I have ~2 million rows of text. Each row can be one or multiple sentences. I need to POS tag the entire corpus. The corpus is a list of strings, for example:
corpus = ["I am awesome. I really am.", "...
0
votes
0answers
42 views
Sentence segmentation rule not working as expected
I have created my own simple sentence segmentation rule to sentencize on a new line (and keep the default segmentation rules as well):
import spacy
nlp = spacy.load('en_core_web_sm')
def ...
0
votes
0answers
19 views
How to get offsets for ElectraTokenizer
I am trying to use ELECTRA model from HuggingFace library. However, I need to get the offsets for ElectraTokenizer, which can be done straightforward, according to docs. Does anyone know how can I get ...
2
votes
1answer
135 views
strtok() returns NULL on subsequent calls even though there are more tokens to be read
I'm having an issue in my program where calling strtok(NULL, "\r\n"); returns a NULL after I made a function call even though there are still tokens in the stream. I've looked over this for a while ...
8
votes
3answers
6k views
get indices of original text from nltk word_tokenize
I am tokenizing a text using nltk.word_tokenize and I would like to also get the index in the original raw text to the first character of every token, i.e.
import nltk
x = 'hello world'
tokens = nltk....
1
vote
2answers
244 views
Words used in Bag of words along with frequency in keras tokenizer
I just want to know, how to identify or get a list of words along with their frequency that are considered for bag of words by keras tokenizer.
Consider the below example
from tensorflow.keras....
0
votes
0answers
10 views
File Parsing and Tokenization with “\t”
I'm currently playing with the following git repo - https://github.com/threelittlemonkeys/rnn-encoder-decoder-pytorch
I'm having issues setting up an input file. According to the github, we need ...