Questions tagged [tokenize]
Tokenizing is the act of splitting a string into discrete elements called tokens.
2,520
questions
4
votes
2answers
2k views
How to reconstruct text entities with Hugging Face's transformers pipelines without IOB tags?
I've been looking to use Hugging Face's Pipelines for NER (named entity recognition). However, it is returning the entity labels in inside-outside-beginning (IOB) format but without the IOB labels. So ...
0
votes
0answers
110 views
BERT Chinese NER task: how to tokenize letters in it?
I am working on a Chinese NER project. I am not sure how to split the Chinese sentence to align the labels. To be specific, how to split English-like words in Chinese sentences.
My original sentence ...
1
vote
2answers
45 views
Splitting/Tokenizing a sentence into string words with special conditions
I am trying to implement a tokenizer to split string of words.
The special conditions I have are: split punctuation . , ! ? into a separate string
and split any characters that have a space in them i....
0
votes
1answer
488 views
Resource punkt not found. But, it is downloaded and installed
I have the following columns in a dataframe.
Unnamed: 0, title, publication, author, year, month, title.1, content, len_article, gensim_summary, split_words, first_100_words
I am trying to run this ...
1
vote
0answers
41 views
dplyr wrong output in next word prediction function
I'm trying to reproduce the word prediction algorithm as in https://rpubs.com/akselix/word_prediction
library(quanteda)
library(dplyr)
library(tidyr)
library(tokenizers)
library(splitstackshape)
#...
1
vote
1answer
135 views
Error while using unnest_tokens() while passing a function to the token
Error in unnest_tokens.data.frame(., entity, text, token = tokenize_scispacy_entities, :
Expected output of tokenizing function to be a list of length 100
The unnest_tokens() works well for a ...
2
votes
1answer
25 views
Extracting words from a string into dynamic 2D char array
I have a dynamic char array that contains a string. I'm trying to extract all the words from this string into a dynamic 2d char array. Here's my code:
int rows = 1;
char *input_words = malloc((rows)*...
0
votes
1answer
34 views
Tokenizing string from dynamic array into multiple lines in static 2D char array
I have a dynamic array that holds a string containing '\n' characters, so this string is made up of multiple lines. I'm trying to extract the lines and put them all into a 2D char array and I'm ...
0
votes
0answers
35 views
c++ how to split space separated wstring keeping the values inside quotes?
I am reading a registry key in wstring. The value i get in wstring looks like below :
"C:\Program Files (x86)\Folder\folder\binary.exe" "C:\Program Files
(x86)\folder\folder_cfg.json" "C:\...
0
votes
0answers
33 views
Unable to get logic behind Unigram Model Language used for tokenization in nlp
Quoting from a blog post:
"One important assumption that each subword occurs independently, which is highly unlikely because the occurrences of some subwords should be highly correlated. But this ...
0
votes
0answers
116 views
Android Room Fts4 tokenizer unicode61 and tokenizerArgs remove_diacritics=2 not working
I'm working on implementing a search function using androidx.room
My Fts Entity
import androidx.room.Entity;
import androidx.room.Fts4;
import androidx.room.FtsOptions;
@Entity(tableName = "...
2
votes
1answer
201 views
Python nltk incorrect sentence tokenization with custom abbrevations
I am using nltk tokenize library to split up english sentences.
Many sentences contain abbreviations such as e.g. or eg. thus I updated the tokenizer with these custom abbreviations.
I found a strange ...
0
votes
1answer
50 views
error while trying to tokenizing a string and insert each part in an array
while (fgets(buff, sizeof (char*)*100, file)) {
if (line == 0) {
fgets(buff, sizeof (char*)*100, file);
line++;
}
int i = 0;
char *p = strtok(buff, ",");
char *...
-1
votes
1answer
62 views
how to remove the empty quotes from the list using python?
I have a python script that pre-processes the text before I can make the text analysis. Some of the functions to clean the text are:
to remove strings that are less than two characters.
to tokenize ...
1
vote
1answer
39 views
How can I build an index and a boolean query in elastic search that performs prefix matching on some fields and exact phrase matching on other fields?
I would like to create an index that contains a field called "home_city".
I will need to perform a search on this index that returns all documents with a home_city field that is exactly "Little Rock".
...
-1
votes
1answer
207 views
iOS String: remove prefix and suffix by CharacterSet [closed]
For a given String in Swift I need to remove prefix and suffix characters that belong to a predefined character set.
I can use components(separatedBy:) with the character set and get rid of the empty ...
-1
votes
1answer
177 views
How can i tokenize all rows in a specific column from a csv file using Python?
I'm doing a sentiments analysis using Python (I'm still a rookie with this specific programming language). I have some Twitter data in a csv file that I need to pre-process before doing the real ...
1
vote
1answer
2k views
AttributeError: 'spacy.tokens.doc.Doc' object has no attribute 'lower'
I am working on appending texts in a list, and then change the texts to be word embeddings and then do machine learning. The 'insts' in 'articles' is collected by using spacy, but then I am faced with ...
0
votes
0answers
35 views
Block inside pytest.raises context manager not getting executed
I have been writing a tokeniser for a little language I'm developing in my free time, and I want to test it with negative input. My code can raise two kinds of Exceptions, an IndentationError and a ...
0
votes
0answers
109 views
Python NLP - ValueError: could not convert string to float: 'UKN'
I'm trying to train a random forest regressor to predict the hourly wage of an employee given the job description supplied. Note, I've signed an NDA and cannot upload real data. The below "observation"...
0
votes
1answer
50 views
Splitting string in java on a 2 character delimeter [duplicate]
I have a String in the format first<-0 or i<-length where the <- is used instead of the assignment operator.
I want to break it in 3 parts i.e. left side, operator and right side.
Then ...
1
vote
2answers
91 views
Tokenizing string in (old) Lua
I am trying to tokenize NMEA sentence using Lua. The current Lua version (available at https://www.lua.org/cgi-bin/demo) works as expected:
local index = 0
for token in string.gmatch("$HEHDT,99.00,,T*...
0
votes
0answers
18 views
NLTK Custom tokenization with Python regex [duplicate]
I am working in text processing using python. I have a requirement similar to the one in the thread Modify NLTK word_tokenize to prevent tokenization of parenthesis
But mine is whenever there is this ...
0
votes
1answer
49 views
word_tokenize with same code and same dataset, but different result, why?
Last month, I tried to tokenize text and create a of words to see which word shows up frequently. Today, I want do it again in the same dataset with the same code. It still works but the result is ...
1
vote
0answers
26 views
Edge n-gram suggestions and 'starts with' keyword in Elasticsearch
I'm trying to build a food search engine on Elasticsearch that should meet following use cases -
If the user searches for 'coff' then it should return all the documents with phrase 'coffee' in their ...
0
votes
0answers
72 views
Implement Trie by word token level, not character level in Python
I am trying to implement a Trie in Python by using word tokens (as opposed to characters) and return a list of indices by phrases that are found. I am stripping any punctuation after splitting up the ...
0
votes
3answers
44 views
How can I count the number of numbers in a string
I want to count the number of numbers in s string.
In this string:
"2019 was a great year for 10 fortunate people in ages 20 to 60."
The result should be 4 (2019, 10, 20, 60)
Thanks
0
votes
0answers
22 views
How to add an empty line between sentences after tokenizing
I am dealing with a problem where I need the data in a specific format as shown in desired output.
I've tried below code but not able to add empty line after the last token of sentence.
my_list = [['...
0
votes
1answer
464 views
Elasticsearch “max_ngram_diff” working for “edge_ngram” but not for “ngram_tokenizer”
I've got an Elasticsearch index where I've set "max_ngram_diff": 50, but somehow it seems only to be working for the edge_ngram tokenizer but not for the ngram tokenizer.
I've made this two requests ...
0
votes
0answers
137 views
Python NLP Text Tokenization based on custom regex
I am processing large amount of text for custom (NER) Named Entity Recognition using Spacy. For text pre-processing I am using nltk for tokenization..etc.
I am able to process one of my custom ...
0
votes
0answers
34 views
tokenizing a dataframe using unnest_tokens gives me error?
I am new to R and been trying to tokenize a data.frame using unnest_tokens. My data.frame was obtained from a csv file and has the following column :
ID, TITLE, DESCRIPTION, OUTCOMES, DATE .
I ...
0
votes
0answers
14 views
back to original after word-tokenize
I have 10,563 reviews and I applied pre-processing steps like tokenizing, normalization, stop word removal and etc. my problem is how to save the results into excel file back as reviews.
cl_data = []...
0
votes
0answers
181 views
Tensorflow crashes memory google colab
I'm working on word Embeddings for arabic dialect -like slang of a region- while preprocessing the data:
load json
extract the lines
clean from urls,emojis nd other
remove any list has minimum of 2 ...
0
votes
3answers
232 views
Tokenizing a string and return it as an array
I'm attempting to tokenize a passed string, store the tokens in an array and return it. I'm running this on ubuntu. I'm stumped when it comes to this language apparently.
Sample input: coinflip 3
My ...
0
votes
0answers
48 views
How to search human names in a dataframe?
I am trying to search human name from the dataframe. I have a very large dataset where i need to tokenize everyword but human names should be full names. I am just creating this as an example.
Date ...
1
vote
1answer
153 views
How to tokenize a Roman numeral term in ElasticSearch?
When creating tokenizer by registering token chars as below, Roman 'X' cannot be registered.(Test ES Version : ES6.7, ES5.6)
"tokenizer": {
"autocomplete": {
"type": "...
0
votes
1answer
190 views
Converting String to array of Tokens in Java [duplicate]
I'm trying to convert a string into an array of tokens, and while I've been finding questions similar to this all over the internet, no one is explaining how to specifically convert the words into a ...
0
votes
1answer
36 views
Is there a way to give a StanfordCoreNLP pipeline raw text and a list of tokens as input?
I'm doing some natural language processing with Arabic. Since I'm working with a couple different NLP tools in tandem, I want to be able to be able to give raw text to a StanfordCoreNLP pipeline, but ...
0
votes
1answer
13 views
Is there a way to save my processed Dataframes in Jupyter Notebooks
I'm new to Jupyter Notebooks and I was wondering if there is a way to save my (Preprocessed Data) instead of re running the file each time. I'm working on a NLP project where steps like (Lemmatization ...
-2
votes
1answer
17 views
Tokenizer and print it
After tokenizer my list of strings im trying to get the value of the words and its number's associate. f.e: the = 3
how can I do it?? (python)
here is the code
sentences_train, sentences_test, ...
0
votes
1answer
122 views
Error in loading NLTK resources: “Please use the NLTK Downloader to obtain the resource:\n\n”
I adapted the following code from Susan Li's post, but incurred an error when the code tries to tokenize text using NLTK's resources (or, there could be something wrong with "keyed vectors" loaded ...
0
votes
2answers
79 views
How to tokenize words and input them into another file?
I can only get stop words to implement into the document and then create a new file with the stop words removed. I cannot get word tokenize, porterstemmer, or sent tokenize to process.
import io
...
0
votes
0answers
64 views
Understanding Hate-Speech classification with Python. Currently stuck with “tokenization” (TweetTokenizer)
i'm still very new to Python, and for a University project I am doing classification of hate speech on twitter. I have been following various guides on how to set up some basic sentiment analysis etc, ...
2
votes
1answer
565 views
How can I get Spacy to stop splitting both hyphenated numbers and words into separate tokens?
Thanks for looking. I am using spaCy to perform Named Entity Recognition on a block of text, and I am having a peculiar problem I can't seem to overcome. Here is a sample code:
from spacy.tokenizer ...
0
votes
0answers
43 views
how to calculate no of tokens in a text file in nlp
from urllib import request
from redditscore.tokenizer import CrazyTokenizer
tokenizer = CrazyTokenizer()
url = "http://www.site.uottawa.ca/~diana/csi5386/A1_2020/microblog2011.txt"
with open('...
0
votes
1answer
99 views
How to print a string based on the postion in c?
I am working on a question which requires me to print a string given a field-number at that position. The strings should be read from a file.
file.txt
C is a language.
lex lexical analyser
(blank ...
-1
votes
2answers
57 views
how to tokenize a text by nltk python
i have a text like this:
Exception in org.baharan.dominant.dao.core.nonPlanAllocation.INonPlanAllocationRepository.getAllGrid()
with cause = 'org.hibernate.exception.SQLGrammarException: could not ...
0
votes
1answer
38 views
text getting overwritten in output file
from urllib import request
from redditscore.tokenizer import CrazyTokenizer
tokenizer = CrazyTokenizer()
url = "http://www.site.uottawa.ca/~diana/csi5386/A1_2020/microblog2011.txt"
for line in request....
0
votes
0answers
21 views
How to output a list to a file?
from urllib import request
from redditscore.tokenizer import CrazyTokenizer
tokenizer = CrazyTokenizer()
url = "http://www.site.uottawa.ca/~diana/csi5386/A1_2020/microblog2011.txt"
for line in request....
0
votes
1answer
165 views
Text length exeeds maximum - How to increase it?
import nltk
from urllib import request
from redditscore.tokenizer import CrazyTokenizer
tokenizer = CrazyTokenizer()
url = "http://www.site.uottawa.ca/~diana/csi5386/A1_2020/microblog2011.txt"
...