Questions tagged [tokenize]
Tokenizing is the act of splitting a string into discrete elements called tokens.
2,554
questions
-3
votes
0answers
11 views
How do I tokenize this sentence to have a city as one token but also output the rest of the sentence? [closed]
If I have the sentence "I Love New York"
how do I tokenize it so it makes New York one token. I did:
import re
text= "I Love New York"
re.findall(city_regex, text)
it only output &...
0
votes
0answers
10 views
I have an error with tokenizing CSV file with NLTK, how would i fix this?
I'm new to NLP and python, although I am sure this code should work, could someone tell me why i'm getting this error and how to fix it?
here is my code:
import nltk
import numpy as np
import pandas ...
-1
votes
1answer
19 views
RNN classifier for generating advices
I am try to make a project in which farming advices are generated based on weather condition on a specific district. I have a sample dataset for now, as shown below.
state
district
month
rainfall
...
0
votes
1answer
21 views
Separate sentences ending with a scientific reference number in r
I am working on a project where one of the steps is to separate text of scientific articles into sentences. For this, I am using textrank which I understands it looks for . or ? or ! etc. to ...
1
vote
1answer
15 views
spacy how do I make a matcher which is noun-noun without white space within it?
I tried to make a matcher which could detect words like
'all-purpose'
I was trying to make a pattern like
pattern=[{'POS':'NOUN'}, {'ORTH':'-'},{'POS':'NOUN'}]
However, I realized that it only find ...
0
votes
0answers
12 views
Is it possible to test whether a tokenizer can losslessly tokenize and detokenize a given corpus solely from its vocabulary?
I am trying to test whether a given vocabulary list contains the tokens necessary to reconstruct a corpus of text losslessly. That is, if a tokenizer trained on a corpus of text were to attempt to ...
1
vote
1answer
21 views
Tokenize list of strings without comma separation
I'm still new to Python and want to know how I can tokenize a list of strings without every word being seperated by a comma.
For example, starting from a list like ['I have to get groceries.','I need ...
0
votes
1answer
73 views
Regex: Only want space character before and after match
I am using Regex tokenizer for a text passage, and I would like to extract all words that only have white space before and after that. Here is my code:
tokenizer = RegexpTokenizer('[0-9a-z][^\s\']*[a-...
0
votes
0answers
24 views
SpaCy Lemmatizer working differently for a direct string and tokenized string
I have two sentences one which is send to lemmatizer with a full string. The other is sent in tokenized version to the following functions.
nlp = spacy.load("en_core_web_lg")
def ...
0
votes
0answers
30 views
Python Tokenizer, sentence gets cut off at newline character
I am trying to build a tokenizer with python, I think it is going on pretty well but there is one problem that I cannot seem to solve. I'll try to be as elaborative as possible:
This is the text I am ...
0
votes
0answers
10 views
subword tokenization providing own subword vocab
Is there any utility/library that providing a customized subword vocabulary, it can do subword tokenization encoding for text sequence?
1
vote
0answers
15 views
RegistryError: Cant't find 'spacy.Tokenizer.v1' in registry spacy -> tokenizers. Available names: none
I am trying to:
Tokenize CV from a pdf
This is what I have done:
pages = []
nlp = spacy.load("fr_core_news_sm")
test = "C:\\Users\\admin\\OneDrive\\Bureau\\Stage\\file.pdf"
def ...
-1
votes
1answer
24 views
How do I browse a column of strings and convert any integers to the word in Python?
I am creating a function that will go through a column of strings. In some of these strings there are numbers, and some strings there are not. If a string contains a number, I would like to change ...
0
votes
0answers
15 views
Sentiment Analysis error (“module 'nltk.tokenize' has no attribute 'api'”)
I tried to run the Twitter Sentiment Analysis in Python, but I encounter this error:
module 'nltk.tokenize' has no attribute 'api'
Just wondering if you could let me know how can I debug this?
I have ...
0
votes
1answer
60 views
Split a sentence by words just as BERT Tokenizer would do?
I'm trying to localize all the [UNK] tokens of BERT tokenizer on my text. Once I have the position of the UNK token, I need to identify what word it belongs to. For that, I tried to get the position ...
-6
votes
3answers
178 views
Extract string-token objects from string?
Does Delphi (10.4) have a string-tokenizer that extracts string-token-objects from a string in a similar way as below?
MyPhrase := 'I have a simple word and a complex Word: A lot of WORDS.';
MyTokens ...
0
votes
1answer
36 views
Handling multiple ERC721 Tokens in the same Contract
Is it possible for the same contract to handle multiple ERC721 tokens? And if so, how?
Letās say youāre making a game and you need to track two types of ERC721 tokens: unique Weapons and unique ...
1
vote
1answer
49 views
Tokenize a sentence where each word contains only letters using RegexTokenizer Scala
I am using spark with scala and trying to tokenize a sentence where each word should only contain letters. Here is my code
def tokenization(extractedText: String): DataFrame = {
val ...
-1
votes
1answer
36 views
Using Python, NLTK, to analyse German text
I am a beginner in Python and currently trying to use NLTK to analyze German text (extract the German noun and it's frequency of German text) by following this tutorial: https://datascience.blog.wzb....
1
vote
0answers
27 views
Struggling creating a simple list parser in TypeScript without any mutation
I'm trying to write a very rudimentary lisp parser in TypeScript that turns a string (in this case the, the lisp code) into an syntax tree (nested arrays representing the parsed s-expressions) while ...
0
votes
1answer
14 views
Apache Camel split with new line token and use aggregation stategy
I have the following route:
from("file:/home/tmp/test?move=.done")
.routeId("file")
.split(body().tokenize("\n"),new GroupedBodyAggregationStrategy())
....
0
votes
0answers
18 views
Is it possible to detokenize DataFrame in PySpark?
I'm using app.zelp.com to perform NLP. After tokenization and removing the stopwords, I would like to detokenize the remaining words and export to csv. Is that possible?
%python
# Start Spark session
...
0
votes
2answers
53 views
How to untokenize BERT tokens?
I have a sentence and I need to return the text corresponding to N BERT tokens to the left and right of a specific word.
from transformers import BertTokenizer
tz = BertTokenizer.from_pretrained("...
0
votes
0answers
28 views
How to use glove with spacy tokenizer? (python, error)
I am attempting some sentiment analysis (The following links are essentially the same as what I'm working on) and need to build a a vocab from amazon reviews that have associated good/bad/ok labels. ...
2
votes
3answers
46 views
Solr wildcard search incorrect result
I have some unexpected results when i make wildcard queries. I am using solr 6.6.0. edismax handler inside solr ui. The following query return results as expected without wildcard - firstNames:James, ...
0
votes
1answer
17 views
<nltk.tokenize.casual.TweetTokenizer at 0x7f7fec4d5970> issue
it might be a basic question but I am stuck here not really sure what went wrong.
df['text'] contains the text data that I want to work on
text_sents=df.text
tokens = []
for uni in text_sents:
...
0
votes
3answers
55 views
Find indexes of unquoted words in a string using `re.finditer()` method
I'm trying to find the position (index) of unquoted words in a string, but all my tests have been unsuccessful.
For the string: string='foo "bar" baz' I'd like to get
(0, 3) # This for foo
...
0
votes
1answer
29 views
strtok() not reading past the first input piece
I am trying to write a Linux shell replacement for an operating systems class and am having trouble parsing the input strings. I am able to read in the very first line of a string input, but once it ...
-2
votes
1answer
32 views
Tokenizing without breaking up key phrases [closed]
I have a string of text like s = 'hi, welcome to grade 3'
currently when I tokenize the string I get
tokens = ['hi', 'welcome', 'to', 'grade', '3']
How can I tokenize the string without generating ...
0
votes
0answers
11 views
Stem Completion is Taking too long without results
I'm working on a corpus of 7800 tweets, everything is working correctly but when I try to use stemCompletion to convert the stemmed words into complete words, the execution takes hours without any ...
0
votes
0answers
6 views
Transfer pdf content into nltk Copera bigrams
from my own list of pdf files I would like to generate a word vector. So I transfer all texts in a list. So far so good, but to take advantage of nltk I need to have data available in:
nltk.corpus....
0
votes
2answers
91 views
In HuggingFace tokenizers: how can I split a sequence simply on spaces?
I am using DistilBertTokenizer tokenizer from HuggingFace.
I would like to tokenize my text by simple splitting it on space:
["Don't", "you", "love", "š¤", &...
0
votes
1answer
28 views
Lucene tokenization / filters not working as expected | Solr analysis confusion
I am trying to figure out the correct configuration for my analyzer configuration in my Solr/Lucidworks setup.
The results that I am seeing in Solr analysis seem to indicate that I should be getting ...
0
votes
0answers
19 views
Encoding error: Train BERT from scratch in Vietnamese language
I follow this tutorial How to train a new language model from scratch using Transformers and Tokenizers.
In Section 2. Train a tokenizer, after training by my own Vietnamese text data, I look at the ....
1
vote
0answers
60 views
Monaco editor custom token provider behaviour
I am attempting to implement a tokenizer with quite specific behaviour, where for example, in the following situation:
1: Line 1
2: Line 2
3:
4: Line 4
5: Line 5
6: Line 6
7:
8:
9: Line 9
...
0
votes
1answer
36 views
How to use Tokenizer (Keras)? Unable to generate tokens on Character level
goal: vectorizing on character-level
problem: output is not a unique number per character/letter, instead all letters are converted to 1
Question: What is wrong with my code?
I have a dataframe (df). ...
0
votes
0answers
10 views
nltk.word_tokenize isn't fast enough
import pandas as pd
import numpy as np
import csv
import nltk
from nltk.corpus import stopwords
from nltk.stem import SnowballStemmer
from nltk.tokenize import word_tokenize
from nltk.tokenize import ...
0
votes
2answers
53 views
How to separate comma from word (tokenization)
I have some problem with tokenization, the assignment is to separate a sentence into words.
This is what I have done at the moment.
def tokenize(s):
d = []
start = 0
while start < ...
0
votes
1answer
42 views
How to build a simple tokenizer
I would like to know how to build a very simple tokenizer. Given a dictionary d (in this case a list) and a sentence s I would like to return all possible tokens (=words) of the sentence. Here is what ...
0
votes
1answer
39 views
Removing stop words from a pandas column
import nltk
nltk.download('punkt')
nltk.download('stopwords')
import datetime
import numpy as np
import re
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem....
0
votes
1answer
47 views
How to specify beginning-of-line keywords in ANTLR grammars (which also works for the first input line)
This is a question about the remaining problem of the solution proposed for another Stackoverflow question about beginning-of-line keywords.
I am writing an ANTLR4 lexer and parser for a programming ...
0
votes
0answers
29 views
Why using GPT2Tokenizer convert Arabic characters to symbols?
I am trying to use GPT2 for Arabic text classification task as follows:
tokenizer = GPT2Tokenizer.from_pretrained(model_path)
model = GPT2ForSequenceClassification.from_pretrained(model_path,
...
0
votes
2answers
46 views
NoSuchElementException when reading from a pipe-delimited text file
I am getting a NoSuchElementException when reading from a "|" (pipe) delimited text file.
I think this is the part that causes the error:
public void readFromFile(String file)
{
operas....
0
votes
0answers
18 views
I created block marking extension for VS Code, how can I access to parsed tokens of code in a file?
Explanation DEMO with react.js:
https://youtu.be/qrEKSe8VkYQ
The extension itself:
https://youtu.be/ZmvLgIDJNg8
Please help, I'm trying to finish creating an extension for VS Code. It works fine with ...
-3
votes
1answer
38 views
how to tokenize a column of a df and loop over each token in each row? [closed]
I have a Pandas data frame like this:
Title_lower_case word_count id
0 customer relation executive 6 0
1 purchase in-charge 6 0
2 ...
-1
votes
2answers
77 views
Swift String Tokenizer / Parser [closed]
Hello there fellow Swift devs!
I am a junior dev, and I'm trying to figure out a best way to tokenize / parse Swift String as an exercise.
What I have is a string which looks like this:
let string = &...
0
votes
2answers
30 views
NLTK TweetTokennizer is not workin on google colab notebook
I have downloaded all the packages of NLTK with the command of nltk.download().
but the thing is that when I try to import TweetTokenizer I get the error.
tokenizer = TweetTokenizer(preserve_case=...
0
votes
1answer
20 views
How does the BERT tokenizer result in an input tensor shape of (b, 24, 768)?
I understand how the BERT tokenizer works thanks to this article:
https://albertauyeung.github.io/2020/06/19/bert-tokenization.html
However, I am confused about how this ends up as the final input ...
0
votes
2answers
46 views
After tokenizing string with hypen '-', hypen will get eliminated!! how to fix this in c++98
below shell script saved file name as add.sh
#!/bin/sh
MY_EXEC="add";
add -Units 32 -raid 111 -log add.log
My code will read this file add.sh and store contents in vector. the vector[2] ...
0
votes
0answers
20 views
Generate a new field using filbeat tokenizer
I'm looking for how to add a new field using filbeat tokenizer.
Sample log file
20/01/17 11:23:33 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-51977e54-0775
need to create a new ...