Questions tagged [tokenize]
Tokenizing is the act of splitting a string into discrete elements called tokens.
2,980
questions
0
votes
0
answers
11
views
What padding values should be used for huggingface tokenizers?
I am using Mbart50 to convert Nepalese to English and am not sure what values I should use for padding. I need to figure out the padding values for both English and Nepalese.
My tokenizer code:
...
0
votes
0
answers
8
views
How to modify a trained SentencePiece tokenizer to stop splitting the chatml tokens?
We are using a pre-trained SentencePiece tokenizer (the SentencePiece tokenizer from Google, not huggingface), and we would like to preserve the chatML tokens:
<|im_start|> and <|im_end|>
...
0
votes
1
answer
39
views
tokenizer and parser returns wrong answer for a postfix notation
I have coded a tokenizer and a recursive parser for a postfix expression. My code is the following:
import re
token_patterns = [
('OPERATOR', r'[+\-*/]'),
('NUMBER', r'\d+'),
('WHITESPACE'...
0
votes
1
answer
39
views
Understand the difference between the arguments "text" and "text_target" in the bert tokenizer from the huggingface transformers library [duplicate]
From the transformers library by huggingface
from transformers import BertTokenizer
tb = BertTokenizer.from_pretrained("bert-base-uncased")
tb is not a wordpiece tokenizer. It has arguments ...
1
vote
0
answers
17
views
How to use NLTK tokeniser on text with quotation marks?
NLTK's tokeniser is acting very strangely when I try to tokenise a text with quotation marks in it.
I have a .txt file like this:
This text has some "quotation marks" to really 'make things' ...
0
votes
0
answers
37
views
Special token being printed out when generating text from LLM
Hi i have a question about a llm printing out special token as well when generating an answer.
Here is an example:
from utils.prompter import Prompter
# from utils.util import postprocessing, ...
-2
votes
0
answers
19
views
Library of tokenize_2grams
name 'tokenize_2grams' is not defined
Could you plz tell me what the library of this function (tokenize_2grams) is?
tokenize_2grams("Thomas Jefferson began building Monticello at the age of 26.&...
-1
votes
1
answer
24
views
nltk python library word tokenizatio error [closed]
I am trying to tokenize a file.
`AttributeError Traceback (most recent call last)
<ipython-input-8-81ae6f78b554> in <cell line: 4>()
2 robert = open('...
0
votes
1
answer
58
views
Get list of tokenized files and deploy them
My google foo is failing me.
We have a system compiled of 10's of thousands of files.
hundreds of which we have an environment token {#...#} in them. most all are PL/SQL, so we deploy individual files ...
0
votes
1
answer
36
views
Issue with python's string find method returning -1
I'm trying to write a function that takes a string and returns the token positions. The function works fine when tokens = query_string.split() but if I try to use the string lower method, as shown in ...
-1
votes
1
answer
47
views
Displaying top 5 keywords for every category using group_by function
I am trying to find top 5 keywords in reviews for every category of products for which I have the following code
# Group by category and count keyword frequencies
keyword_counts <- filtered_data %&...
0
votes
1
answer
50
views
Map BERT token indices to Spacy token indices
I’m trying to make Bert’s (bert-base-uncased) tokenization token indices (not ids, token indices) map to Spacy’s tokenization token indices. In the following example, my approach doesn’t work becos ...
0
votes
0
answers
32
views
Result for LoRA fine-tuning not good
I'm fine-tuning the vicuna-7b model.
I use 1,000 samples to fine-tune LoRA, like this:
{"instruction": "the weather is hot, please open air conditioner.", "input":"&...
0
votes
0
answers
31
views
Need help to replace tokenize with XSLT 1.0 (1.1) [duplicate]
I'm working on a script, that allows me to translate table columns with a fixed width to columns with a percentage distribution (coming from some PDF production and adapting for HTML output).
I worked ...
0
votes
0
answers
13
views
DistilBert tokenization does not add pounds (##) at the start of in-word tokens after increasing vocabulary
I’m enriching DistilBert tokenizer with new tokens from new corpus. DistilBert uses WordPiece tokenizer, and based on the Huggingface NLP course, the inference is done by finding the “longest possible ...
0
votes
0
answers
27
views
Replace special [unusedX] tokens in a tokenizer to add domain-specific words in Bert based models - huggingface
Let’s say I have domain-specific word that I want to add to the tokenizer I am using for fine-tuning a model further. Tokenizer for BERT is one of those tokenizers that has [[unusedX] tokens]. One of ...
0
votes
0
answers
19
views
Identifying most useful domain-specific tokens for adding to the existing tokenizer in huggingface transformers
I am using a model for token classification task on a medical domain using huggingface transformers. Unfortunately, I don’t have enough data to set up a new tokenizer and train a new model from ...
0
votes
0
answers
26
views
Using Swift RegexBuilder in a lexer - collapsing ChoiceOf match output tuple to (SubString, TypeOfMatch) type
I'm implementing a parser using Swift's RegexBuilder for the lexer stage. I would like to understand if my regex ChoiceOf statement could be improved.
Specifically in the example code below, when ...
0
votes
0
answers
37
views
How can I authenticate user with JWT(JSON web token), while keeping the user in session and then log out?
I am a beginner in Angular and need to authenticate a user with JWT. In the login method I have only passed a string to validate user for login, for now as I haven't created a fake backend for testing....
0
votes
2
answers
94
views
How do I looping through mutiple lines and tokenizing, returning an array containing all the tokens?
I posted a (terrible formatted and lackluster) question earlier asking about how I can pass an array as an input parameter and return that modified array. After some more messing around I found out ...
1
vote
1
answer
520
views
Calculating total tokens for API request to ChatGPT including functions
Hello Stack Overflow community,
I've been working on integrating ChatGPT's API into my project, and I'm having some trouble calculating the total number of tokens for my API requests. Specifically, I'...
1
vote
1
answer
77
views
Parsing PHP file in order to get an array of parameters
I have a PHP file with following content:
<?
require($_SERVER["DOCUMENT_ROOT"]."/bitrix/header.php");
$APPLICATION->IncludeComponent(
"hoo:account",
"...
-1
votes
1
answer
24
views
Ho to Extract DistilBERT Embeddings from a list containg 5000 records..?
After tokenizing the dataset then we try to extract DistiBert Embeddings on our dataset(contains 5000 text records in dataframe) memory error occurred for the following code:
outputs = model(**...
4
votes
1
answer
527
views
OpenAI API: What would be a good strategy to handle 80+ function calling?
My business handles a variety of entities (job, invoice, quote, resource, vehicle, contact, person, message, alert, etc.).
My goal is to use OpenAI function calling to allow my users to ask "...
0
votes
0
answers
74
views
How do I train my NER model using Trainer from huggingface?
I'm having trouble using Trainer from HuggingFace to train my NER model.
My data looks like this :
DatasetDict({
train: Dataset({
features: ['id', 'tokens', 'ner_tags', '__index_level_0__']...
0
votes
1
answer
85
views
How to make the simplest minimal huggingface tokenizer?
I would like to make the simplest possible huggingface tokenizer for unit testing code logic. It doesnt have to work well, just produce an output similar to any other tokenizer.
Reading the course on ...
0
votes
0
answers
103
views
HuggingFace Tokenizer for multiple columns
I have a dataset that has 2 text columns and 1 output column.
text_a text_b output
"This is a bird" "This is a pencil" 0
"This is a cat&...
1
vote
0
answers
54
views
Some doubts about SentencePiece
I recently encountered some questions when I was learning Google’s SentencePiece.
BPE, WordPiece and Unigram are all common subword algorithms, so what is the relationship between SentencePiece and ...
0
votes
1
answer
182
views
How to concatenate a split word using NLP caused by tokenizers after machine translation?
Russian translation produces the following result, is there a NLP function which we can use to concatenate as "Europe's" in the following string?
"Nitzchia Protector Todibo can go to ...
1
vote
1
answer
100
views
Splitting a word into two words with spaCy
I'm facing an issue where I need to split a single 'word' into two words due to missing spaces or new lines in the received text. My intention is to establish a pipeline (spaCy 3.5.4) for this task ...
1
vote
0
answers
238
views
When to set `add_special_tokens=False` in huggingface transformers tokenizer?
this is the default way of setting tokenizer in the Hugging Face "transformers" library:
from transformers import BertForSequenceClassification,BertTokenizer
tokenizer=BertTokenizer....
0
votes
1
answer
66
views
Why is my streamlit app not correctly summarizing my mp3-transcription?
I'm working on a Streamlit app that processes MP3 files. The main steps include:
Uploading an MP3 file.
Splitting the audio into smaller chunks using pydub.
Transcribing these chunks using OpenAI.
...
0
votes
0
answers
260
views
CLIP model from `open_clip` module returns single embedding for 77 tokens
I'm using the open_clip module to obtain text embeddings from the CLIP model. When I tokenize a list of a single text sequence and pass them to the model's encode_text method, I expect to get ...
1
vote
0
answers
96
views
Byte-level BPE tokenizer for handing Bigram and Trigram
I'm currently employing the HuggingFace tokenizer to tokenize a textual database, and here's how I'm doing it:
from tokenizers import ByteLevelBPETokenizer
from tokenizers import normalizers
...
0
votes
1
answer
50
views
How to use spaCy Matcher to create a pattern for rule-based matching for a sequence that is only interpreted as a single token
I am new to nlp and spaCy but I am using it for my project. I am trying to use spaCy's Matcher class to create a pattern to extract information from clinical summaries, specifically mentions of IQ ...
0
votes
0
answers
24
views
ValueError: Expected input batch_size (1052) to match target batch_size (508) when fine tuning GPT 2 model
Hello there I'm attempting to train a GPT 2 model how to summarize passages without compromising their emotional impact. Consider summarizing a chapter from a book, but we want the reader to ...
0
votes
0
answers
22
views
Proper teatment of lists of word tokens in a df for clustering - advice needed
I have been provided with tokenized text that has been previously generated. I would like to keep these tokens and use them in my analysis. For each item of interest, there are multiple lists of ...
0
votes
0
answers
15
views
Configure Solr tokenizer to preserve original term using solr.PatternTokenizerFactory
My FQDN is www.one.two.ac.uk and I'd like to parse it such that one and two and one.two are searchable. Here is what I've got so far: -
<fieldType name="hostfqdn" class="solr....
1
vote
1
answer
99
views
Can't find model 'en_core_web_lg'. It doesn't seem to be a Python package or a valid path to a data directory. Eventhough they are in same directory
I am trying different text processing model. I am trying to use spacy and it's model en_core_web_lg.
import spacy
import spacy.language
from spacy_langdetect import LanguageDetector
from spacy....
0
votes
0
answers
20
views
Adding Spaces to Language Model Output for User-Friendly Display
large language models generate a list of tokens, but to show these tokens to a user we need to add spaces between these tokens, so how do we know where to put these spaces to show an appropriate text?
...
0
votes
2
answers
47
views
How to apply tokenize to a particular column in a dataframe by using python?
I have a dataframe having three columns. One of the column from the dataframe need to apply tokenization. I am getting TypeError : expected string or bytes-like object, got 'float' .
import pandas as ...
0
votes
1
answer
74
views
Adding custom words to the tokenizer in scispacy
The tokens that I am dealing are biomedical terms, which are not tokenized properly by my spacy Named-Entity-Recognition(NER) model, words that are supposed to be together are divided abruptly. ...
4
votes
2
answers
56
views
How to get javascript regex to return everything in the string as part of the matches, not just the matched parts
Test string:
*This* is a test.[15] I would like[16] To figure _out how_ to do this.[20]
RegEx:
/\[\d+\]|\*.*\*|_.*_/gu
Actual Results:
Match 1: *This*
Match 2: [15]
Match 3: [16]
Match 4: _out how_
...
1
vote
1
answer
110
views
CS50 AI project 6b Questions : code doesn't handle all tokenization correctly
I have submitted this project 4 times till now and it always get rejected with the same message :
Your code does not handle all tokenization correctly, which can lead to parsing and interpretation ...
0
votes
0
answers
99
views
Tokenizing large text datasets
I am trying to work on a text generation project. I downloaded the WikiBooks dataset from Kaggle:
https://www.kaggle.com/datasets/dhruvildave/wikibooks-dataset
And when I try to create a dataset to ...
0
votes
1
answer
53
views
'BpeTrainer' object cannot be converted to 'Sequence' when training Bpetokenizer
I found this class on
import os
from tokenizers.models import BPE
from tokenizers import Tokenizer
from tokenizers.decoders import ByteLevel as ByteLevelDecoder
from tokenizers.normalizers import NFKC,...
1
vote
2
answers
54
views
Removing stopwords also removes spaces between words during frequency distribution
I am looking to remove stopwords from text to optimise my frequency distribution results
My initial frequency distribution code is written:
# Determine the frequency distribution
from nltk.tokenize ...
0
votes
0
answers
100
views
update monaco-editor configurations in standalone browser editor
I'm using the monaco.editor.colorize and monaco.editor.tokenize API in my app for colorizing and formatting a long line of logs (I defined the language id), however I find no way of configuring, for ...
0
votes
1
answer
102
views
Java/JavaFX - Shopping Cart System - Tokenizing data into an Arraylist
So I'm working on a school assignment. I have to create a shopping cart system that displays books in a ListView and have buttons/menu items that can add those books to a cart or remove them.
The ...
2
votes
0
answers
260
views
Transformers tokenizer batch encode a list of lists of strings
I have my dataset in the format of a list(A) of lists(B) of strings:
foo = [['asdf', 'hello', 'john matthew'],
['asdf', 'good bye', 'luke caleb'],
['asdf', 'see you tomorrow', 'peter ...