Questions tagged [tokenize]

Tokenizing is the act of splitting a string into discrete elements called tokens.

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

What padding values should be used for huggingface tokenizers?

I am using Mbart50 to convert Nepalese to English and am not sure what values I should use for padding. I need to figure out the padding values for both English and Nepalese. My tokenizer code: ...
Ryan Marr's user avatar
0 votes
0 answers
8 views

How to modify a trained SentencePiece tokenizer to stop splitting the chatml tokens?

We are using a pre-trained SentencePiece tokenizer (the SentencePiece tokenizer from Google, not huggingface), and we would like to preserve the chatML tokens: <|im_start|> and <|im_end|> ...
vgoklani's user avatar
  • 11k
0 votes
1 answer
39 views

tokenizer and parser returns wrong answer for a postfix notation

I have coded a tokenizer and a recursive parser for a postfix expression. My code is the following: import re token_patterns = [ ('OPERATOR', r'[+\-*/]'), ('NUMBER', r'\d+'), ('WHITESPACE'...
Little's user avatar
  • 3,393
0 votes
1 answer
39 views

Understand the difference between the arguments "text" and "text_target" in the bert tokenizer from the huggingface transformers library [duplicate]

From the transformers library by huggingface from transformers import BertTokenizer tb = BertTokenizer.from_pretrained("bert-base-uncased") tb is not a wordpiece tokenizer. It has arguments ...
figs_and_nuts's user avatar
1 vote
0 answers
17 views

How to use NLTK tokeniser on text with quotation marks?

NLTK's tokeniser is acting very strangely when I try to tokenise a text with quotation marks in it. I have a .txt file like this: This text has some "quotation marks" to really 'make things' ...
AdeDoyle's user avatar
  • 361
0 votes
0 answers
37 views

Special token being printed out when generating text from LLM

Hi i have a question about a llm printing out special token as well when generating an answer. Here is an example: from utils.prompter import Prompter # from utils.util import postprocessing, ...
JS J's user avatar
  • 33
-2 votes
0 answers
19 views

Library of tokenize_2grams

name 'tokenize_2grams' is not defined Could you plz tell me what the library of this function (tokenize_2grams) is? tokenize_2grams("Thomas Jefferson began building Monticello at the age of 26.&...
fatemeh khoshhal's user avatar
-1 votes
1 answer
24 views

nltk python library word tokenizatio error [closed]

I am trying to tokenize a file. `AttributeError Traceback (most recent call last) <ipython-input-8-81ae6f78b554> in <cell line: 4>() 2 robert = open('...
i222025 Amna Javaid's user avatar
0 votes
1 answer
58 views

Get list of tokenized files and deploy them

My google foo is failing me. We have a system compiled of 10's of thousands of files. hundreds of which we have an environment token {#...#} in them. most all are PL/SQL, so we deploy individual files ...
Jon Larsen's user avatar
0 votes
1 answer
36 views

Issue with python's string find method returning -1

I'm trying to write a function that takes a string and returns the token positions. The function works fine when tokens = query_string.split() but if I try to use the string lower method, as shown in ...
Helpme's user avatar
  • 21
-1 votes
1 answer
47 views

Displaying top 5 keywords for every category using group_by function

I am trying to find top 5 keywords in reviews for every category of products for which I have the following code # Group by category and count keyword frequencies keyword_counts <- filtered_data %&...
kartik trivedi's user avatar
0 votes
1 answer
50 views

Map BERT token indices to Spacy token indices

I’m trying to make Bert’s (bert-base-uncased) tokenization token indices (not ids, token indices) map to Spacy’s tokenization token indices. In the following example, my approach doesn’t work becos ...
lrthistlethwaite's user avatar
0 votes
0 answers
32 views

Result for LoRA fine-tuning not good

I'm fine-tuning the vicuna-7b model. I use 1,000 samples to fine-tune LoRA, like this: {"instruction": "the weather is hot, please open air conditioner.", "input":"&...
user497032's user avatar
0 votes
0 answers
31 views

Need help to replace tokenize with XSLT 1.0 (1.1) [duplicate]

I'm working on a script, that allows me to translate table columns with a fixed width to columns with a percentage distribution (coming from some PDF production and adapting for HTML output). I worked ...
Gorly's user avatar
  • 13
0 votes
0 answers
13 views

DistilBert tokenization does not add pounds (##) at the start of in-word tokens after increasing vocabulary

I’m enriching DistilBert tokenizer with new tokens from new corpus. DistilBert uses WordPiece tokenizer, and based on the Huggingface NLP course, the inference is done by finding the “longest possible ...
Carlos Muradyan's user avatar
0 votes
0 answers
27 views

Replace special [unusedX] tokens in a tokenizer to add domain-specific words in Bert based models - huggingface

Let’s say I have domain-specific word that I want to add to the tokenizer I am using for fine-tuning a model further. Tokenizer for BERT is one of those tokenizers that has [[unusedX] tokens]. One of ...
Carlos Muradyan's user avatar
0 votes
0 answers
19 views

Identifying most useful domain-specific tokens for adding to the existing tokenizer in huggingface transformers

I am using a model for token classification task on a medical domain using huggingface transformers. Unfortunately, I don’t have enough data to set up a new tokenizer and train a new model from ...
Carlos Muradyan's user avatar
0 votes
0 answers
26 views

Using Swift RegexBuilder in a lexer - collapsing ChoiceOf match output tuple to (SubString, TypeOfMatch) type

I'm implementing a parser using Swift's RegexBuilder for the lexer stage. I would like to understand if my regex ChoiceOf statement could be improved. Specifically in the example code below, when ...
Philip Pegden's user avatar
0 votes
0 answers
37 views

How can I authenticate user with JWT(JSON web token), while keeping the user in session and then log out?

I am a beginner in Angular and need to authenticate a user with JWT. In the login method I have only passed a string to validate user for login, for now as I haven't created a fake backend for testing....
Banick's user avatar
  • 45
0 votes
2 answers
94 views

How do I looping through mutiple lines and tokenizing, returning an array containing all the tokens?

I posted a (terrible formatted and lackluster) question earlier asking about how I can pass an array as an input parameter and return that modified array. After some more messing around I found out ...
thestranger's user avatar
1 vote
1 answer
520 views

Calculating total tokens for API request to ChatGPT including functions

Hello Stack Overflow community, I've been working on integrating ChatGPT's API into my project, and I'm having some trouble calculating the total number of tokens for my API requests. Specifically, I'...
Mdogdope's user avatar
1 vote
1 answer
77 views

Parsing PHP file in order to get an array of parameters

I have a PHP file with following content: <? require($_SERVER["DOCUMENT_ROOT"]."/bitrix/header.php"); $APPLICATION->IncludeComponent( "hoo:account", "...
Viacheslav Ravdin's user avatar
-1 votes
1 answer
24 views

Ho to Extract DistilBERT Embeddings from a list containg 5000 records..?

After tokenizing the dataset then we try to extract DistiBert Embeddings on our dataset(contains 5000 text records in dataframe) memory error occurred for the following code: outputs = model(**...
Prasad Joshi's user avatar
4 votes
1 answer
527 views

OpenAI API: What would be a good strategy to handle 80+ function calling?

My business handles a variety of entities (job, invoice, quote, resource, vehicle, contact, person, message, alert, etc.). My goal is to use OpenAI function calling to allow my users to ask "...
Johann's user avatar
  • 12.2k
0 votes
0 answers
74 views

How do I train my NER model using Trainer from huggingface?

I'm having trouble using Trainer from HuggingFace to train my NER model. My data looks like this : DatasetDict({ train: Dataset({ features: ['id', 'tokens', 'ner_tags', '__index_level_0__']...
Akram H's user avatar
  • 43
0 votes
1 answer
85 views

How to make the simplest minimal huggingface tokenizer?

I would like to make the simplest possible huggingface tokenizer for unit testing code logic. It doesnt have to work well, just produce an output similar to any other tokenizer. Reading the course on ...
Nathan McCoy's user avatar
  • 3,104
0 votes
0 answers
103 views

HuggingFace Tokenizer for multiple columns

I have a dataset that has 2 text columns and 1 output column. text_a text_b output "This is a bird" "This is a pencil" 0 "This is a cat&...
user3104352's user avatar
  • 1,100
1 vote
0 answers
54 views

Some doubts about SentencePiece

I recently encountered some questions when I was learning Google’s SentencePiece. BPE, WordPiece and Unigram are all common subword algorithms, so what is the relationship between SentencePiece and ...
korangar leo's user avatar
0 votes
1 answer
182 views

How to concatenate a split word using NLP caused by tokenizers after machine translation?

Russian translation produces the following result, is there a NLP function which we can use to concatenate as "Europe's" in the following string? "Nitzchia Protector Todibo can go to ...
user352290's user avatar
1 vote
1 answer
100 views

Splitting a word into two words with spaCy

I'm facing an issue where I need to split a single 'word' into two words due to missing spaces or new lines in the received text. My intention is to establish a pipeline (spaCy 3.5.4) for this task ...
Maciek's user avatar
  • 21
1 vote
0 answers
238 views

When to set `add_special_tokens=False` in huggingface transformers tokenizer?

this is the default way of setting tokenizer in the Hugging Face "transformers" library: from transformers import BertForSequenceClassification,BertTokenizer tokenizer=BertTokenizer....
Yilmaz's user avatar
  • 38.6k
0 votes
1 answer
66 views

Why is my streamlit app not correctly summarizing my mp3-transcription?

I'm working on a Streamlit app that processes MP3 files. The main steps include: Uploading an MP3 file. Splitting the audio into smaller chunks using pydub. Transcribing these chunks using OpenAI. ...
Patrick Schmidt's user avatar
0 votes
0 answers
260 views

CLIP model from `open_clip` module returns single embedding for 77 tokens

I'm using the open_clip module to obtain text embeddings from the CLIP model. When I tokenize a list of a single text sequence and pass them to the model's encode_text method, I expect to get ...
hanugm's user avatar
  • 1,217
1 vote
0 answers
96 views

Byte-level BPE tokenizer for handing Bigram and Trigram

I'm currently employing the HuggingFace tokenizer to tokenize a textual database, and here's how I'm doing it: from tokenizers import ByteLevelBPETokenizer from tokenizers import normalizers ...
Eghbal's user avatar
  • 3,944
0 votes
1 answer
50 views

How to use spaCy Matcher to create a pattern for rule-based matching for a sequence that is only interpreted as a single token

I am new to nlp and spaCy but I am using it for my project. I am trying to use spaCy's Matcher class to create a pattern to extract information from clinical summaries, specifically mentions of IQ ...
Tarran's user avatar
  • 11
0 votes
0 answers
24 views

ValueError: Expected input batch_size (1052) to match target batch_size (508) when fine tuning GPT 2 model

Hello there I'm attempting to train a GPT 2 model how to summarize passages without compromising their emotional impact. Consider summarizing a chapter from a book, but we want the reader to ...
Damika's user avatar
  • 680
0 votes
0 answers
22 views

Proper teatment of lists of word tokens in a df for clustering - advice needed

I have been provided with tokenized text that has been previously generated. I would like to keep these tokens and use them in my analysis. For each item of interest, there are multiple lists of ...
Linda Smith's user avatar
0 votes
0 answers
15 views

Configure Solr tokenizer to preserve original term using solr.PatternTokenizerFactory

My FQDN is www.one.two.ac.uk and I'd like to parse it such that one and two and one.two are searchable. Here is what I've got so far: - <fieldType name="hostfqdn" class="solr....
linux-guy1001's user avatar
1 vote
1 answer
99 views

Can't find model 'en_core_web_lg'. It doesn't seem to be a Python package or a valid path to a data directory. Eventhough they are in same directory

I am trying different text processing model. I am trying to use spacy and it's model en_core_web_lg. import spacy import spacy.language from spacy_langdetect import LanguageDetector from spacy....
jill gosrani's user avatar
0 votes
0 answers
20 views

Adding Spaces to Language Model Output for User-Friendly Display

large language models generate a list of tokens, but to show these tokens to a user we need to add spaces between these tokens, so how do we know where to put these spaces to show an appropriate text? ...
hakim47's user avatar
  • 33
0 votes
2 answers
47 views

How to apply tokenize to a particular column in a dataframe by using python?

I have a dataframe having three columns. One of the column from the dataframe need to apply tokenization. I am getting TypeError : expected string or bytes-like object, got 'float' . import pandas as ...
Sanky Ach's user avatar
  • 335
0 votes
1 answer
74 views

Adding custom words to the tokenizer in scispacy

The tokens that I am dealing are biomedical terms, which are not tokenized properly by my spacy Named-Entity-Recognition(NER) model, words that are supposed to be together are divided abruptly. ...
cbac's user avatar
  • 11
4 votes
2 answers
56 views

How to get javascript regex to return everything in the string as part of the matches, not just the matched parts

Test string: *This* is a test.[15] I would like[16] To figure _out how_ to do this.[20] RegEx: /\[\d+\]|\*.*\*|_.*_/gu Actual Results: Match 1: *This* Match 2: [15] Match 3: [16] Match 4: _out how_ ...
Justin808's user avatar
  • 21k
1 vote
1 answer
110 views

CS50 AI project 6b Questions : code doesn't handle all tokenization correctly

I have submitted this project 4 times till now and it always get rejected with the same message : Your code does not handle all tokenization correctly, which can lead to parsing and interpretation ...
Zeyad moussa's user avatar
0 votes
0 answers
99 views

Tokenizing large text datasets

I am trying to work on a text generation project. I downloaded the WikiBooks dataset from Kaggle: https://www.kaggle.com/datasets/dhruvildave/wikibooks-dataset And when I try to create a dataset to ...
Nikita Andriievskyi's user avatar
0 votes
1 answer
53 views

'BpeTrainer' object cannot be converted to 'Sequence' when training Bpetokenizer

I found this class on import os from tokenizers.models import BPE from tokenizers import Tokenizer from tokenizers.decoders import ByteLevel as ByteLevelDecoder from tokenizers.normalizers import NFKC,...
Brian Hode's user avatar
1 vote
2 answers
54 views

Removing stopwords also removes spaces between words during frequency distribution

I am looking to remove stopwords from text to optimise my frequency distribution results My initial frequency distribution code is written: # Determine the frequency distribution from nltk.tokenize ...
Alexander's user avatar
0 votes
0 answers
100 views

update monaco-editor configurations in standalone browser editor

I'm using the monaco.editor.colorize and monaco.editor.tokenize API in my app for colorizing and formatting a long line of logs (I defined the language id), however I find no way of configuring, for ...
Frank Li's user avatar
0 votes
1 answer
102 views

Java/JavaFX - Shopping Cart System - Tokenizing data into an Arraylist

So I'm working on a school assignment. I have to create a shopping cart system that displays books in a ListView and have buttons/menu items that can add those books to a cart or remove them. The ...
Newbie123's user avatar
2 votes
0 answers
260 views

Transformers tokenizer batch encode a list of lists of strings

I have my dataset in the format of a list(A) of lists(B) of strings: foo = [['asdf', 'hello', 'john matthew'], ['asdf', 'good bye', 'luke caleb'], ['asdf', 'see you tomorrow', 'peter ...
loose the fools juice's user avatar

1
2 3 4 5
60