Questions tagged [bert-language-model]

BERT, or Bidirectional Encoder Representations from Transformers, is a method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks. BERT uses Transformers (an attention mechanism that learns contextual relations between words or sub words in a text) to generate a language model.

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

Truncating a training dataset so that it fits exactly within the context window

I have a dataset where the total tokens once tokenised is around 5000. I was to feed that into a BERT-style model so I have to shrink it down to 512 tokens but I want to rearrange the text to train it ...
Shafiq Jetha's user avatar
  • 1,396
-5 votes
0 answers
27 views

Should I use TensorFlow or PyTorch for my project? [closed]

I'm working on a mentor-mentee matching project where I aim to use deep learning techniques and language models like BERT or GPT to analyze textual profiles of mentors and mentees for suitable matches....
Ayush Kumar's user avatar
0 votes
0 answers
58 views

How to implement LoRA? [closed]

I want to implement LoRA to fine-tune a BERT model. My code returns me all weights and I freeze them with .requires_grad=False. Now I want to add the BAx multiplication of LoRA to the weights of BERT ...
Christian01's user avatar
0 votes
1 answer
21 views

Memory-efficient BERT Text Embedding for Large Dataset Preprocessing in TensorFlow

I'm working with a dataset containing approximately 920,614 rows and multiple columns, including "orig_item_title," "sub_item_title," "is_brand_same," and "...
krishna kaushik's user avatar
0 votes
0 answers
10 views

Pretraining BERT Models from scratch vs Further Pretraining

I want to pretrain an Arabic BERT model on domain-specific data to make it suitable for a specific domain problem, which is the classification of citizen reviews about government services into ...
Ghada Mansour's user avatar
-1 votes
1 answer
31 views

Bert topic clasiffying over a quarter of documents in outlier topic -1

I am running Bert topic with default options import pandas as pd from sentence_transformers import SentenceTransformer import time import pickle from bertopic import BERTopic llm_mod = "all-...
RM-'s user avatar
  • 998
0 votes
1 answer
45 views

Assign transformer layers to BERT weights

I print the weight names and shape of the BERT transformer. Now, I want to assign the printed weights to the layers in the transformers architecture: In the following, I can assign query, key and ...
Christian01's user avatar
0 votes
1 answer
15 views

Hugging Face BERT custom model cannot produce config.json when trainer(model) is saved

`Hello experts, I am trying to write BERT model with my own custom model (adding layer end of BERT).It goes well and I would like to save model to avoid future training. I am using trainer API and so,...
Nwe Nwe's user avatar
-1 votes
0 answers
13 views

Predicting Tweet User engagement with BERT

I can predict user engagement given the tweet contents and the time it was posted. My initial approach uses a regression model on the BERT embeddings for the tweet contents. However, the time the ...
Anwesh saha's user avatar
0 votes
1 answer
44 views

How to resolve BERT HF Model ValueError: too many values to unpack (expected 2)?

I have a dummy dataset of two text columns and labels as below. import tensorflow as tf from transformers import BertTokenizer, TFAutoModelForSequenceClassification import numpy as np from datasets ...
krishna kaushik's user avatar
0 votes
1 answer
31 views

Understand the difference between the arguments "text" and "text_target" in the bert tokenizer from the huggingface transformers library [duplicate]

From the transformers library by huggingface from transformers import BertTokenizer tb = BertTokenizer.from_pretrained("bert-base-uncased") tb is not a wordpiece tokenizer. It has arguments ...
figs_and_nuts's user avatar
0 votes
1 answer
23 views

Understanding output_attentions

I have a question about output_attentions, and I need to make a heatmap about the attention of the final layer of the BERT model. But I do not know if the output_attentions[0] is the first or last ...
Mara de Jess Garcia Santiago's user avatar
0 votes
0 answers
12 views

BERT preprocessor to process .xlsx file

I am doing a music artist classification using BERT in Tensor Flow. The goal is that given lyrics, the model should be able to identify the artist. My dataset is organized in a .xlsx file with three ...
Muana Kasongo's user avatar
0 votes
1 answer
26 views

fine-tune BERT for semantic textual similarity

I’m trying to use BERT (or any language embedding models) to solve a semantic text similarity problem: given a product A, find product B, which is basically the same underlying product, with a few key ...
user9689002's user avatar
0 votes
1 answer
22 views

Clarification on Padding Process in BERT Model Construction

In my endeavor to construct a BERT model from the ground up for the purpose of gaining hands-on experience and a comprehensive understanding of the model, I have encountered a point of confusion ...
Ali Haider Ahmad's user avatar
-3 votes
0 answers
31 views

Rewrite a sentence Using GPT-2 [closed]

I am seeking to prompt GPT-2 to generate a sentence starting with 'The actor is ...'. Initially, GPT-2 generates 'The actor and his wife allegedly met with Jeev before the film was released in May ...
Baha Rababah's user avatar
-1 votes
0 answers
16 views

Error when Fine Tuning MobileBERT for Question Answering

i have a problem when i try to fine tuning MobileBERT for question answering model from Transformer using TensorFlow. I have a dataset in .txt format, and each line consists of a question and an ...
Gabriel Fernando's user avatar
-1 votes
0 answers
14 views

The raw output of the CNN classifier of a BERT+CNN GUI application is way off the raw output of the training model

I made a BERT + CNN model that classifies hate speech from twitter. While training, it showed positive metrics with around 89-90% in accuracy, precision, recall, and F1 score. However, when I tried to ...
Zeetro_'s user avatar
  • 19
1 vote
1 answer
36 views

Not able to see results from trained huggingface model

I'm new to huggingface and after reading the documentation, I've been trying to fine-tune DNABERT2 on my simple dataset. Basically, the idea is I have some DNA sequences that are labeled as '1' or '0',...
youtube's user avatar
  • 295
0 votes
1 answer
41 views

Implementing Dynamic Data Sampling for BERT Language Model Training with PyTorch DataLoader

I'm currently in the process of building a BERT language model from scratch for educational purposes. While constructing the model itself was a smooth journey, I encountered challenges in creating the ...
Ali Haider Ahmad's user avatar
0 votes
0 answers
18 views

BERT classifier adding previous and next row as context

I'm working on a classifier in order to sort the text of a page into utterances and other textual information. I've annotated some training data and wondering if there is a way to pass this contextual,...
fiskdill's user avatar
-2 votes
0 answers
11 views

need to implement the model contains embedding layer (AraBERT) plus netowrk layer (LSTM) be passed to graph attention network (GAT)

need to implement an aspect based sentiment analysis model on Arabic dataset, the model contains embedding layer (AraBERT) and it's output will be passed to netowrk layer (LSTM) and the output of LSTM ...
Shahid Ali Ghulam Qadir's user avatar
0 votes
0 answers
35 views

I can't train a model with mobilbert or Bert in tflite-model-maker

I'm trying to create a neural network for text classification, and according to the script from tensorflow and google, you can use MobilBert for this. First example - tensorflow and Second example - ...
Skyain's user avatar
  • 1
0 votes
1 answer
35 views

Why replace the masked token with random token in bert?

Masking in Bert is: Take 15% of all the tokens in the sequence. These are to be used in computing the MLM loss 80% of the time retain the mask 10% of the time replace the mask with the original token ...
figs_and_nuts's user avatar
0 votes
1 answer
45 views

AttributeError: Can't pickle local object. How to fix this issue?

I am trying to save a BERT model and am reaching the following error. I already tried using model.save(), but that gave me another error so I am trying to find a workaround. Here is the code I used to ...
Sid R.'s user avatar
  • 19
-1 votes
0 answers
19 views

AssertionError: Index file doesn't match expected format. Make sure that --dataset-impl is configured properly

image showing error !fairseq-train --task language_modeling data-bin/wikitext-103 --save-dir checkpoints/transformer_wikitext-103 --arch transformer_lm_wiki103 --max-update 286000 --max-lr 1.0 --t-...
amudala gopikrishna's user avatar
-1 votes
0 answers
25 views

How to extract the mixed_features from BLIP Caption and VQA?

My situation: Model: Latest version of BLIP as in BLIP-V1 The feature I want to extract: In "med.py", in the Class "BertLMHeadModel", there is a variation called "...
Yu Yan's user avatar
  • 11
0 votes
0 answers
40 views

While fine-tuning Bert - ValueError: not enough values to unpack (expected 2, got 1)

I have a binary annotated dataset and I am trying to fine-tune a bert model on it. This is how I created my dataset for PyTorch: # Defining parameters MAX_LEN = 100 TRAIN_BATCH_SIZE = 8 ...
matteodrive's user avatar
-1 votes
0 answers
16 views

multi label with multi class bert model

i want to get model with multi label with multi class For example, the data format is: train = ['text1', 'text2', .....] label = [[0,3,1,2,1],[1,3,0,2,1], ....] When you input data, the model predicts ...
kgko1199's user avatar
0 votes
1 answer
18 views

Am trying to use spacy with the bert transformer model to classify some values, I however keep getting this error

The code is in python, using spacy v3, the model is (en_core_web_trf)..I suspect it might be the cause, that am using the emphasized textwrong syntax, but i need to know for sure. what the hell else ...
levi mungai's user avatar
0 votes
0 answers
65 views

How do I finetune an NLP model?

I get an error when runing a python code to finetune a NLP model. WHAT I WANT TO DO I want to train CamemBERT and I got my inspiration for my code from https://huggingface.co/docs/transformers/tasks/...
CCDXDX's user avatar
  • 9
1 vote
1 answer
30 views

tgt_key_padding_mask in pytorch transformers BertModel

While going through the transformer documentation in PyTorch, I see that the tgt_key_padding_mask of shape (batch_size, tgt_seq_len) is used to indicate irrelevance of some parts of tgt because of ...
carpet119's user avatar
0 votes
1 answer
34 views

extracting names and associated labels from text with language model

I am trying to extract information from scientific literature on microalgae and i need to be able to scan a text for various names and find their corresponding category. As an simple example, say I ...
user2737728's user avatar
-1 votes
1 answer
83 views

Labelling 100k dataset for BERT-NER

How i label dataset untill 100k++ effectively?I will use it for BERT-NER?and if there is method can you give me like code/tutorial/source for implementing?thank you!BTW, dataset i will use for my BERT-...
Muh. Rayi's user avatar
1 vote
1 answer
61 views

Understanding Input of BERT pre-training

I have a question about BERT's pre-training - The original paper mentions that MLM and NSP are done simultaneously. How does this work exactly? Let's say we have a sample [CLS] Sentence 1 [SEP] ...
raveitin's user avatar
3 votes
1 answer
42 views

Semantic search with pretrained BERT models giving irrelevant results with high similarity

I'm trying to create a Semantic search system and have experimented with multiple pretrained models from the SentenceTransformers library: LaBSE, MS-MARCO etc. The system is working well in returning ...
Aftaab Zia's user avatar
0 votes
0 answers
40 views

System Memory crash and CUDA Out of Memory Issues with BERT Model Classification

Question: I'm encountering memory-related issues when attempting to classify a large number of entries using a custom-trained BERT model. I've tried running the code both on my local system and Google ...
Sundharesan Kumaresan's user avatar
1 vote
0 answers
196 views

How to add specialized words to a trained `bert-base-uncased` embedding using the text R package for PCA mapping?

I am using the trained bert-base-uncased model with the text R package for embedding of software help requests and produce a 2d word plot. Words that are split into tokens do not get a mapping. The ...
George Ostrouchov's user avatar
0 votes
0 answers
12 views

Unable to load the downloaded tensorflow model from SciBert Github repository

I am new to using the Bert model and its derivatives. I am trying to use SciBert for my own model and plan to fine-tune it for my specific task. I have downloaded the recommended Tensorflow model from ...
Moulik Gupta 50's user avatar
0 votes
0 answers
28 views

Problem with BERT model classifying intents

So I am working on a voice assistant project. And using Joint-BERT to classify the intent so they can be executed, however when for example the transcription says: Turn on light one in the bedroom the ...
Bechara EL Murr's user avatar
0 votes
1 answer
44 views

How to un-batch in PyTorch?

I have a very simple text encoding script that implements BERT. I want to pass a set of texts through BERT and store it's output for further investigation. I create a dataloader with tokenized text, ...
hithisispeter's user avatar
0 votes
0 answers
54 views

Train new Word Embedding for mBART

TL;DR: I want to train a (set of) new word embedding(s) for mBART instead of training it for BERT—how do I do that? Background: I found an interesting code here: https://github.com/tai314159/PWIBM-...
TiMauzi's user avatar
  • 225
0 votes
0 answers
82 views

How to download huggingface bert-base-uncased in China

I need to use huggingface bert-base-uncased in China. I tried this on my local computer (It has VPN installed), and it is working fine. from transformers import AutoTokenizer, AutoModelForMaskedLM ...
Adnan Ali's user avatar
  • 2,910
1 vote
0 answers
38 views

BERTopic Visualization in dark

I want to change the default visualizations within BERTopic to display a dark theme rather than a white or bright theme. Basically I'm trying to do: import plotly.io as pio pio.templates.default ...
RobjSky's user avatar
  • 21
0 votes
0 answers
17 views

Preserving formatting in a BERT-tokenized string

If I tokenize some string from transformers import AutoTokenizer t = AutoTokenizer.from_pretrained('bert-base-cased') tokens = t.tokenize("I don't think the situation is quite as cut-and-dried ...
tolUene's user avatar
  • 584
0 votes
0 answers
19 views

Giving the same for every question for a given context

I am using Simple Transformers Question & Answer BERT model. I have created a custom dataset for question and answer model and I have trained the model using the custom dataset. It is asked to ...
swetha reddy's user avatar
0 votes
0 answers
60 views

The expanded size of the tensor (1536) must match the existing size (512) at non-singleton dimension 1

I'm trying to train my BERT model for sentences binary classification task (Inputs are 2 sentences and output label: 1 not similar and 0 for not similar). The problem arises when I trained my model, ...
A Random Guy's user avatar
0 votes
1 answer
50 views

Map BERT token indices to Spacy token indices

I’m trying to make Bert’s (bert-base-uncased) tokenization token indices (not ids, token indices) map to Spacy’s tokenization token indices. In the following example, my approach doesn’t work becos ...
lrthistlethwaite's user avatar
0 votes
0 answers
44 views

How to make bert training reproduceable

I'm currently using PyTorch to fine-tune a binary classification model. I've successfully trained a satisfactory model, however, I couldn't reproduce the training results when others asked me to go ...
Lee Ian's user avatar
0 votes
0 answers
23 views

ValueError: unable to interpret topic as either a list of tokens or a list of ids

i want to get the coherence for my model, however, it returns the evalue error instead topic_id_to_name = { -1: ['obligation', 'intelligence', 'interview', 'responsibility', 'assistance', 'support', '...
reia ambrade's user avatar

1
2 3 4 5
37