Questions tagged [batch-normalization]
Batch Normalization is a technique to improve learning in neural networks by normalizing the distribution of each input feature in each layer across each minibatch to N(0, 1).
376
questions
0
votes
0
answers
28
views
Building model matrix to correct for batch effect with biological and technical replicates
I recently conducted some MASS SPEC for my samples. Each sample was run thrice through the machine. However, there was a large space of time between the first run and the consequent second and third ...
0
votes
1
answer
26
views
RuntimeError: Expected number of channels in input to be divisible by num_groups, but got input of shape [64, 16, 32, 32] and num_groups=32
I have EfficientNet working fine on my dataset. Now, I changed all the batch norm layers into group norm layers. I have already done this process with other networks like vgg16 and resnet18 and all ...
1
vote
0
answers
41
views
Tensorflow Batch Normalization results dependent on number of channels in input
I'm trying to port weights for batch normalization from tensorflow to pytorch and I encountered a strange issue in tensorflow. The below code excerpt computes batchnorm with 10 channels with input of ...
0
votes
0
answers
28
views
how to convert tf.contrib.layers.batch_norm to tf2.0
I am converting the following code
import tensorflow.compat.v1 as tf
def conv_layer(input_tensor,name,kernel_size,output_channels,initializer=tf.keras.initializers.VarianceScaling,stride=1,bn=False,...
0
votes
0
answers
15
views
ValueError: Exception encountered when calling layer "batch_normalization_4" (type BatchNormalization)
I am trying to design densenet using model-subclass method. In which I created one block of 5 different layers which is repeated (using for loop) as per user's input. problem is after 1st iteration ...
0
votes
1
answer
41
views
Pytorch: Disable only nn.Dropout() without using model.eval()
nn.Dropout() can be disabled by using model.eval().However by using .eval(), nn.BatchNorm1d() are also disabled. Because the distributions between train and test sets are different, I'd like to ...
0
votes
0
answers
24
views
Missed setting `training=True` call method in tensorflow, any problem?
I have trained a model in tensroflow for 4 days, and achieved good test and train loss, convereged well. But later realised that, I haven't forwarded the training=True argument in the call() method of ...
1
vote
1
answer
91
views
What do BatchNorm2d's running_mean / running_var mean in PyTorch?
I'd like to know what exactly the running_mean and running_var that I can call from nn.BatchNorm2d.
Example code is here where bn means nn.BatchNorm2d.
vector = torch.cat([
torch.mean(self.conv3....
0
votes
0
answers
19
views
feature scaling vs Batch normalisation
In batch normalisation, sigma and beta are used retain the "expressive power of the activation". (From "Batch Normalization: Accelerating Deep Network Training by Reducing Internal ...
0
votes
0
answers
19
views
What is the meaning of this FFDNet paper's line
I just reviewing one paper of FFDNet which is "FFDNet : Toward a Fast and Flexible Solution for CNN-Based Image Denoising"
In page 6 of this paper
" When the training error keeps ...
0
votes
0
answers
31
views
Calculating Batch normalization
Part 1
Im going through this article and wanted to try and calculate a forward and backward pass with batch normalization.
When doing the steps after the first layer I get a batch norm output that are ...
0
votes
1
answer
90
views
Batch Normalization in 4D data over features instead of channels (PyTorch)
I am trying to implement batch normalization in my CNN via nn.BatchNorm2d. My data is of size (N, C, H, W) where N is the batch size, C is the number of channels, and HxW is the image size. Now in 1d ...
0
votes
3
answers
70
views
Why is keras and BatchNormalization underlined?
I am trying to run a CNN python code, but at the top of the code, the following line has keras and BatchNormalization underline in red.
from keras.layers.normalization import BatchNormalization
I don'...
0
votes
1
answer
322
views
cannot import name 'BatchNormalization' from 'keras.layers.normalization'
I'm learning ObjectDetection from this website
I have installed ImageAI,Tensorflow and Keras.
Then when I run this in python
from imageai.Detection import ObjectDetection
I got
Traceback (most ...
0
votes
0
answers
45
views
'RaggedTensor' object has no attribute 'set_shape' when passing RaggedTensor to BatchNormalization layer in Tensorflow 2.8
I am trying to pass a 3D RaggedTensor of shape (batch_size=None, 1, 136) to a batch normalization layer, but am receiving the error in the title. I assume because RaggedTensors can very in dimensions, ...
1
vote
0
answers
40
views
BatchNormalization Bijector Wrong Result with "prob" method
I am trying to implement a normalizing flow according to the RealNVP model for density estimation.
First, I am trying to make it work on the "moons" toy dataset.
The model produces the ...
0
votes
1
answer
215
views
Is it a good idea to train BatchNorm layer with running stats?
As far as I know, BatchNorm will use batch stats in train mode, but use running stats (running_mean/running_var) in eval mode. How about just always use running stats in both train and eval mode?
In ...
0
votes
1
answer
131
views
Not able to switch off batch norm layers for faster-rcnn (PyTorch)
I'm trying to switch off batch norm layers in a faster-rcnn model for evaluation mode.
I'm doing a sanity check atm:
@torch.no_grad()
def evaluate_loss(model, data_loader, device):
val_loss = 0
...
0
votes
0
answers
45
views
Shuffle BN for MoCo in Tensorflow 2
I want to implement unsupervised contrastive learning model MoCo in TF2, but I have no idea how to implement the essential trick mentioned in the paper - Shuffling BN. I think I understand what ...
0
votes
0
answers
49
views
How do we know what type of normalization is suitable for our deep learning model?
according to this post three types of normalization are:
types of Normalization
In Batch Normalization, we compute the mean and standard deviation across the various channels for the entire mini ...
0
votes
1
answer
255
views
Tensorflow keras BatchNormalization for higher than 4-dimension Tensor (video input)
I'm trying to implement S3D[https://arxiv.org/pdf/1712.04851.pdf] for video classification and I encountered a problem with BatchNormalization.
Since the implementation that I'm dealing with is video ...
0
votes
1
answer
72
views
Layer normalization Weight re-scaling and re-centering
In the paper on Layer normalization (https://arxiv.org/pdf/1607.06450.pdf) they say that
layer normalization is invariant to scaling of the entire weight
matrix and invariant to a shift to all of the ...
1
vote
1
answer
57
views
Does batch normalization replace 'layers.experimental.preprocessing.Rescaling' in CNN models?
Does batch normalization replace 'layers.experimental.preprocessing.Rescaling' in CNN models?
Or we should first normalize the data and then use BN in the CNN model.
0
votes
0
answers
40
views
Batch Norm layers prevent learning when using pre-loaded architectures
I have tried using a few pre-loaded architectures in Tensorflow including tf.keras.applications.efficientnet.EfficientNetB3 and tf.keras.applications.MobileNetV3Large. I am working on medical images ...
1
vote
1
answer
141
views
batch size without batch normalization
I'm working on image super-resolution tasks with EDSR as a baseline model. Following EDSR, I'm not using any batch-norm layers in my model. I suddenly came up with a stupid question about batch-sizes.
...
0
votes
0
answers
36
views
transform tensorflow1.14 model to pytorch
I'm trying to transform my tensorflow1.14 model to pytorch, but I'm getting different result and I don't know why. I think there is something wrong with CNN/BN/optimizer. Can anyone figure what I did ...
0
votes
0
answers
29
views
Am I need to differentiate expected value and variance in BatchNorm backward implementation?
I'm implementing BatchNorm layer now. I use Module (similar to nn.Module ) as base class, but change some methods -- for forward-pass and backward-pass:
class BatchNormalization(Module):
EPS = 1e-...
0
votes
2
answers
394
views
Using BatchNorm1d layer with Embedding and Linear layers for NLP text-classification problem throws RuntimeError
I am trying to create a neural network and train my own Embeddings. The network has the following structure (PyTorch):
import torch.nn as nn
class MultiClassClassifer(nn.Module):
#define all the ...
0
votes
1
answer
450
views
Pytorch running_mean, running_var and num_batches_tracked are updated during training, but I want to fix them
In pytorch, I want to use a pretrained model and train my model to add a delta to the model result, that is:
╭----- (pretrained model) ------ result ---╮
input------------- (my model) --------...
0
votes
0
answers
86
views
Tensorflow 1.0 slim.batch_norm
I am using a model that was written in tensorflow 1.15. There batch normalization layer declared:
with tf.variable_scope(name) as scope:
slim.batch_norm(
input,
...
0
votes
2
answers
371
views
Changing BatchNormalization momentum while training in Tensorflow 2
I want batch normalization running statistics (mean and variance) to converge in the end of training, which requires to increase batch norm momentum from some initial value to 1.0. I managed to change ...
1
vote
0
answers
184
views
Batch Normalization Inverse Computation using Pytorch
I have implemented a BatchNorm class for computing both BatchNormalization and its inverse, when I test it with tensors with 1 batch, it works properly, but when I test it for multi-batch tensors it ...
0
votes
0
answers
67
views
Batch Normalization in Convolution layer
I understood the concept of Batch Normalization(BN) in FC layers but in the convolution layer there
is a thing that is ambiguous to me after we train our model and get the four needed parameters, ...
0
votes
0
answers
35
views
ZCA whitening/sphering images for U-Net as a pre-processing step
Zero Phase Component Analysis Sphering or ZCA sphering for image pre-processing has been widely used for CNNs as a wonderful normalization method, which I found intriguing.
But as I'm delving into the ...
0
votes
0
answers
23
views
Is there a way to read op params using graph apis or graph structure from TF 2.x keras batchnorm layer?
In TF1.x we can read Batch normalization layer’s parameters such as epsilon, momentum using graph structures / graph APIs. How could we achieve same in TF 2.x without the use of high-level keras APIs ...
0
votes
1
answer
63
views
i am building my keras transfer learning model like shown below One thing i am unable to do is set training=False for batchNorm layers in xception
As it you can see I dont wanna change the way I built this model there is a way to change but that converts xception model into some functional model and In model summary it just shows Xception ...
1
vote
1
answer
777
views
Why does Keras BatchNorm produce different output than PyTorch?
Torch:'1.9.0+cu111'
Tensorflow-gpu:'2.5.0'
I came across a strange thing, when using the Batch Normal layer of tensorflow 2.5 and the BatchNorm2d layer of Pytorch 1.9 to calculate the same Tensor , ...
0
votes
1
answer
92
views
Sequential batch processing vs parallel batch processing?
In deep learning based model training, in general batch of inputs are passed. For example for training a deep learning model with [512] dimensional input feature vector, say for batch size= 4, we ...
1
vote
1
answer
279
views
Why is it that when viewing the architecture in Netron, the normalization layer that goes right after the convolutional layer is not shown?
I test some changes on a Convolutional neural network architecture. I tried to add BatchNorm layer right after conv layer and than add activation layer. Then I swapped activation layer with BatchNorm ...
1
vote
1
answer
118
views
How to deal with batch normalization for multiple datasets?
I am working on a task of generating synthetic data to help the training of my model. This means that the training is performed on synthetic + real data, and tested on real data.
I was told that batch ...
0
votes
0
answers
56
views
plotRLE error because of the negative values after normalization(RUVg) of RNA-seq counts
Thanks for your attention to my question.
Here is my question.
I usually use RUVg normalization for normalize unwanted variations like batch effect and etc. and I use plotRLE and plotPCA functions ...
0
votes
1
answer
52
views
What is exact logic of performing batch normalization in deep learning?
After reading the research paper on batchnorm and its various descriptions in forums, I am still not clear how the basic computations are performed. The core of my questions is: a vector is normalized ...
0
votes
1
answer
526
views
Using Global Sum Pooling instead of Global Average Pooling in a CNN
I have a CNN whose basic structure is as follows,
convolutional layers -> global average pooling -> flatten -> dense -> output
The network that I have is independent of input size, so I ...
0
votes
1
answer
92
views
Every run (using pytorch+cuda) is different w/ batch norming, even when RNGs are seeded
Why does my model with batch norm layer behave differently in every running, when the model without batch norm layer performs the same. In my model, random seed has been set by:
np.random.seed(args....
1
vote
1
answer
99
views
tf.keras.BatchNormalization giving unexpected output
import tensorflow as tf
tf.enable_eager_execution()
print(tf.keras.layers.BatchNormalization()(tf.convert_to_tensor([[5.0, 70.0], [5.0, 60.0]])))
print(tf.contrib.layers.batch_norm(tf....
1
vote
1
answer
930
views
PyTorch BatchNorm2d Calculation
I am trying to understand the mechanics of PyTorch BatchNorm2d through calculation. My example code:
import torch
from torch import nn
torch.manual_seed(123)
a = torch.rand(3,2,3,3)
print(a)
print(...
0
votes
1
answer
338
views
Run 10 batch job in parallel kotlin
I have one batch, that makes some job in a sync method. It can take some time. For optimizing, I want to do create async method, that will execute 10 batches in one time in parallel. Is it possible in ...
1
vote
1
answer
379
views
Is batch normalization useful for small networks?
We know batch normalization (BN) speeds up training of deep neural networks. But does it help with small neural networks as well? I have been experimenting with a 6-layer convolutional-MLP network and ...
1
vote
1
answer
60
views
Model.train() with pre-trained weights makes results all 0 while model.eval() is fine
thanks for your attention to this matter.
I want to continue to train a model with its pre-trained weights. When I evaluate this pre-trained model with model.eval(), everything is fine and the model ...
3
votes
3
answers
2k
views
How to freeze batch-norm layers during Transfer-learning
I am following the Transfer learning and fine-tuning guide on the official TensorFlow website. It points out that during fine-tuning, batch normalization layers should be in inference mode:
Important ...