Questions tagged [array-broadcasting]
Broadcasting (or singleton expansion) applies a function element-wise across one or more multidimensional arrays, matching shapes of the arguments by repeating missing or singleton dimensions. Be sure to also tag the programming language; many languages with strong array support have implicit or explicit broadcasting behaviors, sometimes with idiosyncratic rules.
642
questions
2
votes
1answer
37 views
How to use the output of argmin as index with Numpy [duplicate]
I want to find the location of minima along a given axis in a rank-3 numpy array. I have obtained these locations with np.argmin, however I'm not sure how to "apply" this to the original ...
1
vote
1answer
23 views
Clever way to broadcast like a switch statement through a Pandas Dataframe
I have piecewise operation that I would do with a switch in other languages. Is there a way thru slicing or broadcasting that I can be efficient with doing something like the following? the data ...
1
vote
1answer
28 views
NumPy Insert Function: what are the effects of using a single integer index vs a list of indices? How is shape affected?
I have been studying NumPy for a while and a strange behavior stopped me.
I hope the following code snippets could help:
This is the array that I will be working with :
e = np.arange(1, 10).reshape((3,...
0
votes
2answers
24 views
Matrix Subtraction | ValueError: operands could not be broadcast together with shapes (1,30) (30,455)
I am performing what seemingly appears to be a legal operation in Linear Algebra but not for Numpy Python.
To give context; I am manually setting up an ANN, performing Backpropagation.
Here, I "...
-2
votes
1answer
35 views
Effective numpy array calculating in Python
I need to calculate this special variance estimate (see pic. below). I have feature matrix X - dxl (d - # features, l - # objects). It's simply to do this in for cycles:
var_list = []
for i ...
0
votes
1answer
23 views
ValueError: Invalid broadcasting comparison with block values - how to resolve it in pythonic way
Hi I have two data frames and trying to compare the values in it but facing a ValueError in broadcasting:
dict_1 = {'a': {0: [{'value': 'A123',
'label': 'Professional'},
{'value': 'B141', '...
0
votes
0answers
16 views
ValueError: Invalid broadcasting comparison with block values - pandas in pythonic way
Hi I have two data frames and trying to compare the values in it but facing a ValueError in broadcasting:
dict_1 = {'a': {0: [{'value': 'A123',
'label': 'Professional'},
{'value': 'B141', '...
3
votes
5answers
61 views
How to raise every element of a vector to the power of every element of another vector?
I would like to raise a vector by ascending powers form 0 to 5:
import numpy as np
a = np.array([1, 2, 3]) # list of 11 components
b = np.array([0, 1, 2, 3, 4]) # power
c = np.power(a,b)
desired ...
1
vote
1answer
24 views
Numpy - Searching in a 4D matrix (AKA messed-up meshgrids)
I am sorry if a similar question has been already posted in some way, but I could not find it anywhere so far. My problem is the following:
Suppose I have a 4D numpy matrix like this one
M= array([[[[...
0
votes
1answer
34 views
computing efficiently pairwise similarity/dissimilarity with ray and numpy
I would like to load a huge matrix from a parquet file and distribute the distance computation across several nodes in order to both save memory and speedup the computing.
So the input data own 42 000 ...
0
votes
1answer
22 views
How do I update 3 columns of dataframe A with 3 respective columns of dataframe B with one np.where statement
In the code below, I have created df_d that has daily data and a df_i that has intraday data with 5 min intervals.
I have 3 columns in df_d (Volume, Volume1 and Volume2) that I want to propagate to ...
2
votes
1answer
45 views
Array-Broadcasting in Cython Memoryview
I created a typed memoryview in cython and would like to multiply it by a scalar:
import numpy as np
import math
cimport numpy as np
def foo():
N = 10
cdef np.double_t [:, :] A = np.ones(...
0
votes
1answer
31 views
Mismatch of shapes when using advanced indexing
I am developing a custom classifier that acts like an ensemble, combining minor classifiers together and the output of the ensemble is actually a majority voting. One important thing to mention is ...
2
votes
1answer
30 views
Extract fixed number of elements per row in numpy array
Suppose I have an array a, and a boolean array b, I want to extract a fixed number of elements from the valid elements in each row of a. The valid elements are the ones indicated by b.
Here is an ...
2
votes
3answers
84 views
How can I repeat an array m times [duplicate]
I have an array, e.g. arr = [1, 2, 3, 4], and m = 3. I want to make a matrix with m rows, repeating that array. The output of the example would be
[[1, 2, 3, 4],
[1, 2, 3, 4],
[1, 2, 3, 4]]
How can ...
1
vote
1answer
24 views
How to average over conditionally selected numpy array entries in a 4D array based on an index from a 3D array
I want to average over conditionally selected elements in a 4D numpy array, based on an index using a 3D array.
In other words, my 4D array DATA has these dimensions: [ntime,nz,ny,nx]
where as my 3D ...
0
votes
1answer
18 views
Does pytorch broadcast consume less memory than expand?
Does pytorch operations using broadcast consume less memory than expand? For example, are the following two programs different in memory usage?
import torch
x = torch.randn(20,1)
y = torch.randn(1,20)
...
0
votes
0answers
29 views
Broadcast solve for triangular matrix
I have k triangular matrices of size d x d. For each one, I want to solve for n vectors (of dimension d).
As far as I know, numpy.linalg.solve allows me, using broadcasting, to perform all solves ...
1
vote
0answers
25 views
Torch distributed broadcast and reduce between CPU/ GPU devices
Using the torch.distributed package. I am trying to move tensors from CPU -> GPU0, GPU1 in two separate processes and update the master version (on CPU).
Assume I have two GPU's connected. One on ...
1
vote
1answer
38 views
Why isn't broadcasting with numpy faster than a nested loop
I have a calculation in my code that get carried out thousands of times and I wanted to see if I could make it faster as it is currently using two nested loops. I assumed that if I used broadcasting I ...
2
votes
1answer
30 views
Swap 2 numpy arrays based on condition from different arrays
I have 4 arrays, A,B,C,D. A and B have shape (n,n) and C/D have shape (n,n,m). I am trying to set it up so that when an element of A is greater than B, that array of length m belongs to C. In ...
1
vote
1answer
30 views
What is the right approach in this kind of broadcast?
So,what is the right approach in this broadcasting ?
I have used a for loop to verify my broadcasting output.
As you can see, it missed to broadcast the second element.
Any idea on this?
from numpy ...
0
votes
2answers
41 views
How to broadcast sum of list of list?
How to broadcast the sum of list of list in an efficient way?
below is a working code, but its not quite efficient when list1 has nth value like 30 elements.
Any improvement on this?
from numpy import ...
0
votes
2answers
52 views
numpy broadcasting - explanation of trailing axes
Question
Please elaborate the answer in Numpy array broadcasting rules in 2012, and clarify what trailing axes are, as I am not sure which "linked documentation page" the answer refers to. ...
0
votes
1answer
25 views
Multiplication of the type self.w.value[x]
I am reading a segment of a program that says
self.w.value[x]
Here, w.value has the shape (2000, 30)
x is an input, which has the shape (200, )
After calling self.w.value[x], I get an output that is (...
0
votes
1answer
35 views
Difference between x[:] and x[…] in Numpy
I am confused about the difference between x[:] and x[...] in Numpy.
For example, I have this 2-d array
[[4, 1, 9],
[5, 2, 0]]
When I tried to print out x[:] and x[...], they both gave me the same ...
0
votes
1answer
29 views
Compute multiplication (element-wise) over multiple batches of images using broadcasting
I am currently working on the mnist dataset to create a CNN.
My input is
X: Array of shape (batch_size, n_channels, image_height, image_width)
F: The filter to apply. Array of shape (n_channels, ...
0
votes
1answer
38 views
How to get the first character of all values out of numpy array (shaped (n, m)) filled with strings [duplicate]
So I have a large array shaped like (n, m) filled with strings. A small example would be:
string_array = np.array([
['hello', 'world'],
['greetings', 'all'],
['merry', 'christmas']
])
I'd ...
0
votes
1answer
35 views
Efficient numpy broadcasting not found
It may be an easy problem but I could not find any practical solution. My code has following code segment involving 3 nested for loops. The target is to create a specialized intensity matrix for my ...
0
votes
1answer
21 views
Efficient pytorch broadcasting not found
I have the following code snippet in my implemenatation. There is a nested for loop with 3 loops. In the main code the 3D coordinates of the original system is stacked as a 1D vector of constinous ...
-1
votes
1answer
57 views
Matrix/array operations inside a python function
I am trying to use the content of a python function to make a contour plot. For example I try:
import numpy as np
import matplotlib.mlab as mlab
import matplotlib.pyplot as plt
# Building my equation ...
0
votes
2answers
42 views
Using np.transpose to make arrays broadcast
I have a broadcasting error like the following:
ValueError: operands could not be broadcast together with shapes (84,36) (84,36,210,45)
Is there a way to get around this? I tried using np.transpose()...
2
votes
0answers
29 views
Generate all broadcast-able shapes from a given shape in numpy
I have an array shape
eg - (10, 3)
I am looking for a function in numpy that can generate all possible shapes having at least 1 dimension, that can broadcast with the given shape.
output eg - [(1,), (...
2
votes
1answer
72 views
What is the most efficient way to broadcast an operation on slices of PyTorch Tensors?
I have a tensor T of shape (b, r)
I want to do an operation for each (r), in a way that it gets parallelized by the GPU
The naive implementation, in numpy for simplicity, would look something like:
...
0
votes
0answers
31 views
match numpy arrays on rows
I have two arrays arr_all and arr_sub which by construction is such that all rows of arr_sub are in arr_all. The output I am looking for is a 1D array of size arr_sub.shape[0] which has the indices of ...
-1
votes
1answer
31 views
Compute distances between 2 dataframes based on boolean matrix as a mask
I have 2 dataframes where columns are features and rows are different items.
import pandas as pd
import numpy as np
import random
random.seed(0)
data1 = {'x':random.sample(range(1,100), 4), 'y':...
1
vote
2answers
39 views
How to evaluate a function in n variables in numpy?
How do I evaluate a function in n variables in numpy? For simplicity, let n = 3. Consider the following example:
x, y, z = numpy.linspace(0, 1, 100), numpy.linspace(0, 1, 100), numpy.linspace(0, 1, ...
1
vote
1answer
47 views
Which numpy function calls is cProfile hiding from me?
Solution
As it turns out, np.sum is a python function calling np.add.reduce. This latter ufunc call is reported by cProfile, I presume because this is still a python object. np.maximum and np.subtract ...
0
votes
0answers
18 views
Broadcast or tile numpy matrix and apply summation
Maybe I'm just overlooking something terribly trivial, but I can't seem to figure this out.
I'm looking for the optimal (fast, numpythonic) method to:
sum two matrices (A + B), by
summing the values ...
0
votes
1answer
34 views
Speeding up with np.select() or if else clause
Question for Python GIS developers. I have a high density 3D point cloud with up to 73 observations per pixel. Cloud was obtained with SFM. I am fusing the DEM made from the cloud as new bands in a ...
0
votes
1answer
45 views
How to convert array into special items of structured array and revert it back?
I want to perform some numpy methods on items of structured array instead of numbers. So, for example, while working with array of integers of shape (4, 3), I need to convert it to array of items of ...
0
votes
0answers
11 views
Error when broadcasting DataFrames with overlapping indices
I am attempting to calculate the monthly simulated revenue from electricity generated at each river system in a region by taking the product of a generation (gen) DataFrame and a price (futures) ...
0
votes
0answers
35 views
Numpy using multidimensional array to index a 1D array
I do not understand the following code, i.e. the last part of it.
max = np.max(rel_coords, axis=0)
min = np.min(rel_coords, axis=0)
bins = [np.arange(low, high) for low, high in zip(min, max)]
...
1
vote
1answer
33 views
Vectorisation and broadcasting
I need to vectorise the following for loop and I am new to broadcasting and vectorisation (and generally object orientated programming is new to me).
width = 1000
height = 400
for v in range(height):
...
2
votes
1answer
63 views
Broadcasting a vector into another vector of different size in numpy or python
I have Np links (pipes) and Nj junctions (nodes). Each link k has a start node i and end node j, and a link value of b. I want to calculate the contribution of links to each node by adding b if the ...
0
votes
2answers
50 views
Multiplying a 2D numpy array with every row of a 2D numpy array (Without using for loop)
I have two 2D numpy arrays as shown below
Matrix_A is a 2D array of shape(3,3)
Matrix_B is a 2D array of shape(9,3)
Matrix A = [[ 0. -1. 0.]
[ 1. 0. 0.]
[ 0. 0. 1.]]
...
1
vote
1answer
46 views
How to get a vector by indexing a matrix with pair of vectors?
Given two vectors that are the components of a list of indices, how do I index a matrix by them pairwise? If I just use them as is, I get the cartesian product (as is consistent with fancy indexing ...
3
votes
1answer
51 views
Add multiple np.newaxis as needed?
I would like to pairwise compare (with <=) all elements of two NumPy ndarrays A and B, where both arrays can have arbitrary dimensions m and n, such that the result is an array of dimension m + n.
...
1
vote
1answer
44 views
Why I can not calculate distance between two numpy array?
1.I had two numpy arrays which are data_test and data_train respectively
data_partial_test = data_test[:2000,:]
test_lable = label_test
print(test_lable.shape)
print(data_partial_test[...
1
vote
1answer
46 views
Between shapes () and (1,), why can I perform regular but not in-place operations?
When I try to broadcast in-place from shape (1,) to shape (), numpy raises ValueError: non-broadcastable output operand with shape () doesn't match the broadcast shape (1,). I understand that ...