8

I need a function that decimates, removes m in n of, a numpy array. For example to remove 1 in 2 or remove 2 in 3. So an array which is: [7, 4, 3, 5, 9, 2, 4, 1, 6, 8]

decimated by 1:2 would become: [7, 3, 9, 4, 6]

I wonder if it is possible to reshape the array from 1d array N long to one that is 2d and N/2, 2 long then drop the extra dimension?

Ideally, rather than just dump the decimated samples, I would like to find the maximum value across each set (in this example pair) of values. For example: [7, 5, 9, 4, 8]

Is there a way to find the maximum value across each set rather than just to drop it?

The added challenge is that the point here is to plot the values.

The decimation is required because plotting every value is taking too long meaning that I have to reduce the size of an array before plotting it but I need to do this quickly. So for or while loops would take too long.

4
  • 3
    To answer part of your question you can subsample every other variable by indexing using [::2]
    – BenT
    Commented Jun 13, 2019 at 17:33
  • For quick plotting see stackoverflow.com/questions/54449631/…
    – user2699
    Commented Jun 13, 2019 at 17:46
  • BenT: so that would do the decimation thing. Thanks. Very easy at that level.
    – Richard
    Commented Jun 13, 2019 at 17:58
  • Thanks user2699, that looks really useful.
    – Richard
    Commented Jun 13, 2019 at 18:01

3 Answers 3

6

A quick and dirty way is

k,N = 3,18
a = np.random.randint(0,10,N) #[9, 6, 6, 6, 8, 4, 1, 4, 8, 1, 2, 6, 1, 8, 9, 8, 2, 8]
a = a[:-k:k] #[9, 6, 1, 1, 1]

This should work regardless of k dividing into N or not.

2
  • Thank you, this is just what I needed! Also I think this is the better answer because it is quick and doesn't make use of special libraries.
    – user185160
    Commented Oct 18, 2023 at 17:17
  • 1
    That's a really neat solution but I think the answer should be: k,N = 3,18 a = [9, 6, 6, 6, 8, 4, 1, 4, 8, 1, 2, 6, 1, 8, 9, 8, 2, 8] a = a[:-k+1:k] #[9, 6, 1, 1, 1, 8]
    – Johned
    Commented Mar 1 at 0:29
5

It is worth being afraid of simply throwing out readings, because significant readings can be thrown out.

For the tasks that you described, it is worth using decimation.

Unfortunately it is not in numpy, but it is in scipy.

In the code below, I gave an example when discarding samples leads to an error.

enter image description here

As you can see, the original data (blue) has a peak. And manual thinning can just skip it (green). If you apply deciamation from the library, then it will be included in the result (orange).

from scipy import signal
import matplotlib.pyplot as plt
import numpy as np
downsampling_factor = 2

t = np.linspace(0, 1, 50)
y = list(np.random.randint(0,10,int(len(t)/2))) + [50] + list(np.random.randint(0,10,int(len(t)/2-1)))


ydem = signal.decimate(y, downsampling_factor)
t_new = np.linspace(0, 1, len(ydem))

manual_decimation = y[:-downsampling_factor:downsampling_factor]
t_manual_decimation = np.linspace(0, 1, len(manual_decimation))

plt.plot(t, y, '.-', t_new, ydem, 'o-', t_manual_decimation,  manual_decimation, 'x-')
plt.legend(['data', 'scipy decimate', 'manual decimate'], loc='best')
plt.show()

In general, this is not such a trivial task, please be careful.

UPD: note that the length of the vector must be greater than 27.

3

to find the maximum:

1) k divides N:

k,N = 3,18
a = np.random.randint(0,10,N)
a
# array([0, 6, 6, 3, 7, 0, 9, 2, 3, 2, 5, 4, 2, 6, 9, 6, 3, 2])
a.reshape(-1,k).max(1)
# array([6, 7, 9, 5, 9, 6])

2) k does not divide N:

k,N = 4,21
a = np.random.randint(0,10,N)
a
# array([4, 4, 6, 0, 0, 1, 7, 8, 2, 3, 0, 5, 7, 1, 1, 5, 7, 8, 3, 1, 7])
np.maximum.reduceat(a, np.arange(0,N,k))
# array([6, 8, 5, 7, 8, 7])

2) should always work but I suspect 1) is faster where applicable

8
  • As far as I can work out Paul, this technique (1) does not work. The array remains 18 elements long. Tried it a couple of ways and with some variations and no decimation takes place.
    – Richard
    Commented Jun 15, 2019 at 4:18
  • The method (2) seems to work when N does divide by k, so one can use (2) instead of (1).
    – Richard
    Commented Jun 15, 2019 at 4:22
  • Thank you very much for the suggestions by-the-way Paul. Should have started with that!
    – Richard
    Commented Jun 15, 2019 at 4:25
  • @Richard you are welcome. Weird, though that 1) doesn't work for you. If you verbatim copy 1) does it not print a six element array in the end? Commented Jun 15, 2019 at 6:38
  • I did, and no it didn't. It returns the original array. I am using Python 3. Does that make a difference?
    – Richard
    Commented Jun 15, 2019 at 8:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.