2

I need to generate samples from a list of numbers in a scenario where I might have the situation that I need to sample more numbers than I have. More explicitly, this is what I need to do:

  • Let the total number of elements in my list be N.

  • I need to sample randomly without replacement from this list M samples.

  • If M <= N, then simply use Numpy's random.choice without replacement.

  • If M > N, then the samples must consist X times all the N numbers in the list, where X is the number of times N fully divides M, i.e. X = floor(M/N) and then sample additional M-(X*N) remainder samples from the list without replacement.

For example, let my list be the following:

L = [1, 2, 3, 4, 5]

and I need to sample 8 samples. Then firstly, I sample the full list once and additional 3 elements randomly without replacement, e.g. my samples could then be:

Sampled_list = [1, 2, 3, 4, 5, 3, 5, 1]

How can I implement such a code as efficiently as possible in terms of computation time in Python? Can this be done without for-loops?

At the moment I'm implementing this using for-loops but this is too inefficient for my purposes. I have also tried Numpy's random.choice without replacement but then I need to have M <= N.

Thank you for any help!

1
  • Why close this question? What is wrong with it?
    – jjepsuomi
    Sep 15, 2016 at 9:20

4 Answers 4

3

You can concatenate the results of repeat and random.choice:

np.concatenate((np.repeat(L, M // len(L)), np.random.choice(L, M - M // len(L))))

First, the sequence is repeated as often as necessary, then a choice is made for the remaining number needed; finally, the two arrays are concatenated.

Note that you can easily determine whether choice works with replacement or without, using the replace parameter:

replace : boolean, optional -- Whether the sample is with or without replacement

0
2

I would just wrap numpy's random.choice() like so:

L = [1, 2, 3, 4, 5]

def wrap_choice(list_to_sample, no_samples):
    list_size = len(list_to_sample)
    takes = no_samples // list_size
    samples = list_to_sample * (no_samples // list_size) + list(np.random.choice(list_to_sample, no_samples - takes * list_size))
    return samples

print(wrap_choice(L, 2))   # [5, 1]
print(wrap_choice(L, 13))  # [1, 2, 3, 4, 5, 1, 2, 3, 4, 5, 3, 3, 1]

Edit: There is no need to check for the length. The algorithm you have for when the requests are more than the list's length also works when this is not the case.

6
  • Almost exactly what I need, just without replacement :) Thank you very much! Also I want it to be as fast as possible :)
    – jjepsuomi
    Sep 15, 2016 at 9:28
  • there is no replacement. Notice the 3, 3, 1 at the end of the second call's print.
    – Ma0
    Sep 15, 2016 at 9:30
  • All the suggestions given were good and almost equal in time, but I think yours was ultimately the fastest :)
    – jjepsuomi
    Sep 15, 2016 at 10:40
  • Does not work when L is a np.array() due to difference between list multiplication and array multiplication. Example: wrap_choice(np.array(L), 13). If this is a requirement OP would be better off using one of the np.repeat() solutions.
    – mhawke
    Sep 15, 2016 at 11:07
  • @mhawke as far as i understand the question, both the original structure as well as the final one (with the samples) are or have to be lists. The only thing that is an np.array() here is what the np.random.choice() produces
    – Ma0
    Sep 15, 2016 at 11:36
1

Here is what might be a solution for the case where 0 < M-N < max(L) :

import numpy as np
from numpy.random import random

l = np.array([1, 2, 3, 4, 5])
rand = [ i for i in l[np.argsort(np.amax(l))[:M-N]] ]

new_l = np.concatenate(l,rand)

Here is an example :

l = np.array([1,2,3,4,5])
M, N = 7, len(l)
rand = [i for i in l[np.argsort(np.random(np.amax(l)))][:M-N]]
new_l = np.concatenate(l,rand)

And here is the output :

new_list = np.array([1,2,3,4,5,3,4])
1
  • Actually I just realized that it just works for cases where M-N <= np.max(L).
    – MMF
    Sep 15, 2016 at 9:41
1

Use divmod() to get the number of repetitions of the list and the remainder/shortfall. The shortfall can then be randomly selected from the list using numpy.random.choice().

import numpy as np

def get_sample(l, n):
    samples, shortfall = divmod(n, len(l))
    return np.concatenate((np.repeat(l, samples), np.random.choice(l, shortfall, False)))


>>> get_sample(range(100), 10)
array([91, 95, 73, 96, 18, 37, 32, 97,  4, 41])
>>> get_sample(range(10), 100)
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2,
   2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4,
   4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6, 6,
   6, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 9, 9,
   9, 9, 9, 9, 9, 9, 9, 9])
>>> get_sample([1,2,3,4], 0)
array([], dtype=int64)
>>> get_sample([1,2,3,4], 4)
array([1, 2, 3, 4])
>>> get_sample([1,2,3,4], 6)
array([1, 2, 3, 4, 4, 3])
>>> get_sample([1,2,3,4], 6)
array([1, 2, 3, 4, 3, 2])

>>> get_sample(list('test string'), 6)
array(['n', 's', 'g', 's', 't', ' '], 
  dtype='|S1')
>>> get_sample(np.array(list('test string')), 4)
array(['r', 't', 's', 'g'], 
  dtype='|S1')
1
  • Updated to use np.repeat() instead of list multiplication because multiplication of a np.array behaves differently. This produces slightly different output where the repeated items are grouped individually together, but that seems to agree with the requirements.
    – mhawke
    Sep 15, 2016 at 10:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.