8

Given any DataFrame 2-dimensional, you can call eg. df.sample(frac=0.3) to retrieve a sample. But this sample will have completely shuffled row order.

Is there a simple way to get a subsample that preserves the row order?

2
  • I guess you can simply do: df.head(100) or df.tail(100) this would select first 100 rows from top or bottom
    – YOLO
    Jan 4, 2020 at 20:24
  • I know you already accepted an answer, I just want to double check what you mean by get a subsample that preserves the row order. Do you just want to sort the result sample, or change the way the sampling itself works?
    – AMC
    Jan 4, 2020 at 23:10

2 Answers 2

7

What we can do instead is use df.sample(), and then sort the resultant index by the original row order. Appending the sort_index() call does the trick. Here's my code:

df = pd.DataFrame(np.random.randn(100, 10))
result = df.sample(frac=0.3).sort_index()

You can even get it in ascending order. Documentation here.

4
  • Thanks, somehow I couldn't find this. Yet, maybe there is another option as this seems to be adding a sorting overhead in time complexity (works for me, through).
    – Nicolas
    Jan 4, 2020 at 21:00
  • Hmmm. If we sample randomly from the DataFrame, though, it's very hard for me to imagine mathematically enforcing this ordering constraint while preserving uniform randomness. Jan 4, 2020 at 21:10
  • Shouldn't the probability of each item involved in the result be equal to the frac value? If so, you could walk through the df and compare a uniform random value with the frac value and include or exclude depending on the result. That should result in O(n) time complexity.
    – Nicolas
    Jan 6, 2020 at 20:47
  • Oh, that sounds reasonable. I wonder how sample is implemented internally.... Jan 6, 2020 at 20:49
4

The way the question is phrased, it sounds like the accepted answer does not provide a valid solution. I'm not sure what the OP really wanted; however, if we don't assume the original index is already sorted, we can't rely on sort_index() to reorder the rows according to their original order.

Assuming we have a DataFrame with an arbitrary index

df = pd.DataFrame(np.random.randn(100, 10), np.random.rand(100))

We can reset the index first to get a RangeIndex, sample, reorder, and reinstate the original index

df_sample = df.reset_index().sample(frac=0.3).sort_index().set_index("index")

And this guarantees we maintain the original order, whatever it was, whatever the index.

Finally, in case there's already a column named "index", we'll need to do something slightly different such as rename the index first, or keep it in a separate variable while we sample. But the principle remains the same.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.