I have large CSVs where I'm only interested in a subset of the rows. In particular, I'd like to read in all the rows which occur before a particular condition is met.
For example, if read_csv
would yield the dataframe:
A B C
1 34 3.20 'b'
2 24 9.21 'b'
3 34 3.32 'c'
4 24 24.3 'c'
5 35 1.12 'a'
...
1e9 42 2.15 'd'
is there some way to read all the rows in the csv until col B exceeds 10. In the above example, I'd like to read in:
A B C
1 34 3.20 'b'
2 24 9.21 'b'
3 34 3.32 'c'
4 24 24.3 'c'
I know how to throw these rows out once I've read the dataframe in, but at this point I've already spent all that computation reading them in. I do not have access to the index of the final row before reading the csv (no skipfooter please)
csv
, grab the rows one at a time, stuff them in a list of lists, stop once you get the last row that you want, and then build aDataFrame
out of the resulting list of lists.