7

I'm writing a general program to read and plot large amounts of data from .txt files. Each file has a different number of columns. I do know that each file has 8 columns that I'm not interested in, so I can figure out the number of relevant columns that way. How can I read the data and sort each relevant column's data into a separate variable?

This is what I have so far:

datafile = 'plotspecies.txt'
with open(datafile) as file:
    reader = csv.reader(file, delimiter=' ', skipinitialspace=True)
    first_row = next(reader)
    num_cols = len(first_row)
    rows = csv.reader(file, delimiter = ' ', quotechar = '"')
    data = [data for data in rows]

num_species = num_cols - 8

I've seen people say that pandas is good for this sort of thing, but I can't seem to import it. I'd prefer a solution without it.

5
  • What do you want to sort exactly?
    – Maxime B
    Commented Jun 10, 2016 at 14:34
  • The data from each column, so that I can work with each species as a single variable.
    – evtoh
    Commented Jun 10, 2016 at 14:37
  • Can you provide an example of one row? Or give us any detail about the file? It's very hard to answer without that. Commented Jun 10, 2016 at 14:39
  • What kind of details do you need? The files are thousands of rows, with 8 irrelevant columns and a variable number of useful ones.
    – evtoh
    Commented Jun 10, 2016 at 14:48
  • Just one column that shows the delimiter and whether or not there's a header row would be very helpful. Commented Jun 10, 2016 at 15:11

2 Answers 2

13

Pandas is in fact the right solution here. The issue is that in order to robustly handle something where you aren't certain of the underlying structure there's a lot of edge cases you have to watch out for, and trying to shoe-horn it into the csv module is a recipe for headaches (though it can be done)

As far as why you can't import pandas the reason is that it doesn't come with python by default. One of the most important things to consider when picking up a language is the ecosystem of packages it gives you access to. Python happens to be one of the best in the respect, so to ignore everything that's not a part of standard python is to ignore the best part of the language.

If you're on a windows environment you should start by getting conda set up. This will allow you to seamlessly explore many of the packages available to python users with little overhead. This includes pandas, which is in fact the right way to handle this problem. See this link for more info on installing conda: http://conda.pydata.org/docs/install/quick.html

Once you're got pandas installed it's as easy as this:

import pandas
test = pandas.read_csv(<your_file>)
your_Variable = test[<column_header>]

Easy as that.

If you really, really don't want to use things that aren't in core python then you can do this with something like what follows, but you haven't given enough detail for an actual solution:

def col_var(input_file, delimiter):
    # get each line into a variable
    rows = open(input_file).read().splitlines()

    # split each row into entries
    split_rows = [row.split(delimiter) for row in rows]

    # Re-orient your list
    columns = zip(*split_rows)  

The least intuitive piece of this is the last line, so here's a little example showing you how it works:

>>> test = [[1,2], [3,4]]
>>> zip(*test)
[(1, 3), (2, 4)]
4
  • I'm on OS X. I installed Anaconda, but I can't get pandas from it. I'm likely doing something wrong.
    – evtoh
    Commented Jun 10, 2016 at 14:39
  • If you're on OS X then you don't have to bother with conda. Just use pip. I find pip to be much easier and less of a hassle than conda. pip.pypa.io/en/stable/installing Commented Jun 10, 2016 at 14:40
  • If I use pip it returns an error saying that I don't have permission.
    – evtoh
    Commented Jun 10, 2016 at 14:45
  • I installed pandas (somehow). Thanks for your advice!
    – evtoh
    Commented Jun 10, 2016 at 14:58
1

Well, you can use the csv module provided there is some kind of delimiter within the rows that sets the columns appart.

import csv

file_to_read_from = 'myFile.txt'

#initializing as many lists as the columns you want (not all)
col1, col2, col3 = [], [], []
with open(file_to_read_from, 'r') as file_in:
    reader = csv.reader(file_in, delimiter=';') #might as well be ',', '\t' etc
    for row in reader:
        col1.append(row[0]) # assuming col 1 in the file is one of the 3 you want
        col2.append(row[3]) # assuming col 4 in the file is one of the 3 you want
        col3.append(row[5]) # assuming col 6 in the file is one of the 3 you want

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.