0

I have 176 .tsv files as a result of a gene alignment looking like these:

target_id length tpm
ENST0001 12 100
ENST0001 9 5

In these files, I expect a certain overlap between target_id columns but not complete, so I would like to do a full join and keep all rows. Additionally, I am interested in keeping only the tpm values of each file and rename the column according to the file name.

The expected dataframe would be something similar to:

target_id SRR100001 SRR100002
ENST0001 100 7
ENST00015 5 0

I am aware of the join function in bash, but it can be used for two files per time and if I understood correctly I cannot select specific columns...

Thank you in advance!

EDIT: The files are named as SRR*.tsv

2
  • What's your directory structure look like? Files are called e.g. SRR100001.tsv?
    – Albert
    Mar 22, 2021 at 15:18
  • That is correct. The pattern is SRR*.tsv. I will add it to my post for clarity.
    – Rina
    Mar 22, 2021 at 15:20

1 Answer 1

1

Let me know if this code works for you, it's hard to test without having the files.

import re
import os
import sys
import pandas as pd

tpm_dict = {}

for fn in os.listdir(sys.argv[1]):
    if re.match('.*\.tsv$', fn):
        header = fn.replace('.tsv', '')
        this_df = pd.read_csv(os.path.join(sys.argv[1], fn), sep='\t')
        for i, row in this_df.iterrows():
            try:
                tpm_dict[row['target_id']][header] = row['tpm']
            except KeyError:
                try:
                    tpm_dict[row['target_id']] = {header: row['tpm']}
                except:
                    print(f"Problem in {fn} at row {i}")

df = pd.DataFrame.from_dict(tpm_dict, orient='index')
df.to_csv('joined.tsv', sep='\t')

Save as tsvjoin.py and then run python3 tsvjoin.py <folder with TSVs>

Edit: typos

7
  • Thanks Albert. I get however an error FileNotFoundError: [Errno 2] File SRR1146093.tsv does not exist: 'SRR1146093.tsv'. Any ideas why?
    – Rina
    Mar 22, 2021 at 15:44
  • My bad, I forgot to append the directory when reading. Try it now.
    – Albert
    Mar 22, 2021 at 15:49
  • Thanks for the quick reply and fix! I got another error now TypeError: 'str' object cannot be interpreted as an integer
    – Rina
    Mar 22, 2021 at 15:53
  • I made some changes, you can try it again
    – Albert
    Mar 22, 2021 at 16:22
  • I still get the same error, unfortunately, plus something new. TypeError: 'str' object cannot be interpreted as an integer During handling of the above exception, another exception occurred: ........ KeyError: 'tpm'
    – Rina
    Mar 22, 2021 at 16:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.