0

Need help merging two dictionaries using the keys from one to look at values in another. If returns true it would append its own values into the other dictionary (updating it.. but not overwriting already present values)

The code (sorry first custom script ever):

otuid2clusteridlist = dict()
finallist = otuid2clusteridlist
clusterid2denoiseidlist = dict()

#first block, also = finallist we append all other blocks into.
for line in open('cluster_97.ucm', 'r'):
    lineArray = re.split('\s+',line)
    otuid = lineArray[0]
    clusterid = lineArray[3]
    if otuid in otuid2clusteridlist:
        otuid2clusteridlist[otuid].append(clusterid)
    else:
        otuid2clusteridlist[otuid] = list()
        otuid2clusteridlist[otuid].append(clusterid)

#second block, higher tier needs to expand previous blocks hash
for line in open('denoise.ucm_test', 'r'):
    lineArray = re.split('\s+', line)
    clusterid = lineArray[4]
    denoiseid = lineArray[3]
    if clusterid in clusterid2denoiseidlist:
        clusterid2denoiseidlist[clusterid].append(denoiseid)
    else:
        clusterid2denoiseidlist[clusterid] = list()
        clusterid2denoiseidlist[clusterid].append(denoiseid)  

#print/return function for testing (will convert to write out later)
for key in finallist:
    print "OTU:", key, "has", len(finallist[key]), "sequence(s) which", "=", finallist[key]

Block one returns

OTU: 3 has 3 sequence(s) which = ['5PLAS.R2.h_35336', 'GG13_52054', 'GG13_798']
OTU: 5 has 1 sequence(s) which = ['DEX1.h_14175']
OTU: 4 has 1 sequence(s) which = ['PLAS.h_34150']
OTU: 7 has 1 sequence(s) which = ['DEX12.13.h_545']
OTU: 6 has 1 sequence(s) which = ['GG13_45705']

Block two returns

OTU: GG13_45705 has 4 sequence(s) which = ['GG13_45705', 'GG13_6312', 'GG13_32148', 'GG13_35246']

So the goal is to add block two's out put into block one. I would like it to add in like this

...
 OTU: 6 has 4 sequence(s) which = ['GG13_45705', 'GG13_6312', 'GG13_32148', 'GG13_35246']

I attempted dic.update but it just adds block twos contents into block one since the key is not present in block one.

I think my issue is more complicated, I need block two to look within block one's value for its key and append values into that list.

I have been trying for loops and .append (similar to the code already wrote) but I am lacking the overall knowledge of python to solve this.

Ideas?

Additions,

Some subsets of the data:

cluster_97.ucm (block one's file):

5 376 * DEX1.h_14175 DEX1.h_14175
6 294 * GG13_45705 GG13_45705
0 447 98.7 DEX22.h_37221 DEX29.h_4583
1 367 98.9 DEX14.15.h_35477 DEX27.h_779
1 443 98.4 DEX27.h_3794 DEX27.h_779
0 478 97.9 DEX22.h_7519 DEX29.h_4583

denoise.ucm_test (block two's file):

11 294 * GG13_45705 GG13_45705
11 278 99.6 GG13_6312 GG13_45705
11 285 99.6 GG13_32148 GG13_45705
11 275 99.6 GG13_35246 GG13_45705

I picked these subsets because the 2nd line in file one is what file two would would be updating.

If anyone wants to give it a shot.

2
  • OK, the first dictionary maps numerical ids to clusterids. The second maps clusterids to denoiseids. I have two questions. 1) In the first dictionary, if an id is mapped to multiple clusterids, what should happen? Should all the sets of denoiseids be appended? 2) Can denoiseids be the same as clusterids? And if so, should this process be "transitive"? In other words, if you append a bunch of denoiseids to a list in dict one, and some of them are also clusterids, should the denoiseids of those clusterids also be appended?
    – senderle
    Dec 2, 2012 at 2:52
  • @ 1) Are you asking if 1 = x could 2 = x as well? If that is the case no. The ID will 100% be unique in each file, when looking between files its different(I dereplicated the data set, clustered. So I made like sequences all equal 1 of 15 ID's over 5 different files). I would like to expand backwards up my pipeline. Meaning these 15 numeric numbers (block one) will ultimately account for 100,000 unique IDs. 2) yes, your getting at the root of my problem here. these number numbers only occur in the last file, all other files are linked via various ID's that slowly funnel down to just 15.
    – jon_shep
    Dec 2, 2012 at 3:02

1 Answer 1

0

Updated to reflect the matching on the values...

I think the solution to your problem can be found in the fact that lists a mutable in Python and variables with mutable values are just references. So we can use a second dictionary mapping the value to the list.

import re

otuid2clusteridlist = dict()
finallist = otuid2clusteridlist
clusterid2denoiseidlist = dict()
known_clusters = dict()

#first block, also = finallist we append all other blocks into.
for line in open('cluster_97.ucm', 'r'):
    lineArray = re.split('\s+',line)
    otuid = lineArray[0]
    clusterid = lineArray[3]
    if otuid in otuid2clusteridlist:
        otuid2clusteridlist[otuid].append(clusterid)
    else:
        otuid2clusteridlist[otuid] = list()
        otuid2clusteridlist[otuid].append(clusterid)

    # remeber the clusters
    known_clusters[clusterid] = otuid2clusteridlist[otuid]

#second block, higher tier needs to expand previous blocks hash
for line in open('denoise.ucm_test', 'r'):
    lineArray = re.split('\s+', line)
    clusterid = lineArray[4]
    denoiseid = lineArray[3]
    if clusterid in clusterid2denoiseidlist:
        clusterid2denoiseidlist[clusterid].append(denoiseid)
    else:
        clusterid2denoiseidlist[clusterid] = list()
        clusterid2denoiseidlist[clusterid].append(denoiseid)

    # match the cluster and update as needed
    matched_cluster = known_clusters.setdefault(clusterid, [])
    if denoiseid not in matched_cluster:
        matched_cluster.append(denoiseid)



#print/return function for testing (will convert to write out later)
for key in finallist:
    print "OTU:", key, "has", len(finallist[key]), "sequence(s) which", "=", finallist[key]

I was not sure if you needed clusterid2denoiseidlist or not, so I added a new known_clusters to hold the mapping from values to lists.

I'm not sure I covered all the edge cases in your real problem, but this generates the desired output given the supplied test inputs.

5
  • JimP, thanks I tried it no change in output. But, I do love to learn new commands.
    – jon_shep
    Dec 2, 2012 at 3:03
  • You are correct, I missed the fact that you were wanting to match on the value from the first block during the second block. I'll fix the answer.
    – JimP
    Dec 2, 2012 at 5:07
  • JimP, Thank you for the update. I will take a peak in the morning at the results.
    – jon_shep
    Dec 2, 2012 at 6:33
  • Yes sir! I could probably kiss you for that help. So I am going to add in 2 more chunks of code following the same rules. Could I just call on and hold in the same known_cluster and matched_cluster ? after each section? Or should I make a new holding dict() and matching dict() for each part? I would end with ~12 dictionaries that all added to one single final one.
    – jon_shep
    Dec 2, 2012 at 17:38
  • Yes, by all means, just keep populating those two dictionaries. Since your end goal is to end up with just one, there is no reason not to just use those and save the processing overhead of merging the results.
    – JimP
    Dec 4, 2012 at 3:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.