I have three lists as follows.
mylist = [[5274919, ["my cat", "little dog", "fish", "rat"]],
[5274920, ["my cat", "parrot", "little dog"]],
[5274991, ["little dog", "fish", "duck"]]]
myconcepts = ["my cat", "little dog"]
hatedconcepts = ["rat", "parrot"]
For each concept in myconcepts
, I want to get the count every other concept connected to it using mylist
. Then remove the hatedconcepts
from it. So, my output should looks like as follows.
{"my cat": [("my cat", 2), ("little dog", 2), ("fish", 1)],
"little dog": [("little dog", 3), ("my cat", 2), ("fish", 2), ("duck", 1)]}
I was using this code to do it.
import collections
myoutput = []
for concept in myconcepts:
mykeywords = []
for item in mylist:
if concept in item[1]:
for mykeyword in item[1]:
if mykeyword in hatedconcepts:
pass
else:
mykeywords.append(mykeyword)
if len(mykeywords) > 0:
sorted_keywords = collections.Counter(mykeywords).most_common()
myoutput.append(tuple((concept, sorted_keywords)))
print(myoutput)
The output of the code is:
[('my cat', [('my cat', 2), ('little dog', 2), ('fish', 1)]), ('little dog', [('little dog', 3), ('my cat', 2), ('fish', 2), ('duck', 1)])]
However now I have a huge mylist
with a size of 3GB and nearly 9000 myconcepts
. The hatedconcepts
count is only 20. It looks like it takes about two weeks to run using my current code. The main reason for this could be that my current program is O^3
which is not very efficient. Therefore, I am looking for ways to make my current program more efficient. I am even fine with pythonic solutions that even take 5-6 days to run. Please let me know your thoughts.
I have added a portion of mylist
in: https://drive.google.com/file/d/1M3EhIRwwKwD3Kv4zDsmXaH1D73tx0eF3/view?usp=sharing just to get some idea how it looks like.
I am happy to provide more details if needed.
ram
if the list lays in a file then threads wont help you (IO Bound process
)