I currently am working on a bioinformatics project that currently involves a dictionary corresponding to about 10million unique keys, which each return a subset of categorical strings.
I currently use unpickle a dictionary object, but my main issue is that unpickling takes a very long time. I also need to iterate through a file, generating a set of keys(~200) for each row, lookup the keys, appending the list to a list-of-lists, and then subsequently flattening the list to generate a counter object of value frequencies for each row, and I have heard that a SQL database like structure would end up trading load times for lookup times.
The file that has keys typically contain about 100k rows and so this was my best solution, however it seems like even on faster pcs with increased ram, num of cores, and NVME storage that the time spent on loading the database is extremely slow.
I was wondering what direction (different database structure, alternatives to pickle such as shelves or mashall, parallelizing the code with multiprocess) would provide an overall speed up (either through faster loading times, faster lookup, or both) to my code?
Specifically: Need a create databases of the format key -> (DNA sub-sequence) : value ->[A,B,C,Y,Z] on the order of 1e6/1e7 entries.
When used, this database is loaded, and then given a query file (1e6 DNA sequences to query), perform a lookup of all the sub sequences in each sequence do the following.
For each query:
- slice the sequence into subsequences.
- Lookup each subsequence and return the list of categoricals for each subsequence
- Aggregate lists using collections.Counter
I was wondering how to either:
- Speed up the loading time of the database, either through a better data structure, or some optimization
- Generally improve the speed of the run itself (querying subsequences)