1

I currently am working on a bioinformatics project that currently involves a dictionary corresponding to about 10million unique keys, which each return a subset of categorical strings.

I currently use unpickle a dictionary object, but my main issue is that unpickling takes a very long time. I also need to iterate through a file, generating a set of keys(~200) for each row, lookup the keys, appending the list to a list-of-lists, and then subsequently flattening the list to generate a counter object of value frequencies for each row, and I have heard that a SQL database like structure would end up trading load times for lookup times.

The file that has keys typically contain about 100k rows and so this was my best solution, however it seems like even on faster pcs with increased ram, num of cores, and NVME storage that the time spent on loading the database is extremely slow.

I was wondering what direction (different database structure, alternatives to pickle such as shelves or mashall, parallelizing the code with multiprocess) would provide an overall speed up (either through faster loading times, faster lookup, or both) to my code?

Specifically: Need a create databases of the format key -> (DNA sub-sequence) : value ->[A,B,C,Y,Z] on the order of 1e6/1e7 entries.

When used, this database is loaded, and then given a query file (1e6 DNA sequences to query), perform a lookup of all the sub sequences in each sequence do the following.

For each query:

  1. slice the sequence into subsequences.
  2. Lookup each subsequence and return the list of categoricals for each subsequence
  3. Aggregate lists using collections.Counter

I was wondering how to either:

  1. Speed up the loading time of the database, either through a better data structure, or some optimization
  2. Generally improve the speed of the run itself (querying subsequences)
3
  • 1
    What is the data type of the keys? How many different categorical strings do you have? How long are the strings (min, max and avg)? Oct 30, 2020 at 19:31
  • The keys are strings consisting of DNA ('ATTCGGT') of a fixed length (typically 31). I would say about 100 categorical strings they are names and vary with an average ~10characters ranging from 5 to 20 Oct 31, 2020 at 14:43
  • Ok. So to clarify you want a kind of key-value database that map of type str->str with 10e6 elements with small keys of size ~31 and small values (with no more than 100 different values). Is that correct? How many lookup do you perform? What kind of time do you consider as "extremely slow"? Please edit the question to add those additional useful information. Oct 31, 2020 at 16:44

1 Answer 1

0

I'm not sure there is a right answer here since there are some tradeoff, BUT.

two options come to mind:

1st. consider using panads.DataFrame for the data-stucture. It will allow serialization/deserialization to many formats (I believe CSV should be the fastest but would give SQL a try). as for query time, it should be much faster than a dict for the complex queries.

2nd. key value store, such as MongoDB that has map-reduce and other fancy query capilites, in this case the data is always available without loading times.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.