I have a dictionary d
with around 500 main keys (name1
, name2
, etc.). Each value is itself a small dictionary with 5 keys called ppty1
, ppty2
, etc.), and the corresponding values are floats converted to strings.
I want to extract data faster than I presently do, based on a list of lists of the form ['name1', 'ppty3','ppty4']
(name1
could by any other nameX
and ppty3
and ppty4
could be any other pptyX
).
In my application, I have many dictionaries, but they differ only by the values of the fields ppty1
, ..., ppty5
. All the keys are "static". I do not care if there are some preliminary operations, I would just like the processing time of one dictionary to be, ideally, much faster than now. My poor implementation, consisting in looping over every field takes about 3 ms.
Here is the code to generate d
and fields
; this is just to simulate dummy data, it does not need to be improved:
import random
random.seed(314)
# build dictionary
def make_small_dict():
d = {}
for i in range(5):
key = "ppty" + str(i)
d[key] = str(random.random())
return d
d = {}
for i in range(100):
d["name" + str(i)] = make_small_dict()
# build fields
def make_row():
line = ['name' + str(random.randint(0,100))]
[line.append('ppty' + str(random.randint(0,5))) for i in range(2)]
return line
fields = [0]*300
for i in range(300):
fields[i] = [make_row() for j in range(3)]
For example, fields[0]
returns
[['name420', 'ppty1', 'ppty1'],
['name206', 'ppty1', 'ppty2'],
['name21', 'ppty2', 'ppty4']]
so the first row of the output should be something like
[[d['name420']['ppty1'], d['name420']['ppty1'],
[d['name206']['ppty1'], d['name206']['ppty2']],
[d['name21']['ppty2'], d['name21']['ppty4']]]]
My solution:
start = time.time()
data = [0] * len(fields)
i = 0
for field in fields:
data2 = [0] * 3
j = 0
for row in field:
lst = [d[row[0]][key] for key in [row[1], row[2]]]
data2[j] = lst
j += 1
data[i] = data2
i += 1
print time.time() - start
My main question is, how to do improve my code? Few additional question:
- Later, I need to do some operations such as column extraction, basic operation on some entries of
data
: would you recommend storing the extracted values directly in an np.array? - How to avoid extracting the same values multiple times (
fields
has some redundant rows such as['name1', 'ppty3', 'ppty4']
)? - I read that things such as
i += 1
take a little bit of time, how can I avoid them?
append
calls for side-effects inside a listcomp, a trivial microoptimization that also makes the code nicer is replacing all those'name' + str(i)
with string formatting. For example, in a quick test on my laptop, your version takes a couple orders of magnitude longer than'name %d' % (i,)
, for almost 5x as much savings as using a listcomp in place of a for statement. (Although I doubt either one actually matters here.)