Say I have a 2 dictionaries, each with around 100000 entries (each can be of different length):
dict1 = {"a": ["w", "x"], "b":["y"], "c":["z"] ...}
dict2 = {"x": ["a", "b"], "y":["b", "d"], "z":["d"] ...}
I need to perform an operation using these two dictionaries:
- Treat each dict item as a set of mapping (i.e list of all mappings in
dict1
would be"a"->"w"
,"a"->"x"
,"b"->"y"
and"c"->"z"
) - Only keep mappings in
dict1
if the reverse mapping exists indict2
.
The resulting dictionary would be:
{"a": ["x"], "b", ["y"]}
My current solution uses 2 m*n
all zeros dataframes where m
and n
are the lengths of dict1
and dict2
respectively and the index labels are the keys in dict1
and the column labels are the keys in dict2
.
For the first dataframe, I insert a 1
at each value where the index label -> column label represent a mapping in dict1
. For the second dataframe, I insert a 1
at each value where the column label -> index label represent a mapping in dict2
.
I then perform an element-size product between the two dataframes, which only leaves values that have a mapping "a1"->"x1"
in dict1
and "x1"->"a1"
in dict2
.
However, this takes up way too much memory and is very expensive. Is there an alternative algorithm I can use?