I have two dataframes with the following structure:
Dataframe A:
Current Accession | String |
---|---|
A_1 | AAAABBBC |
A_2 | AAACR |
This data frame contains 1 millions strings.
Dataframe B:
Accession | String |
---|---|
C_34 | RRRQAAAABBBC |
C_35 | RAAAABBBC |
C_36 | WWWWAAACR |
I want to get a final dataframe by looking with the substring in dataframe A into dataframe B and create a new column with the new accessions found, results should look like:
Current Accession | String | Mapped Accession |
---|---|---|
A_1 | AAAABBBC | [C_34,C_35] |
A_2 | AAACR | [C_36] |
I have explored join in pyspark but it needs exact match. Which doesn't work with sub-string matching.