1

Create function in Databricks Notebook to remove accents from words

import unicodedata
import sys

from pyspark.sql.functions import translate, regexp_replace

def make_trans():
    matching_string = ""
    replace_string = ""

    for i in range(ord(" "), sys.maxunicode):
        name = unicodedata.name(chr(i), "")
        if "WITH" in name:
            try:
                base = unicodedata.lookup(name.split(" WITH")[0])
                matching_string += chr(i)
                replace_string += base
            except KeyError:
                pass

    return matching_string, replace_string

def clean_text(c):
    matching_string, replace_string = make_trans()
    return translate(
        regexp_replace(c, "\p{M}", ""), 
        matching_string, replace_string
    ).alias(c)

But I am not able to change the value in the dataframe, if I execute the command as select it works, but when I apply this command the following error occurs

Command error: df['productName'] = clean_text(df['productName'])

TypeError: Column is not iterable

This command execute with sucess

df.select(clean_text("productName"))

Do I have to loop one line at a time? Is this correct way to work with spark + databricks?

1 Answer 1

1

Dataframes are immutable so you cannot change the value. You can however add a new column. So in your case:

df = df.withColumn("cleanProductName", clean_text(df['productName']))

That "feels" like duplication at first. But remember the dataframe is immutable so is always the same size. Think of it as a View in a SQL database. Hence the Select works.

If you really want you can drop the old column from the dataframe. But unless you actually use the column (select * from example) it will make no difference to the overall performance.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.