0

I'm applying a function to a Spark RDD, like so:

data_2 = sqlContext.createDataFrame(pandas_df,data_schema)

data_3 = data_2.rdd.map(lambda x: parallelized_func(x, **args*)).collect()

Now, the function parallelized_func looks something like this:

def parallelized_func(a,b,c):
    ####FUNCTION BODY#####
    print("unique identifier for each row in pandas_df")
    return {'df1':df1,'df2':df2}

The issue I'm facing is this: When I run the "data_3 = ..." statement above in a Databricks notebook, I want the to get the unique identifier that I'm printing inside parallelized_func to show up somewhere, on some console, because that would make it easier to debug when there's an issue with any row in the pandas_df dataframe.

I tried checking the std_out and std_err consoles for every executor that's running the jobs, but there's always a whole load of other statements that occupy most of the console (all Spark statements related to various tasks being executed, I assume). I can sometimes find my print statement in this vast sea of other statements, but it's a really inefficient and ineffective way of debugging.

Is there a better way I can go about printing a statement like this? Or a better way of finding it? Can I for instance suppress all other execution-related statements that Spark keeps throwing up on the console?

Attaching a snapshot of the other statements that get printed on the console.

1 Answer 1

0

Print it's not really good solution, because as you said there is tons of logs that spark's write (and print for debugging it's not good also).

  1. You can make logger that write your logs to somewhere else (thats way only your logs will be write over there) such as NFS / whereever u can write it.(even locally on the executors and then check for it)

  2. If you try to find the "corrupted" rows maybe, only for deubgging, filter only the corrupted rows and collect it to the driver, and then u'll can check the rows locally on the notebook.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.