0

I'm using a Python UDF which causes an error during the reduce phase.

java.lang.ClassCastException: java.lang.Double cannot be cast to org.apache.pig.data.DataByteArray

Here's the code for the UDF:

import math
outputSchema("score:double")
def confidenceLowerBound(numerator, denominator, constant):
    raw_score = numerator * 1.0 / denominator
    normalized_interval = math.sqrt( raw_score * (1 - raw_score) / denominator )

    wilson_score = raw_score - constant * normalized_interval
    return wilson_score

This is how I call the udf in pig.

register 'confidence_interval_compute.py' using jython as pyutils;
...
..
A = FOREACH A GENERATE $0, $1, $2, $3, $4, pyutils.confidenceLowerBound($3, $4, 4) AS score PARALLEL 20;
2
  • How does the schema of A look like which you iterate over? Which Pig version do you use? Oct 2, 2012 at 19:51
  • 3
    I hope this is a typo, but you're missing the @ before outputSchema (ie. @outputSchema("score:double") . The decorator will not be invoked if this is missing, meaning Pig will not know the output is a double. Apr 4, 2013 at 14:18

1 Answer 1

0

As @Ian Stevents pointed out in the comments, this is because you are having a typo in the decorator.

You should use

@outputSchema("score:double")

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.