0

I am trying to build a generic conversion program, where input is csv files, and their schemas in a separate file. The output is a compressed parquet file.

when i embed the schema of the csv within the program, it works just fine. however, the momemnt i put the same schema in a separate physical file, I get the following error message after reading the first line from schema file -

Caused by: java.lang.IllegalArgumentException: No enum constant org.apache.parquet.schema.Type.Repetition.

I have tried to look it up but there is literally no reference to this error in Parquet context anywhere.

Just for reference, here is the schema -

message root {
  required binary name (UTF8);
  required binary city (UTF8);
}

please suggest way forward/mistake on my part !!

1 Answer 1

3

I was able to fix this issue by defining the schema as:

message root {   optional int32 voter_id;  optional binary name (STRING);  optional int32 age;  optional binary registration (STRING);  optional double contributions;  optional int32 voterzone;  optional int64 date_time (TIMESTAMP(MILLIS,true));};

That is, all the columns in a single line. I found that, in windows, when we define the columns in multiple lines, invisible end of line characters get added (\r\n) which causes this crash. Alternately, you can define the schema in code as :

String message = "message root { " + 
            "  optional int32 voter_id;" + 
            "  optional binary name (STRING);" + 
            "  optional int32 age;" + 
            "  optional binary registration (STRING);" + 
            "  optional double contributions;" + 
            "  optional int32 voterzone;" + 
            "  optional int64 date_time (TIMESTAMP(MILLIS,true));" + 
            "};";

This works fine too.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.