I need to validate the each column of a data frame with expected length or not. If not, one new column need to be added to the data frame which populates consolidation the validation results. I need this to be written using python.
For example, Below is my data frame structure. And my validation criteria is total length of any columns should not exceed more than 3.
[col1 col2 col3
=========================
AAA BBB CCC
DDDD EEE BBBB
AAA EEEE BBBB
And I am expecting output as below.
col1 col2 col3 length_check
======================================================================================
AAA BBB CCC
DDDD EEE BBBB Expected Length of col1 is 3; Expected Length of col3 is 3
AAA EEEE BBBB Expected Length of col2 is 3; Expected Length of col3 is 3
Your inputs are much appreciated. Thanks
Code:
valid_rdd=parsed_file.map(lambda line: line if len(line)==4 else False)
.filter(lambda line:line!=False)
invalid_rdd=parsed_file.map(lambda line: line if len(line)!=4 else False)
.filter(lambda line:line!=False)
columns=['colA','colB','colC','colD']
df_valid=valid_rdd.toDF(columns)
df1=df_valid.withColumn('length_check', (f.when (f.length('colA')== 1, "True").otherwise("Expected Column length 1 but found "+ str(f.length('colA') + str(df_valid.colA) ))))