0

"|" delimited files should have below column headers

Activity
Activity+ ID
Description
Status

After upload, before start processing the file using SQLLDR, I make sure the uploaded file has exact number of headers, headers names are matching and in same order.

Code:

declare -i header=4
fields=( 
"Activity"
"Activity+ ID"
"Description"
"Status"
)

for i in "Test File.csv"; do
    read -r line < "$i" 

    oldIFS="$IFS"
    IFS=$'|'
    fldarray=( $line );
    IFS="$oldIFS"

    nfields=${#fldarray[@]}     
    if (( nfields < header ))
    then
    printf "error: only '%d' fields in file '%s'\nmissing:" "$nfields" "$i"
    else        
        for item1 in "${header[@]}"; do
          for item2 in "${fields[@]}"; do
           if [[ $item1 != $item2 ]]; then
            Array3+=("$item1")
           fi
         done
        done
        echo "not matching" ${Array3[@]}
        printf "\n\n"
    fi
done

Data:

Activity|Activity+ ID|Description|Status
Test|1234|First activity|Open

This is always printing that Activity+ column is missing, though it is present in the file. After I remove the "+" from header and file uploaded, it is working as expected. How can I change the above code to validate column headers with "+". I referred the answer from bash to identify and verify file headers to build this solution

9
  • Please show the exact content of file the file. "Should have below column headers", what if it does not? Also check that your line endings are ok (dos2unix).
    – Nic3500
    Oct 25, 2021 at 23:58
  • I just ran your code on my system, and "Activity+" is not reported as missing. But I had to guess the format of the input file. Please add so I can check again.
    – Nic3500
    Oct 26, 2021 at 0:08
  • @Nic3500 updated question. Exact column name is "Activity+ ID", i executed this now again and saying "Activity+ ID" missing column header.
    – Pat
    Oct 26, 2021 at 2:03
  • Can the headers show up in different order? The quick & dirty solution would be to compare line 1 with the fixed string Activity|Activity+ ID|Description|Status. Oct 26, 2021 at 13:30
  • @JeffSchaller I cannot do that because i need to know which column header is not matching
    – Pat
    Oct 26, 2021 at 13:50

1 Answer 1

1

NOTE: still a bit confused as to what OP wants to do (eg, header is defined as an integer but later referenced as an array ("${header[@]}"))

Assumptions:

  • print an error if the number of | delimited fields in the first row of the .csv file does not match the number of entries in the fields[] array
  • header fields from the .csv file must be an exact match (spelling and order) as the entries in the fields[] array
  • print the entries from the fields[] array that don't have an exact match with the | delimited fields from the first row of the .csv file

We'll keep the current fields[] array:

fields=("Activity" "Activity+ ID" "Description" "Status")

The pull the first line of the .csv file into the headers[] array:

IFS='|' read -r -a headers < test.csv      # read first line from test.csv, break on '|' delimiter, store in headers[] array

Giving us:

$ typeset -p fields headers
declare -a fields=([0]="Activity" [1]="Activity+ ID" [2]="Description" [3]="Status")
declare -a headers=([0]="Activity" [1]="Activity+ ID" [2]="Description" [3]="Status")

Now make some modifications to OP's if/else/for/fi code:

if [[ "${#fields[@]}" -ne "${#headers[@]}" ]]            # field count mismatch?
then
     echo "error: field count mismatch: expecting ${#fields[@]} / found ${#headers[@]}"
else
    Array3=()                                            # init array Array3[]

    for ((i=0;i<${#fields[@]};i++))                      # loop through indices of fields[] array
    do
        [[ "${fields[$i]}" != "${headers[$i]}" ]] && \   # if same position in both arrays is not a match then ...
        Array3+=("${fields[$i]}")                        # add fields[] entry to Array3[]
    done

    [[ "${#Array3[@]}" -ne 0 ]] && \                     # if Array3[] not empty then ...
    echo "not matching:" ${Array3[@]}                    # print list of fields to stdout
fi

For this particular case, where ${fields[@]} and ${headers[@]} are identical, no output is generated.

Other test cases:

2nd field in headers[] is spelled differently

declare -a fields=([0]="Activity" [1]="Activity+ ID" [2]="Description" [3]="Status")
declare -a headers=([0]="Activity" [1]="Activity+" [2]="Description" [3]="Status")

# the code generates:

not matching: Activity+ ID

headers[] has 3 entries

declare -a fields=([0]="Activity" [1]="Activity+ ID" [2]="Description" [3]="Status")
declare -a headers=([0]="Activity" [1]="Activity+ ID" [2]="Status")

# the code generates:

error: field count mismatch: expecting 4 / found 3

headers[] has 4 entries but all differ from corresponding entry in fields[]

declare -a fields=([0]="Activity" [1]="Activity+ ID" [2]="Description" [3]="Status")
declare -a headers=([0]="Activity+ ID" [1]="Description" [2]="Status" [3]="Activity")

# the code generates:

not matching: Activity Activity+ ID Description Status

From here OP should be able to tweak the code to provide the desired outputs and/or set some variables to use for follow-on conditional operations (eg, abort processing if either echo is triggered, disable follow-on processing if either echo is triggered, etc).

4
  • works perfectly. Thank you so much. For some reason "+" value is causing a problem in the if loop, so i replace it with word "PLUS" then compare and in the final mismatch return value i replace it back to "+".
    – Pat
    Oct 28, 2021 at 14:18
  • one last help, last header value is saying not matching though it is matching. may be it is ending with carriage return or some special char. So before if "[[ "${fields[$i]}" != "${headers[$i]}" ]]" i added these 2 lines ${fields[$i]}=echo ${fields[$i]} | xargs echo -n; ${headers[$i]}=echo ${headers[$i]} | xargs echo -n but still not working
    – Pat
    Oct 28, 2021 at 20:09
  • 1
    assuming the visible characters are identical ... try running head -1 file | od -c to see if there are any non-printing characters in the line (eg, \r); also run typeset -p <array_name> and carefully review the contents of each array entry (eg, does the last entry show an ending of \r?); if you are seeing the character \r the easiest solution would be to first run dos2unix <filename> (to remove \r characters) and then proceed with the rest of the code; if you're unable to address the issue at that point I'd suggest opening a new question to address this new issue
    – markp-fuso
    Oct 28, 2021 at 20:16
  • 1
    Thanks, there was \r and \n at the end, replaced and it works as expected.
    – Pat
    Oct 28, 2021 at 21:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.