1

Good day all, Prosperous new year to all.

I have a file using the pipe character as field delimiter. The aim is to prepend a certain field, in this case field 4, with either a C or R depending on the values of the fields 2 and 3.

File in question:

    11975528|0202|INVOICES|  211311|2020/12/11|00893139P|         |           34.90 |             .00 |            5.24 |           40.14 |MRS L JAMES                                                 |Pr#:558392 M/Aid:PR
    11975529|0202|INVOICES|  211312|2020/12/11|00893140P|         |           34.90 |             .00 |            5.24 |           40.14 |MISS M KHOZA                                                |Pr#:558393 M/Aid:PR
    11975530|0202|INVOICES|  211311|2020/12/11|00893142P|         |           34.90 |             .00 |            5.24 |           40.14 |MRS L JAMES                                                 |Pr#:558392 M/Aid:PR
    11975531|0202|C/NOTES |  211311|2020/12/11|00893142P|20/12/11 |           34.90-|             .00 |            5.24-|           40.14-|MRS L JAMES                                                 |Pr#:558392 M/Aid:PR
    11975532|0202|C/NOTES |  211310|2020/12/11|00893136P|20/12/11 |          115.90-|             .00 |           17.39-|          133.29-|MR S ERASMUS                                                |Pr#:558391 M/Aid:PR

I have passed passed the file into a while read loop, using awk to separate the fields and sed to replace the values:

if [ -f $datext ];                                  #Check if the extract file exits
then
    cat $datext | while IFS= read line
    do
    BCHNO=`echo $line | awk -F"|" '{print $2}'`     #Get BRANCH Number
    TYPE=`echo $line | awk -F"|" '{print $3}'`      #Get Transation type e.g. invoice/credit note
    DEBNO=`echo $line | awk -F"|" '{print $4}'`     #Get debtor account number
    
    if [ $BCHNO -eq 0202 ] && [ "$TYPE" == "INVOICES" ];
        then
            sed -i "s/$DEBNO/C$DEBNO/" $datext
    elif [ $BCHNO -eq 0202 ] && [ "$TYPE" == "C/NOTES" ];
        then
            sed -i "s/$DEBNO/R$DEBNO/" $datext
    fi
    done

When running the above, where there are multiple transactions for the same debtor it keeps prepending the C:

 11975534|0202|INVOICES| CCCCCC 911202|2020/12/11|00893144P|         |    21.17 |             .00 |           3.18 |           24.35 |LEVY CONTROL                                             |Pr#:558384 M/Aid:PR

And the C/NOTES are ignored and also prepended with a C:

11975540|0202|C/NOTES | C 558379|2020/12/11|00893149P|20/12/11 |          173.52-|             .00 |           26.03-|          199.55-|MR D MCCARTHY                                        |Pr#:558379 M/Aid:PR

I have resorted to grep the INVOICES and C/NOTES strings to separate files then doing the substitution on each file which works for the C/NOTES but still have the same issue with INVOICES where it keeps prepending the Cs where there are multiple transactions for the same debtor.

Am I on the correct path in using the combinations, just not using them correctly, or is there a better way to achieve this?

Any guidance would be greatly appreciated.

2
  • 1
    Hi and welcome to Stack Overflow. You might be interested in reading Why is using a shell loop to process text considered bad practice?
    – kvantour
    Commented Jan 5, 2021 at 13:27
  • 1
    Because you make use of awk to extract entries from a vertical-bar delimited file, it will include the spaces (eg. "C/NOTES " != "C/NOTES"). Also, every time you find a match, you are editing the entire file and not just the line you want to edit.
    – kvantour
    Commented Jan 5, 2021 at 13:29

4 Answers 4

2

In this case a sed solution can work without additional loops or too much magic:

if [ -f $datext ];                                  #Check if the extract file exits
then
   sed -i 's/^[^|]*|0202|INVOICES|/&C/;s/^[^|]*|0202|C\/NOTES |/&R/' "${datext}"
fi

Explanation: ^[^|]*|: Starting at the beginning of the line, match everything until the first pipe.
&C: Replace the matched string with the match (the &) followed by a C.
C\/N: Escape the forward slash with a backslash. An alternative is using s#...#...#.

When you want to avoid repeating inserts when you repeat the command, add a space in the match:

sed -i 's/^[^|]*|0202|INVOICES| /&C /;s/^[^|]*|0202|C\/NOTES | /&R /' "${datext}"
2
  • This is very nice and short. I would suggest incorporating the blanks in the fields. I would not be surprised if the user has files with different field-widths
    – kvantour
    Commented Jan 5, 2021 at 22:37
  • 1
    When [:blank:] or other fixes are needed, the command gets dirty and you should switch to awk.
    – Walter A
    Commented Jan 5, 2021 at 22:43
1

Below you find a quick fix of your script. Here are a couple of issues I found:

  • you extract the values of BCHNO, TYPE and DEBNO using awk of the form: awk -F"|" '{print $n}'. Because you use a vertical-bar as a delimiter, the field will include spaces. That is why comparisons of the form [ "$TYPE" == "C/NOTES" ] might fail because TYPE might contain extra spaces at the end.
  • In your conditional statements, you perform sed on the entire file and not just the line of interest, That is why you encounter multiple substitutions on the same line when you have multiple matches (i.e. the multiple C values). You could fix this by adding a simple line counter and instruct sed to only update that one line.
  • Use [ string1 = string2 ] and not [ string1 == string2 ], this is POSIX compliant. Eventhough bash allows the use of ==, it might be that the use of #!/bin/sh could make this fail. (cfr comment of Charles Duffy)
  • Using a cat file to send input into a while loop places the loop in a subshell, which means any state changes you make (changing variables, cd, opening, and closing files, etc.) will be lost when the loop finishes. To avoid that, you may use redirection. So don't do cat file | while read line; do ...; done, but rather while read line; do ...; done <file. See BashFAQ 001

Below you find a quick update of your script.

if [ -f $datext ];                                  #Check if the extract file exits
then
    while IFS= read -r line
    do
    # read variables from $line
    IFS="|" read -r _ BCHNO TYPE DEBNO _ <<<"$line"
    # remove pre and postpend blanks
    BCHNO="${BCHNO//^[:blank:]*}"; BCHNO="${BCHNO//[:blank:]*$}"
    TYPE="${TYPE//^[:blank:]*}";"${TYPE//[:blank:]*$}"
    DEBNO="${DEBNO//^[:blank:]*}";"${DEBNO//[:blank:]*$}"
    if [ $BCHNO -eq 0202 ] && [ "$TYPE" = "INVOICES" ]; then
        line="${line/$DEBNO/C$DEBNO}"
    elif [ $BCHNO -eq 0202 ] && [ "$TYPE" = "C/NOTES" ]; then
        line="${line/$DEBNO/R$DEBNO}"
    fi
    echo "$line" > "${datext}.new"
    done < "$datext"
    mv "${datext}.new" "${datext}"
fi

While the above does exactly what you request it to do, it is written in pure bash. It is strongly recommended to not use bash for data-manipulation of this form. Use tools that have been designed for it. Below you find a simple awk program that does exactly the same.

if [ -f "$datext" ]; then
   awk 'BEGIN{FS=OFS="|"}
        {BCHNO=$2; TYPE=$3; DEBNO=$4}
        (BCHNO=="0202") && (TYPE ~ " *INVOICES *") { $4 = "C"$4 }
        (BCHNO=="0202") && (TYPE ~ " *C/NOTES *")  { $4 = "R"$4 }
        1' "$datext" > "${datext}.new"
   mv "${datext}.new" "${datext}"
fi
5
  • And I just got properly schooled, thank you for the detailed information and the time taken to reply, it is really appreciated. I was a bit hesitant about going fully awk as I know very little at this stage but it would appear that it would be the best to use going forward until I'm done with learning Python
    – NovaOrsa
    Commented Jan 5, 2021 at 14:19
  • I think you need the remove the quotes in the awk line {BCHNO=$2; TYPE=$3; DEBNO=$4}.
    – Walter A
    Commented Jan 5, 2021 at 21:55
  • 1
    I'd suggest changing the ==s to =s inside [ ] for POSIX compliance. Granted, if the shell really is bash == will be available, but better to develop finger memory for practices that'll still work with sh when there's no concrete benefit to be gained by diverging. Commented Jan 5, 2021 at 22:07
  • The field DEPNO is not used. You can consider $2=="0202" {print; next;} and replace TYPE with $3.
    – Walter A
    Commented Jan 5, 2021 at 22:46
  • you can indeed code-golf it, but it loose readability of the code
    – kvantour
    Commented Jan 5, 2021 at 22:57
0

Since you are already using awk within the loop, it would make more sense to use awk as a complete solution:

awk -F \| '$2=="0202" && $3 == "INVOICES" { # Process where the 4th pipe delimited field is equal to 0202 and the third field is equal to INVOICES                   
               OFS="|";
               $4="C"$4 # Prefix the fourth field with C
           } 
           $2=="0202" && $3 == "C/NOTES " { # Process where the 4th pipe delimited field is equal to 0202 and the third field is equal to C/NOTES
               OFS="|";
               $4="R"$4 # Prefix the fourth field with C
           }1' $datext # Use short hand 1 to print the lines (amended or otherwise)
0

This might work for you (GNU sed):

sed -E '1{x;s#^#INVOICESC/NOTESR#;x};G;s#((.+)\s*\|)\s(.*\n.*\2)(.)#\1\4\3#;P;d' file

Create a lookup table for invoices and notes.

Append the lookup to each line and pattern match replacing the first space of the following field by the lookup.

N.B. Extend or shorten the lookup table if maintenance is required.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.