0

I wish to pipe aws cli output which appears on my screen as text output from a powershell session into a text file in csv format.

I have researched the Export-CSV cmdlet from articles such as the below:

https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/export-csv?view=powershell-7.1

I cannot see how to use this to help me with my goal. From my testing, it only seems to work with specific windows programs, not general text output.

An article on this site shows how you can achieve my goal with unix commands, by replacing spaces with commas.

Output AWS CLI command with filters to CSV without jq

The answer with unix is to use sed at the end of the command like so:

aws rds describe-db-instance-automated-backups --query 'DBInstanceAutomatedBackups[*].{ARN:DBInstanceArn,EarliestTime:RestoreWindow.EarliestTime,LatestTime:RestoreWindow.LatestTime}' --output text | sed -E 's/\s+/,/g'

Export-csv` appears to not be able to do this.

Does anyone know how I might replicate what sed is doing here with powershell?

Here is an example of the output that I would like in csv format:

arn:aws:rds:ap-southwest-2:9711387875370:db:catflow--prod     2019-03-03T09:54:29.402Z        2019-03-05T01:25:53Z
arn:aws:rds:ap-southwest-2:9711387875370:db:xyz-prod-rds-golf    2019-03-01T09:04:31.477Z        2019-03-05T01:28:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-stardb   2019-02-01T09:07:30.648Z        2019-03-05T01:27
:20Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-domaindb    2019-02-02T09:04:30.771Z        2019-03-05T01:28
:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-ctz-prod-rds-datavault   2019-02-26T14:14:30.254Z        2019-03-05T01:29
:13Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-gcp-prod-rds-datavault   2019-02-01T14:05:40.456Z        2019-03-05T01:31
:05Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-conformed-datavault-prod    2019-02-02T14:06:26.050Z        2019-03-
05T01:27:02Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-dqm-datavault-prod  2019-02-01T14:12:05.286Z        2019-03-05T01:26
:53Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-prod-dgc-cde-lineage 2019-03-02T09:54:29.053Z        2019-03-05T01:29
:11Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-rec-prod     2019-02-02T22:09:00.673Z        2019-03-05T01:29:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-serve-prod       2019-03-02T09:54:20.729Z        2019-03-05T01:30:21Z
4
  • Can you give us an example of what the text output looks like?
    – Theo
    Mar 4, 2021 at 11:15
  • I have done so Theo thanks.
    – Scouse_Bob
    Mar 5, 2021 at 1:37
  • I see, but that is only the desired output. What I really meant was the raw output from the aws rds ... command you need to converft to CSV
    – Theo
    Mar 5, 2021 at 8:53
  • No mate that IS the raw output... that's what comes on screen following the execution of the command
    – Scouse_Bob
    Mar 5, 2021 at 10:47

2 Answers 2

2

It's possible that you are working with a tab delimited text file, with no headers. The tab separator can look like multiple spaces when it is displayed on your screen.

If this is the case, If so, you can actually read this file with import-csv, but you have to use the -header parameter to supply your own field names, and the -delimiter character to use tab as the delimiter. The tab character has to be specified using the backtick escape mechanism.

For details, see the accepted answer to this question.

If you have control over your data feed, there is an alternative. The aws cli interface has an option to format the output in JSON format. That format will be much easier to import into Powershell in a form you can use.

Edit:

The following script uses the mockup provided by Theo, except that the multiple spaces have been replaced by a tab character. It uses ConvertFrom-Csv rather than Import-Csv, but it's the same idea:

$awsReturn = @"
arn:aws:rds:ap-southwest-2:9711387875370:db:catflow--prod   2019-03-03T09:54:29.402Z    2019-03-05T01:25:53Z
arn:aws:rds:ap-southwest-2:9711387875370:db:xyz-prod-rds-golf   2019-03-01T09:04:31.477Z    2019-03-05T01:28:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-stardb    2019-02-01T09:07:30.648Z    2019-03-05T01:27:20Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-domaindb  2019-02-02T09:04:30.771Z    2019-03-05T01:28:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-ctz-prod-rds-datavault 2019-02-26T14:14:30.254Z    2019-03-05T01:29:13Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-gcp-prod-rds-datavault 2019-02-01T14:05:40.456Z    2019-03-05T01:31:05Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-conformed-datavault-prod   2019-02-02T14:06:26.050Z    2019-03-05T01:27:02Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-dqm-datavault-prod     2019-02-01T14:12:05.286Z    2019-03-05T01:26:53Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-prod-dgc-cde-lineage   2019-03-02T09:54:29.053Z    2019-03-05T01:29:11Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-rec-prod   2019-02-02T22:09:00.673Z    2019-03-05T01:29:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-serve-prod 2019-03-02T09:54:20.729Z    2019-03-05T01:30:21Z
"@


$myarray = $awsreturn | ConvertFrom-Csv -header "Prod","DateStart","DateEnd" -delimiter "`t"

$myarray | Format-Table

$myarray | gm

When I ran it in my environment, it produced the following:

Prod                                                                      DateStart                DateEnd             
----                                                                      ---------                -------             
arn:aws:rds:ap-southwest-2:9711387875370:db:catflow--prod                 2019-03-03T09:54:29.402Z 2019-03-05T01:25:53Z
arn:aws:rds:ap-southwest-2:9711387875370:db:xyz-prod-rds-golf             2019-03-01T09:04:31.477Z 2019-03-05T01:28:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-stardb          2019-02-01T09:07:30.648Z 2019-03-05T01:27:20Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-domaindb        2019-02-02T09:04:30.771Z 2019-03-05T01:28:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-ctz-prod-rds-datavault       2019-02-26T14:14:30.254Z 2019-03-05T01:29:13Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-gcp-prod-rds-datavault       2019-02-01T14:05:40.456Z 2019-03-05T01:31:05Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-conformed-datavault-prod 2019-02-02T14:06:26.050Z 2019-03-05T01:27:02Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-dqm-datavault-prod       2019-02-01T14:12:05.286Z 2019-03-05T01:26:53Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-prod-dgc-cde-lineage     2019-03-02T09:54:29.053Z 2019-03-05T01:29:11Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-rec-prod                 2019-02-02T22:09:00.673Z 2019-03-05T01:29:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-serve-prod                   2019-03-02T09:54:20.729Z 2019-03-05T01:30:21Z




   TypeName: System.Management.Automation.PSCustomObject

Name        MemberType   Definition                                                           
----        ----------   ----------                                                           
Equals      Method       bool Equals(System.Object obj)                                       
GetHashCode Method       int GetHashCode()                                                    
GetType     Method       type GetType()                                                       
ToString    Method       string ToString()                                                    
DateEnd     NoteProperty string DateEnd=2019-03-05T01:25:53Z                                  
DateStart   NoteProperty string DateStart=2019-03-03T09:54:29.402Z                            
Prod        NoteProperty string Prod=arn:aws:rds:ap-southwest-2:9711387875370:db:catflow--prod
1
  • It may be tab delimited, but I don't think that should matter. The \s+ regex will match whitespace, not spaces, so tabs are included. Also, I agree JSON output might be better, but only if he's using Powershell v6+. Powershell v5.1 and below uses MS's JSON parser that has so many MS-specific syntaxes especially with datetimes (that's what made Json.NET so ubiquitous under pre-Core .Net). After getting burned I wouldn't trust it to serialize data from a foreign system until v6+.
    – Bacon Bits
    Mar 17, 2021 at 13:17
1

Lets assume the data returned looks like this mockup (in the question it is strangely formatted):

$awsReturn = @"
arn:aws:rds:ap-southwest-2:9711387875370:db:catflow--prod     2019-03-03T09:54:29.402Z        2019-03-05T01:25:53Z
arn:aws:rds:ap-southwest-2:9711387875370:db:xyz-prod-rds-golf    2019-03-01T09:04:31.477Z        2019-03-05T01:28:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-stardb   2019-02-01T09:07:30.648Z        2019-03-05T01:27:20Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-domaindb    2019-02-02T09:04:30.771Z        2019-03-05T01:28:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-ctz-prod-rds-datavault   2019-02-26T14:14:30.254Z        2019-03-05T01:29:13Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-gcp-prod-rds-datavault   2019-02-01T14:05:40.456Z        2019-03-05T01:31:05Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-conformed-datavault-prod    2019-02-02T14:06:26.050Z        2019-03-05T01:27:02Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-dqm-datavault-prod  2019-02-01T14:12:05.286Z        2019-03-05T01:26:53Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-prod-dgc-cde-lineage 2019-03-02T09:54:29.053Z        2019-03-05T01:29:11Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-rec-prod     2019-02-02T22:09:00.673Z        2019-03-05T01:29:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-serve-prod       2019-03-02T09:54:20.729Z        2019-03-05T01:30:21Z
"@

Then, you can do this:

# Since I don't know if that is one single string or a string array:
if ($awsReturn -isnot [array]) { $awsReturn = $awsReturn -split '\r?\n' }

# write it to csv file
$awsReturn -replace '\s+', ',' | Set-Content -Path 'WhereEver.csv' -PassThru  # PassThru also displays on screen

to get a file that can serve as CSV (although it has no headers or quoted fields)


If you want to use Export-CSV to get a csv file with headers and quoted fields, you need to split the lines and output objects.

Something like this:

# Since I don't know if that is one single string or a string array:
if ($awsReturn -isnot [array]) { $awsReturn = $awsReturn -split '\r?\n' }

# write it to csv file (without headers or quotes values)
$awsReturn | ForEach-Object {
    $data = $_ -split '\s+'  # in this case we know we have 3 fields
    [PsCustomObject]@{
        Prod      = $data[0]
        DateStart = $data[1]
        DateEnd   = $data[2]
    }
} | Export-Csv -Path 'WhereEver.csv' -NoTypeInformation

The WhereEver.csv file will then look like this:

"Prod","DateStart","DateEnd"
"arn:aws:rds:ap-southwest-2:9711387875370:db:catflow--prod","2019-03-03T09:54:29.402Z","2019-03-05T01:25:53Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:xyz-prod-rds-golf","2019-03-01T09:04:31.477Z","2019-03-05T01:28:40Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-stardb","2019-02-01T09:07:30.648Z","2019-03-05T01:27:20Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-domaindb","2019-02-02T09:04:30.771Z","2019-03-05T01:28:40Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:-ctz-prod-rds-datavault","2019-02-26T14:14:30.254Z","2019-03-05T01:29:13Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:-gcp-prod-rds-datavault","2019-02-01T14:05:40.456Z","2019-03-05T01:31:05Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:prod-conformed-datavault-prod","2019-02-02T14:06:26.050Z","2019-03-05T01:27:02Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:prod-dqm-datavault-prod","2019-02-01T14:12:05.286Z","2019-03-05T01:26:53Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:prod-prod-dgc-cde-lineage","2019-03-02T09:54:29.053Z","2019-03-05T01:29:11Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:prod-rec-prod","2019-02-02T22:09:00.673Z","2019-03-05T01:29:40Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:-serve-prod","2019-03-02T09:54:20.729Z","2019-03-05T01:30:21Z"

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.