0

I have many CSVs with raw data that I want to put into one master table. I have imported all the CSVs as temporary tables. Then, I have to use substrings to pull out specific, corresponding info from the temp tables. This works with the state column (varchar) when the value is atomic.

How can I do this for the attribute_list column (json) when the value is not atomic and I can't use string literals?

INSERT INTO master_table (data_file_name, state, attribute_list) 

SELECT
'example_name', substring(data, 1, 2),
'{"percent_green_cover_august" : substring(data, 61, 2), 
"percent_green_cover_september" : substring(data, 63, 2)}'

FROM temp_table; 

EDIT: The problem is that there is about 200 temporary tables that I have uploaded from CSVs. They are all different. They contain one field, data (varchar), that is a series of spaces and numbers such as:

11 1134 4446 48685 989
15 4 4 4 78 90 09 
01932938     838490 111
11 1

I have an excel file that contains rows representing each CSV and the column headers correspond to what the values represent. I then use Python to generate the insert into statements. Originally, each column header would be its own field in the database, but that would generate over 2000 unique columns.

2
  • One option is to store the data as-is from the CSV into a temporary table and then enter it into your main table.
    – zedfoxus
    May 25, 2020 at 17:36
  • Can you give us a sample of what's in data? Also, consider using the JSON functions and operators rather than building it by hand.
    – Schwern
    May 25, 2020 at 18:06

1 Answer 1

0

I would recommend scrapping temp_table. There isn't much value in having the raw CSV text in Postgres.

If your CSV files have a fixed set of fields, create a table with all the same columns as the CSV. Then use copy to import the CSV.

COPY your_table(your, columns)
FROM '/your/csv/file.csv' DELIMITER ',' CSV HEADER;

The path is on the server. If your database is not local, you'll have to copy from STDIN or use /copy from psql. Most Postgres database interfaces provide a copy method to make copying a local file easy.


If your CSV files do not have a fixed set of fields and you need JSON, there's no benefit to doing it in Postgres. It's simpler to do the processing using your favorite programming language. Parse the CSV file, turn the fields into a JSON object, then insert the finished product.

Though given the hint at the nature of your data, it would be easier to work with if you built a more traditional set of tables to store it. For example, instead of percent_green_cover_august, have a table to store attributes for a time range.

create table attributes_by_period (
  whatever_this_is_part_of_id bigint not null references whatever_this_is_part_of(id),
  period tzrange not null,
  percent_green_cover integer check(0 <= percent && percent <= 100)
);

-- Thing 42 had 15% green coverage in August.
-- [ is inclusive, ) is exclusive
insert into attributes_by_period
  (whatever_this_is_part_of_id, period, percent_green_cover)
  values( 42, '[2019-08-01 00:00, 2019-09-01 00:00)', 15 )

When you do build JSON, do not build it by hand. Use the JSON functions and operators such as jsonb_build_object. It's easier, and it will handle all the quoting and escaping for you.

jsonb_build_object(
  'percent_green_cover_august', substring(data, 61, 2), 
  'percent_green_cover_september' : substring(data, 63, 2)
)

See you, Space Cowboy.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.