0

I am importing data to QuestDB with the HTTP method in Python.

The table (test_data) has the following properties:

  • 'name': 'time', 'size': 8, 'type': 'TIMESTAMP'
  • 'name': 'open', 'size': 8, 'type': 'DOUBLE'
  • 'name': 'high', 'size': 8, 'type': 'DOUBLE'
  • 'name': 'low', 'size': 8, 'type': 'DOUBLE'
  • 'name': 'close', 'size': 8, 'type': 'DOUBLE'
  • 'name': 'volume', 'size': 8, 'type': 'DOUBLE'
  • 'name': 'ts', 'size': 8, 'type': 'TIMESTAMP'

Note: the 'time' column is the designated timestamp

The imported data is sourced from a pandas dataframe. The dataframe has the same headers and the 'time' column is the index and the 'ts' column is the timestamp from when the data was acquired. The code shown below regarding the import function.

def to_csv_str(table):
    output = io.StringIO()
    csv.writer(output, dialect="excel").writerows(table)
    return output.getvalue().encode("utf-8")


def write_to_table(df, table="test_data"):
    table_name = table
    table = [[df.index.name] + df.columns.tolist()] +df.reset_index().values.tolist()
    table_csv = to_csv_str(table)
    schema = json.dumps([])
    response = requests.post(
        "http://localhost:9000/imp",
        params={"fmt": "json"},
        files={"schema": schema, "data": (table_name, table_csv)},
    ).json()
    pprint.pprint(response)

The import executes successfully the first time. If I was to rerun the import for the same data (all values are the same except the 'ts' column for when the data was acquired), then one additional row will be appended with all of the same values but the 'ts' column. How can I have the 'time' column be defined in such a way that it is forced to be unique and any import with a row that has a duplicate 'time' value will be omitted?

Example screenshots for a 6 row import below:

  1. Initial import with all rows successfull image 1
  2. Reissued import with only 5 errors (expected 6) image 2
  3. Table data from the web console image 3

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.