I am importing data to QuestDB with the HTTP method in Python.
The table (test_data) has the following properties:
- 'name': 'time', 'size': 8, 'type': 'TIMESTAMP'
- 'name': 'open', 'size': 8, 'type': 'DOUBLE'
- 'name': 'high', 'size': 8, 'type': 'DOUBLE'
- 'name': 'low', 'size': 8, 'type': 'DOUBLE'
- 'name': 'close', 'size': 8, 'type': 'DOUBLE'
- 'name': 'volume', 'size': 8, 'type': 'DOUBLE'
- 'name': 'ts', 'size': 8, 'type': 'TIMESTAMP'
Note: the 'time' column is the designated timestamp
The imported data is sourced from a pandas dataframe. The dataframe has the same headers and the 'time' column is the index and the 'ts' column is the timestamp from when the data was acquired. The code shown below regarding the import function.
def to_csv_str(table):
output = io.StringIO()
csv.writer(output, dialect="excel").writerows(table)
return output.getvalue().encode("utf-8")
def write_to_table(df, table="test_data"):
table_name = table
table = [[df.index.name] + df.columns.tolist()] +df.reset_index().values.tolist()
table_csv = to_csv_str(table)
schema = json.dumps([])
response = requests.post(
"http://localhost:9000/imp",
params={"fmt": "json"},
files={"schema": schema, "data": (table_name, table_csv)},
).json()
pprint.pprint(response)
The import executes successfully the first time. If I was to rerun the import for the same data (all values are the same except the 'ts' column for when the data was acquired), then one additional row will be appended with all of the same values but the 'ts' column. How can I have the 'time' column be defined in such a way that it is forced to be unique and any import with a row that has a duplicate 'time' value will be omitted?
Example screenshots for a 6 row import below: