8

I am inserting a record into a Amazon Redshift table from Python 2.7 using psycopg2 library and I would like to get back the auto generate primary id for the inserted row.

I have tried the usual ways I can find here or in other websites using google search, eg:

conn=psycopg2.connect(conn_str)
conn.autocommit = True

sql = "INSERT INTO schema.table (col1, col2) VALUES (%s, %s) RETURNING id;"

cur = conn.cursor()
cur.execute(sql,(val1,val2))
id = cur.fetchone()[0]

I receive an error on cur.execute line :

ProgrammingError: syntax error at or near "RETURNING"

Does anybody know how to fix this or accomplish the same thing?

I have to use psycopg2 in my code

5 Answers 5

5

Currently not possible with Redshift, as it doesn't support returning the last insert id via the RETURNING syntax. What you might need to do is use a SELECT MAX(id) FROM schema.table; inside a transaction, which probably not quite what you wanted to hear but appears to be the best you can do with the current state of Redshift.

3
  • 3
    Sadly it's not true. I tried Max(id) approach in my case but redshift doesn't increment identity column value sequentially..i.e. it can leave gaps between ids generated. Newly generated id is guaranteed to be unique but may not necessarily be higher than Max(id) , it can be lower than. E.g. current ids on table can be 1,2,6,7,8 and new record is inserted, it can get assigned the id of 3 which is unique but not more than max(id). May 23, 2017 at 1:20
  • @ShrikantPrabhu, thanks for pointing. This article investigates into the issue. Probably the simplest thing to try is to use IDENTITY (1,1) but I am not sure if it helps. Actually not sure how one can use Redshift at all with such lack of information about newly inserted records..
    – MarSoft
    May 28, 2017 at 23:38
  • 2
    @MarSoft I realise this comment is old, however, using IDENTITY(1,1) doesn't help because of the sharded/distributed nature of REDSHIFT. Each node or slice assigns from a different subset of the IDs in a manner that won't duplicate or collide. This means that the nodes don't need to talk to each other to assign an identity, but it also means "sequential"/"in-order" allocation is not guaranteed.
    – MatBailie
    Oct 9, 2018 at 7:58
1

At this moment, Redshift still doesn't support RETURNING syntax, and I couldn't find here a satisfying answer. So I'm posting a generic solution in case anyone needs it.

The only assumption for this solution is that you know how many records you've just inserted. Assuming x is the numbers of records inserted, you can run this query:

SELECT id 
FROM table 
ORDER BY id DESC
LIMIT {x}

Very important! you have to run this query together with the insert one in the same transaction. Otherwise, it won't work.

0

You can also query for the id in a select if you know how to uniquely find the row without id.

0

You could use following query to get the last inserted id from redshift.

SELECT top 1 id from sampletable where created < Getdate() order by created desc;

where 'id' is the field you are interested in and 'created' is the field having datetime information.

The reason being using the created date time information is, if the table is used for bulk insert there is a chance for having records with id's that are sequential like 1,33,35,56,103, etc..... In the above scenario, using max(id) wont work as expected. because the last inserted id could be any number which was not already inserted earlier.

0

Assuming that you aren't (potentially) inserting duplicates, you could do this with python. I'm assuming that your provided code is in a loop; the specifics of the loop don't matter, but put the # before the loop section before the loop.

# before the loop
# you probably have a list or dict of values that you're inputting. Re-use that if you can; if you can't, create a dict
sample_dict = {}
# end before the loop

/* existing code */
conn=psycopg2.connect(conn_str)
conn.autocommit = True

# use this query instead of your current one (just remove the RETURNING clause)
ins_sql = "INSERT INTO schema.table (col1, col2) VALUES (%s, %s);"

# this query will get the id of those last added values
sel_sql = "SELECT id FROM schema.table WHERE col1 = %s AND col2 = %s ORDER BY id DESC LIMIT 1;"

cur = conn.cursor()
cur.execute(ins_sql,(val1,val2))

# new stuff here!
cur.execute(sel_sql,(val1,val2))
id = cur.fetchone()[0]
sample_dict[val1 + '|' + val2] = id

Caveats:

  1. I haven't tested this exact script
  2. This will probably not work if you're inserting duplicate values
  3. You aren't sanitizing your input or using prepared statements (Not 100% sure that's an option, TBH)
  4. You know more about your database and code structure than I do; leverage that knowledge
  5. This is not the most efficient solution, but it should work for what you want.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.