133

I have the following Python code:

cursor.execute("INSERT INTO table VALUES var1, var2, var3,")

where var1 is an integer, var2 and var3 are strings.

How can I write the variable names without Python including them as part of the query text?

6 Answers 6

157
cursor.execute("INSERT INTO table VALUES (%s, %s, %s)", (var1, var2, var3))

Note that the parameters are passed as a tuple, (a, b, c). If you're passing a single parameter, the tuple needs to end with a comma, (a,).

The database API does proper escaping and quoting of variables. Be careful not to use the string formatting operator (%), because

  1. It does not do any escaping or quoting.
  2. It is prone to uncontrolled string format attacks e.g. SQL injection.
9
  • Interesting, why does it work with the vars separately instead of in an array (var1,var2,var3)?
    – Andomar
    May 23, 2009 at 20:31
  • According to the DB API specs, it looks like it can be either way: python.org/dev/peps/pep-0249 May 23, 2009 at 20:57
  • 11
    @thekashyap Read again carefully. What's insecure is using the string formatting operator %. In fact, I say so in the answer. Feb 10, 2014 at 21:21
  • 3
    Downvoted because answer says not to use % but uses it three times. More explanation would be great.
    – eric
    Dec 8, 2019 at 4:09
  • 10
    @eric the answer says do not use the % operator to format the string. Those % in the string are being used by cursor.execute directly, and since it knows it's generating SQL it can do more to protect you. Aug 7, 2020 at 16:21
89

Different implementations of the Python DB-API are allowed to use different placeholders, so you'll need to find out which one you're using -- it could be (e.g. with MySQLdb):

cursor.execute("INSERT INTO table VALUES (%s, %s, %s)", (var1, var2, var3))

or (e.g. with sqlite3 from the Python standard library):

cursor.execute("INSERT INTO table VALUES (?, ?, ?)", (var1, var2, var3))

or others yet (after VALUES you could have (:1, :2, :3) , or "named styles" (:fee, :fie, :fo) or (%(fee)s, %(fie)s, %(fo)s) where you pass a dict instead of a map as the second argument to execute). Check the paramstyle string constant in the DB API module you're using, and look for paramstyle at http://www.python.org/dev/peps/pep-0249/ to see what all the parameter-passing styles are!

0
68

Many ways. DON'T use the most obvious one (%s with %) in real code, it's open to attacks.

Here copy-paste'd from pydoc of sqlite3:

... beware of using Python’s string operations to assemble queries, as they are vulnerable to SQL injection attacks. For example, an attacker can simply close the single quote and inject OR TRUE to select all rows:

# Never do this -- insecure!
symbol = input()

sql = "SELECT * FROM stocks WHERE symbol = '%s'" % symbol
print(sql)

cur.execute(sql)

More examples if you need:

# Multiple values single statement/execution
c.execute('SELECT * FROM stocks WHERE symbol=? OR symbol=?', ('RHAT', 'MSO'))
print c.fetchall()
c.execute('SELECT * FROM stocks WHERE symbol IN (?, ?)', ('RHAT', 'MSO'))
print c.fetchall()
# This also works, though ones above are better as a habit as it's inline with syntax of executemany().. but your choice.
c.execute('SELECT * FROM stocks WHERE symbol=? OR symbol=?', 'RHAT', 'MSO')
print c.fetchall()
# Insert a single item
c.execute('INSERT INTO stocks VALUES (?,?,?,?,?)', ('2006-03-28', 'BUY', 'IBM', 1000, 45.00))
1
  • 12
    Some of the DB-API implementations actually use %s for their variables -- most notably psycopg2 for PostgreSQL. This is not to be confused (though it easily is) with using %s with the % operator for string replacement. I would be really nice if, for portability, we could just have a defined standard way of specifying SQL parameters for DB-API. Nov 24, 2014 at 18:03
28

http://www.amk.ca/python/writing/DB-API.html

Be careful when you simply append values of variables to your statements: Imagine a user naming himself ';DROP TABLE Users;' -- That's why you need to use SQL escaping, which Python provides for you when you use cursor.execute in a decent manner. Example in the URL is:

cursor.execute("insert into Attendees values (?, ?, ?)", (name, seminar, paid))
2
  • 14
    Actually, it is not SQL escaping. It's variable binding, which is far simpler and more direct. The values are bound into the SQL statement after parsing, making it immune to any injection attack.
    – S.Lott
    May 23, 2009 at 20:34
  • 1
    well, whether it's SQL escaping or variable binding depends on how good or bad your database server / DB-API driver is. I've seen some real-world, widely-deployed production databases that have their DB-API driver just do escaping, rather than keeping data and code out-of-band on the wire. Needless to say, I don't have very much respect for those so-called "databases". Mar 12, 2020 at 18:20
7

The syntax for providing a single value can be confusing for inexperienced Python users.

Given the query

INSERT INTO mytable (fruit) VALUES (%s)

Generally*, the value passed to cursor.execute must be wrapped in an ordered sequence such as a tuple or list even though the value itself is a singleton, so we must provide a single element tuple, like this: (value,).

cursor.execute("""INSERT INTO mytable (fruit) VALUES (%s)""", ('apple',))

Passing a single string

cursor.execute("""INSERT INTO mytable (fruit) VALUES (%s)""", ('apple'))

will result in an error which varies by the DB-API connector, for example

  • psycopg2:

    TypeError: not all arguments converted during string formatting

  • sqlite3

    sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 1, and there are 5 supplied

  • mysql.connector

    mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax;


* The pymysql connector handles a single string parameter without erroring. However it's better to wrap the string in a tuple even if it's a single because

  • you won't need to change the code if you switch connector package
  • you keep a consistent mental model of the query parameters being a sequence of objects rather than a single object.
0

Load your data as auto normalised tables, I suggest using this library that infers schema, types the data and has schema evolution too https://pypi.org/project/dlt/

You can even use this library to do an upsert on your structured data afterwards, here's an example where we use an id in the json to update the sql table generated at destination

data = [{'id': 1, 'name': 'John'}]

# open connection
pipe = dlt.pipeline(destination='postgres',
                    dataset_name='raw_data')

# Upsert/merge: Update old records, insert new
# Capture the outcome in load info
load_info = pipe.run(data,
                      write_disposition="merge",
                      primary_key="id",
                      table_name="users")

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.