32

In a django view, I need to append string data to the end of an existing text column in my database. So, for example, say I have a table named "ATable", and it has a field named "aField". I'd like to be able to append a string to the end of "aField" in a race-condition-free way. Initially, I had this:

tableEntry = ATable.objects.get(id=100)
tableEntry.aField += aStringVar
tableEntry.save()

The problem is that if this is being executed concurrently, both can get the same "tableEntry", then they each independently update, and the last one to "save" wins, losing the data appended by the other.

I looked into this a bit and found this, which I hoped would work, using an F expression:

ATable.objects.filter(id=100).update(aField=F('aField') + aStringVar)

The problem here, is I get an SQL error, saying:

operator does not exist: text + unknown
HINT:  No operator matches the given name and argument type(s). You might need to add explicit type casts.

Tried changing to "str(aStringVar)" even though its already a string - no luck.. I found a couple django bug reports complaining about similar issues, but I didn't see a fix or a workaround. Is there some way I can cast aStringVar such that it can be appended to the text of the F expression? BTW - also tried "str(F('aField')) + aStringVar" but that converted the result of the F expression to the string "(DEFAULT: )".

5
  • Can you see the sqlquery Django tries to execute?
    – Willian
    Nov 15, 2011 at 21:58
  • 1
    possible duplicate of Can I use Django F() objects with string concatenation?
    – Alasdair
    Nov 15, 2011 at 21:59
  • I think Daniel's explanation on the other question is pretty definitive. As an alternative, You can write custom sql to perform your updates.
    – Alasdair
    Nov 15, 2011 at 22:02
  • 1
    I searched SO and the web in general and never came across that answer - thanks for pointing me to it.. My current workaround is to use raw SQL with a cursor in django, but its pretty ugly and ties us to postgres which I'd rather avoid.
    – daroo
    Nov 15, 2011 at 22:10
  • So I did the raw thing, and that got me through to now, but I'm using multiprocessing and have two threads that are apparently fighting over the database connection?? I get an error saying "SET TRANSACTION ISOLATION LEVEL must be set before any query". I'm reaching my database knowledge limits here. One question here on SO seemed to indicate I can't do this, but that seems hard to believe, since I expect via a web connection, its entirely possible several threads are all touching the DB at the same time.
    – daroo
    Nov 22, 2011 at 21:04

5 Answers 5

25

You can use the Concat db function.

from django.db.models import Value
from django.db.models.functions import Concat

ATable.objects.filter(id=100).update(some_field=Concat('some_field', Value('more string')))

In my case, I am adding a suffix for facebook avatars URIs like this:

FACEBOOK_URI = 'graph.facebook.com'
FACEBOOK_LARGE = '?type=large'
# ...
users = User.objects.filter(Q(avatar_uri__icontains=FACEBOOK_URI) & ~Q(avatar_uri__icontains=FACEBOOK_LARGE))
users.update(avatar_uri=Concat('avatar_uri', Value(FACEBOOK_LARGE)))

and I get SQL like this (Django 1.9):

UPDATE `user_user` SET `avatar_uri` = CONCAT(COALESCE(`user_user`.`avatar_uri`, ''), COALESCE('?type=large', ''))
WHERE (`user_user`.`avatar_uri` LIKE '%graph.facebook.com%' AND NOT (`user_user`.`avatar_uri` LIKE '%?type=large%' AND `user_user`.`avatar_uri` IS NOT NULL))

The result is all image URIs were changed from http://graph.facebook.com/<fb user id>/picture to http://graph.facebook.com/<fb user id>/picture?type=large

1
  • Not ideally portable: "SQLite, however, does not support the CONCAT() function". Sometimes sqlite is handy eg for testing. May 12, 2020 at 21:28
23

You can override F object in Django with one simple change:

class CF(F):
    ADD = '||'

Then just use CF in place of F. It will place "||" instead of "+" when generating SQL. For example, the query:

User.objects.filter(pk=100).update(email=CF('username') + '@gmail.com')

will generate the SQL:

UPDATE "auth_user" SET "email" = "auth_user"."username" || '@gmail.com'
WHERE "auth_user"."id" = 100 
3
  • 1
    Just to complete this answer (worked for me! thanks!) you should also set sql_mode='PIPES_AS_CONCAT'; if you do not do that, my MySQL considers "||" as logic OR Mar 13, 2015 at 10:57
  • 1
    Good point! I was testing on PostgreSQL. In fact, || is not a standard SQL operator. The standard way is to use CONCAT() function, however it does not fit into this solution... Mar 26, 2015 at 5:14
  • See answer below for portable solution using Concat!
    – merwok
    Aug 1, 2017 at 15:47
3

And if you get this running, it isn't thread safe. While your update is running, some other process can update a model not knowing the data in the database is updated.

You have too acquire a lock, but don't forget this senario:

  1. Django: m = Model.objects.all()[10]
  2. Django: m.field = field
  3. Django: a progress which takes a while (time.sleep(100))
  4. DB: Lock table
  5. DB: Update field
  6. DD: Unlock table
  7. Django: the slow process is finished
  8. Django: m.save()

Now the field update became undone by the model instance in Django (Ghost write)

1
  • Very interesting point (but not really related to question). Do you have some resource discussing how to handle this kind of situation with Django?
    – lajarre
    Mar 17, 2015 at 13:44
3

You can achieve this functionality with Django's select_for_update() operator: https://docs.djangoproject.com/en/dev/ref/models/querysets/#select-for-update

Something like this:

obj = ATable.objects.select_for_update().get(id=100)
obj.aField = obj.aField + aStringVar
obj.save()

The table row will be locked when you call .select_for_update().get(), and the lock will be released when you call .save(), allowing you to perform the operation atomically.

2

seems you can't do this. however, what you are trying to do could be solved using transactions

(looks like you are using postgres, so if you want to do it in one query and use raw sql as suggested, || is the concatenation operator you want)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.