0

We have a Django project with the following problem:

User objects are stored in the auth_user table

These have a One-to-One relation to the Profile object, this object is stored in the yuza_profile table. Its relation to the user is maintained through the user_id field

However, years ago thousands of Profile objects were deleted without removing the linked User objects.

These users without profiles are causing problems in our system and we would like to remove them.

I created the following SQL query to perform this task

DELETE FROM auth_user WHERE id NOT IN (SELECT user_id FROM yuza_profile); 

Since the Profile objects have been deleted I cannot scan for null values on *_id fields - which would be my first move here.

Instead I am looping over two tables that both contain more than 60.000 rows which leads to a slow and inefficient query with more than 6 billion operations (and server timeouts)

Is there a way to speed up this query? I am aware that my query is very inefficient but I do not know of a way to improve it and any help would be greatly appreciated.

EDIT: As requested I've included the schemas below:

user_profile

id - integer
user_id - integer
gender - varchar(2)
birth_date - date
address - varchar(255)
city - varchar(255)
phone_number - varchar(10)
avatar - varchar(255)
---------------------
user_profile_pkey - (id)
user_profile_user_id_key - (user_id)
user_profile_user_id_fkey - (user_id) -> auth_user(id)
user_profile_pkey - (id) UNIQUE
user_profile_user_id_key - (user_id) UNIQUE
auth_user

id - integer
username - varchar(150)
first_name - varchar(30)
last_name - varchar(30)
email - varchar(75)
password - varchar(128)
is_staff - boolean
is_active - boolean
is_superuser - boolean
last_login - timestamp with time zone
date_joined - timestamp with time zone
---------------
auth_user_pkey - (id)
auth_user_username_key - (username)
auth_user_pkey - (id) UNIQUE
auth_user_username_key - (username) UNIQUE

I should note that I've started to wonder if the IDE (PycharmPro) I am using might be a factor - when testing my queries with the SELECT statement the IDE displays queryresults as pages with 500 results per page. The initial query indeed took less than a second - but pressing the 'go to last page' button took more than 2 minutes (for just 16000 results)

7
  • 3
    60K rows is no data at all. That amount probably fits in the CPU's cache. Post the table schemas (including any indexes) and the execution plans. Most likely yuza_profile.user_id and auth_user.id aren't covered by indexes. Lots of data is billions of rows and TBs of data Feb 21, 2019 at 10:41
  • 1
    As for 6Bn operations, that would only happen if there were no indexes and the server had to run a full table scan of the inner table for every row in the outer table. Another possibility is that id and user_id have different types or different collations which would require a conversion from one type to the other before comparison. In that case, any indexes created using the original type are useless Feb 21, 2019 at 10:43
  • 1
    very often not exists performs better than not in. Did you try to rewrite that query? DELETE FROM auth_user WHERE NOT EXISTS (SELECT * FROM yuza_profile p WHERE p.user_id = auth_user.id);
    – user330315
    Feb 21, 2019 at 10:50
  • I wonder if the IDE (PycharmPro) I am using might be a factor - its show database queryresults as pages with 500 results per page. THe initial query indeed took less than a second - but pressing the 'go to last page' button took 2 min.
    – Jasper
    Feb 21, 2019 at 10:56
  • @a_horse_with_no_name - I tried your solution and loading the 'last page' took less than a second, thanks!
    – Jasper
    Feb 21, 2019 at 10:57

2 Answers 2

1

I strongly advise you to use NOT EXISTS rather than NOT IN. It treats NULL values more intuitively. (NOT IN will filter out all rows if any value in the subquery is NULL.)

So, write the query as:

DELETE au FROM auth_user au
    WHERE NOT EXISTS (SELECT 1
                      FROM yuza_profile vp 
                      WHERE vp.user_id = au.id
                     ); 

This query can take advantage of an index on yuza_profile(user_id):

CREATE INDEX idx_yuza_profile_user_id ON yuza_profile(user_id);
1

I'm not completely sure about your approach, I'm assuming you took a cursor-based strategy, but can't you just:

DELETE FROM auth_user WHERE id NOT IN (SELECT user_id FROM yuza_profile);

If you have the proper indices, this DML instruction will not require that much operations.

2
  • 1
    That's exactly the statement that is show in the question
    – user330315
    Feb 21, 2019 at 10:50
  • I edited it! I initially showed a SELECT statement that I left in by error
    – Jasper
    Feb 21, 2019 at 10:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.