I am working on a task where I need to loop through a bunch of IDs and fetch relevant data from the PgSQL server 12.7. I have added a for loop in python and all the relevant code to connect to the database using psycopg2 version 2.9.1
connectStr = "dbname='dbName' user='postgres' password='test123' host='localhost'"
conn = psycopg2.connect(connectStr)
dataList = []
with conn.cursor() as newCursor:
query = "select * from myTable where ID = <ID in the loop>"
newCursor.execute(query)
for row in newCursor:
dataList.append(row[0])
This code runs well for some IDs but then goes into the long-running state(more than 2 hours) for one random ID. This stuck ID when taken out and run separately, executes in 30 seconds. (Avg time for each ID execution is ~30 seconds)
I checked the database level locks and pg_stat_activity
table as well using the query -
SELECT
activity.pid,
activity.usename,
activity.query,
blocking.pid AS blocking_id,
blocking.query AS blocking_query
FROM pg_stat_activity AS activity
JOIN pg_stat_activity AS blocking ON blocking.pid = ANY(pg_blocking_pids(activity.pid));
but as it stands, it is just the "long-running" query and not the "blocking" query.
using the query
select * from pg_stat_activity where state = 'active'
I am able to see "BufFileRead" in the wait_event
column for a few milliseconds, it goes back to Null.
I am explicitly closing all the cursors and connections after each loop still it gets stuck on a "random" ID.
Can someone please guide me here on how to resolve this? I can share more information as needed.
Thank you,
Sanket