39

I'm using Boto to connect to Amazon S3 in my Python program. I'm able to open a connection and upload files to a bucket. I figured I should then close the connection to release resources and, more important, to avoid any security risks from leaving an open connection hanging around. I assumed I should call the close() method. But I tested this as follows: 1. Open connection. 2. Close connection. 3. Upload file to bucket.

I figured step 3 would fail, but the upload worked! So what does close() do? If it doesn't really close the connection, what should I use in place of close()? Or is it just unnecessary to close the connection?

I've looked for the answer in the Boto tutorial, the Boto API reference, and this StackOverflow post, but no luck so far.

Thanks for your help.

0

3 Answers 3

25

Your step 3 worked because boto has code that will automatically re-open closed connections and retry requests on errors. There is little to gain by manually closing the boto connections because they are just HTTP connections and will close automatically after a few minutes of idle time. I wouldn't worry about trying to close them.

9

Under the covers, boto uses httplib. This client library supports HTTP 1.1 Keep-Alive, so it can and should keep the socket open so that it can perform multiple requests over the same connection.

connection.close() does not actually close the underlying sockets. Instead, it removes the reference to the underlying pool of httplib connections, which allows the garbage collector to run on them, and that's when the actual socket close happens.

Obviously, you also can let the garbage collector run by not keeping a reference to the boto connection itself. But there are performance benefits to reusing the boto connection (e.g. see the Keep-Alive note above).

Fortunately, in most cases, you don't have to call connection.close() explicitly. For more details on one case where you DO have to call close, see my answer to the StackOverflow post that's linked in the question.

0

There is at least one instance where leaving this connection open can cause a failure. The answer above led me to the solution. Below, addUnverifiedEmail does an AWS RDS DB insert. So, the boto3 connection is still in scope and active when attempting to do the insert. This is from my Lambda (python).

boto3.client.sign_up(
                ClientId='xxxxxxxxxxxxxxxxxxxxxxxxxx',
                Username=self._user['email'],
                Password=self._user['password'],
                UserAttributes=attributes
)
dbUserInstance.addUnverifiedEmail(self._user['email'])

This resulted in (error 1205 Lock wait timeout exceeded). Below, addUserToCognito creates the boto3 connection and it falls out of scope before the insert.

self.addUserToCognito()
dbUserInstance.addUnverifiedEmail(self._user['email'])

The insert succeeded after making this change. Luckily for me, addUnverifiedEmail was the last function call inside addUserToCognito so it was easy to move it outside. Other, more complex, code may not have the same ability. So, client.close() not actually closing the connection and keeping it closed could be a pretty big flaw.

1
  • I assume that is a boto3.client('cognito-idp')? And the error you are seeing is a MySQL error related to locked records on your RDS instance? I'm confused as to how they are related. Is the cognito-idp writing to MySQL as well? Have you linked cognito to create DB users? Perhaps your addUnverifiedEmail insert is blocked by cognito also doing an insert? Why is it both signing up and adding unverified for the same user email?
    – Davos
    Nov 29, 2018 at 15:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.