105

I am trying to understand when I should use a Resource and when I should use a Client.

The definitions provided in boto3 docs don't really make it clear when it is preferable to use one or the other.

2

2 Answers 2

108

boto3.resource is a high-level services class wrap around boto3.client.

It is meant to attach connected resources under where you can later use other resources without specifying the original resource-id.

import boto3
s3 = boto3.resource("s3")
bucket = s3.Bucket('mybucket')

# now bucket is "attached" the S3 bucket name "mybucket"
print(bucket)
# s3.Bucket(name='mybucket')

print(dir(bucket))
#show you all class method action you may perform

OTH, boto3.client are low level, you don't have an "entry-class object", thus you must explicitly specify the exact resources it connects to for every action you perform.

It depends on individual needs. However, boto3.resource doesn't wrap all the boto3.client functionality, so sometime you need to call boto3.client , or use boto3.resource.meta.client to get the job done.

7
  • 2
    Thanks, this helps a little, the documentation itself is a bit vague, but I guess the key is that clients map to specific API calls that you can make. Sep 2, 2016 at 13:05
  • 1
    Im initializing my client by passing credentials as parameters to the client. import boto3 client = boto3.client( 's3', aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY ). Can i do something similar for resource? im not able to find this in the documentation.
    – Naveen
    Jan 4, 2018 at 22:52
  • 9
    The example didn't show anything that shows the difference.
    – Nabin
    Feb 11, 2018 at 1:45
  • 5
    Anyone need comprehensive example, please take a look here : stackoverflow.com/questions/42809096/…
    – mootmoot
    May 7, 2018 at 6:56
  • 1
    Does anyone have good suggestions on the documentation over the "resource" interfaces? The BOTO3 docs focus mainly on the low-level "client" interfaces.
    – gye
    May 11, 2020 at 20:30
2

If possible use client over resource, especially if dealing with s3 object lists, and then trying to get basic information on those objects themselves.

Client calls s3 10,000/1000 = 10 times and gives you a lot of information on each object in each call..

Resource, I assume calls s3 10,000 times(or maybe same as client??), but if you take that object and try to do something with it, that is probably another call to s3, making this about 20x slower than client.

my Test reveals the following results.

s3 = boto3.resource("s3")
s3bucket = s3.Bucket(myBucket)
s3obj_list = s3bucket.objects.filter(Prefix=key_prefix)
tmp_list = [s3obj.key for s3obj in s3obj_list]
(tmp_list = [s3obj for s3obj in s3obj_list] gives same ~9min results)

When trying to get a list of 150,000 files, took ~9 minutes. If s3obj_list is indeed pulling 1000 files a call and buffering it, s3obj.key is probably not part of it and makes another call.

...some sort of loop, that also sets ContinuationToken...
response = client.list_objects_v2(
   Bucket = bucket,
   Prefix = prefix,
   ContinuationToken=response["NextContinuationToken"],
)
...

Client took ~30 seconds to list the 150,000 files.

I don't know if resource buffers 1000 files at a time but if it doesn't that is a problem.

I also don't know if it is possible for resource to buffer the information attached to the object, but that is another problem.

I also don't know if using pagination could make client faster/easier to use.

Anyone who knows the answer to the 3 questions above please do. I'd be very interested to know.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.