9

I have a Python app running on an OpenWrt router (Chaos Calmer) that uses the boto3 package for creating and updating AWS IoT thing shadows. The app is packaged with its own Python and dependencies (including boto3 and botocore). boto3 depends on botocore, which has a data folder with a bunch of json files for other AWS services unused by my app (S3, dynamoDB, route53, lambda, etc. etc.). Out of all the app's resources, this folder takes up the most space by far. Eventually I will need to do OTA updates over cellular to many of these routers, meaning data usage is a concern. Therefore I would like to trim out as many unused json files from the data folder as possible without breaking the code.

These are the only three boto3 and botocore imports my app uses:

from boto3 import client as boto3_client
from boto3.exceptions import ResourceNotExistsError
from botocore.exceptions import ClientError

How can I determine which json files I can get rid of?

4
  • Note: if you want to do controlled incremental updates, use casync to reuse unchanged parts.
    – o11c
    Commented Jan 8, 2019 at 3:17
  • @o11c thanks for the tip
    – gliff
    Commented Jan 8, 2019 at 19:36
  • 1
    Thanks. I only needed S3. So I removed all data folders, left the python modules in place and added S3 back in. Worked perfectly. Botocore went from 60mb to 1.5mb. Thats a win when you are trying to squeeze your code into a lambda.
    – RodP
    Commented Mar 18, 2022 at 22:35
  • @gliff you should add an answer so I can upvote
    – Clintm
    Commented Aug 12, 2022 at 17:29

1 Answer 1

0

Original update (1-8-19), reposted as an answer (1-13-2023): I figured out how to solve this problem. Instead of removing a handful of files that didn't look relevant, then running the app to make sure it wasn't broken, then removing more files etc. etc., I did the opposite: removed all the files, then slowly added files that looked relevant back until only the necessary files remained. In the end the data folder went from ~35 Mb to < 2 Mb 🙂 (that's a huge deal for my use case)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.