1

I'm trying to set up a daily AWS Glue job that loads data into a RDS PostgreSQL DB. But I need to truncate my tables before loading data into them, since those jobs work on the whole dataset.

To do this, I'm implementing the solution given here: https://stackoverflow.com/a/50984173/11952393.

It uses the pure Python library pg8000. I followed the guidelines in this SO, downloading the library tar, unpacking it, adding the empty __init.py__, zipping the whole think, uploading the zip file to S3 and adding the S3 URL as a Python library in the AWS Glue job config.

When I run the job, the pg8000 module seems to be imported correctly. But then I get the following error:

AttributeError: module 'pg8000' has no attribute 'connect'

I am most certainly doing something wrong... But can't find what. Any constructive feedback is welcome!

2
  • Is it spark or python shell job? Aug 20, 2019 at 19:09
  • Sandeep, it is a Python Spark job. Aug 21, 2019 at 5:51

3 Answers 3

0

Here is what made it work for me.

  1. Do a pip install of the pg8000 package in a separate location

    pip install -t /tmp/ pg8000

  2. You would see 2 directories in the /tmp directory

    pg8000
    scramp
    
  3. Zip the above 2 directories separately

    cd /tmp/
    zip -r pg8000.zip pg8000/
    zip -r scramp.zip scramp/
    
  4. Upload these 2 zip files in an S3 location

  5. While creating the job or the Dev Endpoint mention these 2 zip files in the Python Library Path field

s3://<bucket>/<prefix>/pg8000.zip,s3://<bucket>/<prefix>/scramp.zip
0

Add

install_requires = ['pg8000==1.12.5']

in _setup.py file which is generating .egg file

You should able to access library.

1
  • I've edited this answer to remove the bit about doubling percent signs as that bit isn't correct. import pg8000.dbapi c = pg8000.dbapi.connect("postgres") cursor = c.cursor() cursor.execute("select 'hello' like '%ell%';") for r in cursor.fetchall(): print(r)
    – Tony Locke
    Sep 7, 2021 at 9:11
0

I had this exact same error. It turns out that I was assembling the .zip file to be uploaded to AWS incorrectly (at the command prompt on Mac).

I created the .zip file with:

zip ../function.zip *

which resulted in the directories (such as pg8000, but all the others as well) being included, but not their contents. As such, the error message was true! There were no attributes, because there wasn't even a __init__.py file!

Instead, what should be done is:

zip ../function.zip -r *

When done correctly, the contents of the directories should browsable within the AWS Lambda Code editor.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.