31

I'd like to automate deploying our site to AWS S3. I've written a node script to automate building and uploading the site, but I'd like to have the script automatically run whenever the master branch of our repo is updated on github.

I looked into AWS CodeDeploy, but it looks like that's for specifically deploying to EC2. I've also looked at AWS Lambda, but there doesn't seem to be a clear way to pull a copy of the repo using git so I can run the script.

Any services (preferably tied to AWS) that I can use?

8 Answers 8

12

I had the same goal some time ago and have now released a little tool, which solves the problem at least for me. It uses AWS Lambda and deploys a specific branch of the repository to S3 after push. You can take full advantage of the GitHub deploy key, which has fewer permissions as personal access tokens and can be configured per repository.

Please take a look at github-bucket, it might help you too.

github-s3-deploy-architecture

10

Rather than using an AWS service directly (as you say they nearly all expect a much more complicated setup, deploying to EC2 etc), you might be better off using a CI provider such as Shippable, Codeship or Wercker.

These all have the ability to fire from git updates, run build commands, install utilities into their CI images/containers and copy files to S3.

There's probably some startup which has built an exact tool for your purpose, but they haven't appeared on my radar yet :-)

9

I know it's not git deploy.... But Instead of setting up a CI box, I just used s3cmd.

http://s3tools.org/s3cmd

Executing this command syncs my build directory with s3.

s3cmd sync -r ~/code/mysite/build s3://www.mysite.com --delete-removed

I'm using it on Linux. Not sure what their OSX and Windows stories are.

If you're really after a git push solution, you could set up a timed job which pulls your git repo to a folder and then executes this against it. I do this elsewhere on a cheap Linux VM. Unless you're going to go full CI though, there's probably not much point.

2
  • This could get added into a git post-commit hook and it would do.
    – Zlatko
    Apr 13, 2016 at 8:36
  • Yeah, good idea. It just occurred to me, with the recent announcement that Windows will be able to run Linux binaries, s3cmd will probably work on Windows as well! So cool. blogs.windows.com/buildingapps/2016/03/30/… Apr 13, 2016 at 21:22
7

You can set this up with a very simple 2-step CodePipeline. Basically, you just fill out the different sections in the AWS Console. There's no need for a separate CI tool and its added complexity.

In the first step of the pipline, pull from Github and store in S3. You can easily set this up through the AWS Console.

In the next CodeDeploy step, you can use the AWS CLI (pre-installed in CodeDeploy) to do a

cd /path/to/public/directory && aws s3 sync --acl public-read --delete . s3://your.bucket.name

You'll have to set the environment variables for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY so that AWS CLI can run during your deploy step, and that can also be done in the AWS console for CodeDeploy, in the Advanced section under "environment variables". Once the environment variables have been set and if that AWS user has the correct permissions, you can run any aws-cli command you want inside your CodeDeploy.

Once this is done, when you check in to Github, CodePipeline will kick off, and a few minutes later your files will be on S3.

7
  • How do you set up the pull from Github to S3? Is that with Lambda? Or the CodePipeline?
    – LC1983
    Dec 3, 2017 at 0:01
  • In your CodePipeline wizard, it asks you for the source. Github is one of the supported sources, so choose Github as your source. You'll need to set up Github authentication so that your CodePipeline has permissions to read your Github account (if it's a private repo), but that's all done through the console. The first time I did this, I had no problems doing everything through the console -- the steps are pretty clear. Dec 27, 2017 at 20:01
  • I can get the code in, but can’t set S3 as n output.
    – LC1983
    Dec 27, 2017 at 20:02
  • 1
    You have to write the build scripts to run on CodeDeploy. If you just want to copy the files to S3, your build script would be just the single line of script shown in my answer. Jan 21, 2018 at 16:44
  • 1
    Hi. Actuallly you don't even need a pipeline to do that copy. The only thing you need is a Build Project, that should be created in CodeDeploy. You can link your GitHub account since there and specify in addittional conf. that you want run the build each time your repository changes. It is necessary to create the 2 env. variables that @PatrickChu said and in the BuildSpec section you should select "Use a build command". In my case I used this command: aws s3 sync --acl public-read --delete . s3://MY-BUCKET-NAME-HERE I previously create a new Bucket in S3 where my data is being placed.
    – Ruben
    Nov 11, 2018 at 21:51
6

If you are using TravisCI, the deployment is pretty straight forward. The only part you will need to add to your .travis.yml is :

deploy:                         
  provider: s3                         
  access_key_id: "YOUR AWS ACCESS KEY"
  secret_access_key:
    secure: "w/DlbHt1+IAMENCRYPTED"
  bucket: "YOUR BUCKET"

My blog post explains all the details for the AWS side (user setup, IAM and S3 bucket configurations) as well as the github and travisCI side.

1

Perhaps overpowered for your simple use case, but you could create a very simple CodePipeline to push your github repository to S3.

2
  • How do you select S3 as an output in CodePipeline?
    – LC1983
    Dec 3, 2017 at 0:03
  • See my answer above. You have to write a single line of code in your CodePipeline (or CodeDeploy) to move everything into S3. Mar 8, 2019 at 19:10
0

i also recommend to use codeship,simple and easy, but you need to create IAM user with proper permission (which is policy) to S3 bucket.

the basic plan for codeship is free.

Well there might be a problem so far i can see codeship will not remove files as you remove files in github, after all, s3 is not github repo, but anyway, the putObject operations for lots of github update just works good enough to me.

1
0

I have used Deploy Bot in the past and I have been quite happy with it.

It can push to S3 or even FTP via Git and also it can run your build scripts and even push a notification to Slack for you.

https://deploybot.com/

Not the answer you're looking for? Browse other questions tagged or ask your own question.