44

I want to create a repository and Commit a few files to it via any Python package. How do I do?

I do not understand how to add files for commit.

8 Answers 8

39

Solution using the requests library:

NOTES: I use the requests library to do the calls to GitHub REST API v3.

1. Get the last commit SHA of a specific branch

# GET /repos/:owner/:repo/branches/:branch_name
last_commit_sha = response.json()['commit']['sha']

2. Create the blobs with the file's content (encoding base64 or utf-8)

# POST /repos/:owner/:repo/git/blobs
# {
#  "content": "aGVsbG8gd29ybGQK",
#  "encoding": "base64"
#}
base64_blob_sha = response.json()['sha']

# POST /repos/:owner/:repo/git/blobs
# {
#  "content": "hello world",
#  "encoding": "utf-8"
#}
utf8_blob_sha = response.json()['sha']

3. Create a tree which defines the folder structure

# POST repos/:owner/:repo/git/trees/
# {
#   "base_tree": last_commit_sha,
#   "tree": [
#     {
#       "path": "myfolder/base64file.txt",
#       "mode": "100644",
#       "type": "blob",
#       "sha": base64_blob_sha
#     },
#     {
#       "path": "file-utf8.txt",
#       "mode": "100644",
#       "type": "blob",
#       "sha": utf8_blob_sha
#     }
#   ]
# }
tree_sha = response.json()['sha']

4. Create the commit

# POST /repos/:owner/:repo/git/commits
# {
#   "message": "Add new files at once programatically",
#   "author": {
#     "name": "Jan-Michael Vincent",
#     "email": "[email protected]"
#   },
#   "parents": [
#     last_commit_sha
#   ],
#   "tree": tree_sha
# }
new_commit_sha = response.json()['sha']

5. Update the reference of your branch to point to the new commit SHA (on master branch example)

# PATCH /repos/:owner/:repo/git/refs/heads/:branch
# {
#     "sha": new_commit_sha
# }

Finally, for a more advanced setup read the docs.

7
  • 1
    Thanks for this. It's difficult to follow just a description on how to do this and the code works and made it clear.
    – danl
    Nov 2, 2020 at 13:24
  • 2
    These are just HTTP request/response packages. It doesn't really explain how to do it or what they mean.
    – ingyhere
    Nov 24, 2020 at 17:12
  • I have a problem, When I created a new commit that consists of 2 files added to the repo, it deletes everything in the repo. Only the 2 files in new commits get uploaded. How can I prevent my new commit not deleting files and folders available in the repo Apr 14, 2022 at 7:18
  • 1
    @SANDEEPSS - I had the same issue, it's because you've forgotten to add the base_tree in step 3. Found in the docs: "If not provided, GitHub will create a new Git tree object from only the entries defined in the tree parameter. If you create a new commit pointing to such a tree, then all files which were a part of the parent commit's tree and were not defined in the tree parameter will be listed as deleted by the new commit."
    – Toby Smith
    Jul 3, 2022 at 19:50
  • 1
    You can skip step #2 if you're willing to use utf-8 rather than base64. In step #3, don't provide a sha and instead provide a content.
    – Toby Smith
    Jul 3, 2022 at 20:16
20

You can see if the new update GitHub CRUD API (May 2013) can help

The repository contents API has allowed reading files for a while. Now you can easily commit changes to single files, just like you can in the web UI.

Starting today, these methods are available to you:

1
19

Here is a complete snippet:

def push_to_github(filename, repo, branch, token):
    url="https://api.github.com/repos/"+repo+"/contents/"+filename

    base64content=base64.b64encode(open(filename,"rb").read())

    data = requests.get(url+'?ref='+branch, headers = {"Authorization": "token "+token}).json()
    sha = data['sha']

    if base64content.decode('utf-8')+"\n" != data['content']:
        message = json.dumps({"message":"update",
                            "branch": branch,
                            "content": base64content.decode("utf-8") ,
                            "sha": sha
                            })

        resp=requests.put(url, data = message, headers = {"Content-Type": "application/json", "Authorization": "token "+token})

        print(resp)
    else:
        print("nothing to update")

token = "lskdlfszezeirzoherkzjehrkzjrzerzer"
filename="foo.txt"
repo = "you/test"
branch="master"

push_to_github(filename, repo, branch, token)
2
  • This code will fail when file is not exists in repository. This will work if you are updating existing file. For new file it will throw error as Not Found. Jun 8, 2021 at 19:05
  • What is the correct way to use this API endpoint to create a new file? docs.github.com/en/rest/reference/… The endpoint is for creation too, but using it like this answer only allows updating an existing file.
    – mbtamuli
    Jul 5, 2021 at 12:49
12

Github provides a Git database API that gives you access to read and write raw objects and to list and update your references (branch heads and tags). For a better understanding of the topic, I would highly recommend you reading Git Internals chapter of Pro Git book.

As per the documentation, it is a 7 steps process to commit a change to a file in your repository:

  1. get the current commit object
  2. retrieve the tree it points to
  3. retrieve the content of the blob object that tree has for that particular file path
  4. change the content somehow and post a new blob object with that new content, getting a blob SHA back
  5. post a new tree object with that file path pointer replaced with your new blob SHA getting a tree SHA back
  6. create a new commit object with the current commit SHA as the parent and the new tree SHA, getting a commit SHA back
  7. update the reference of your branch to point to the new commit SHA

This blog does a great job at explaining this process using perl. For a python implementation, you can use PyGithub library.

3

Based on previous answer, here is a complete example. Note that you need to use POST if you upload the commit to a new branch, or PATCH to upload to an existing one.


    import whatsneeded
    
    GITHUB_TOKEN = "WHATEVERWILLBEWILLBE"
    
    def github_request(method, url, headers=None, data=None, params=None):
        """Execute a request to the GitHUB API, handling redirect"""
        if not headers:
            headers = {}
        headers.update({
            "User-Agent": "Agent 007",
            "Authorization": "Bearer " + GITHUB_TOKEN,
        })
    
        url_parsed = urllib.parse.urlparse(url)
        url_path = url_parsed.path
        if params:
            url_path += "?" + urllib.parse.urlencode(params)
    
        data = data and json.dumps(data)
        conn = http.client.HTTPSConnection(url_parsed.hostname)
        conn.request(method, url_path, body=data, headers=headers)
        response = conn.getresponse()
        if response.status == 302:
            return github_request(method, response.headers["Location"])
    
        if response.status >= 400:
            headers.pop('Authorization', None)
            raise Exception(
                f"Error: {response.status} - {json.loads(response.read())} - {method} - {url} - {data} - {headers}"
            )
    
        return (response, json.loads(response.read().decode()))
      
    def upload_to_github(repository, src, dst, author_name, author_email, git_message, branch="heads/master"):
        # Get last commit SHA of a branch
        resp, jeez = github_request("GET", f"/repos/{repository}/git/ref/{branch}")
        last_commit_sha = jeez["object"]["sha"]
        print("Last commit SHA: " + last_commit_sha)
    
        base64content = base64.b64encode(open(src, "rb").read())
        resp, jeez = github_request(
            "POST",
            f"/repos/{repository}/git/blobs",
            data={
                "content": base64content.decode(),
                "encoding": "base64"
            },
        )
        blob_content_sha = jeez["sha"]
    
        resp, jeez = github_request(
            "POST",
            f"/repos/{repository}/git/trees",
            data={
                "base_tree":
                last_commit_sha,
                "tree": [{
                    "path": dst,
                    "mode": "100644",
                    "type": "blob",
                    "sha": blob_content_sha,
                }],
            },
        )
        tree_sha = jeez["sha"]
    
        resp, jeez = github_request(
            "POST",
            f"/repos/{repository}/git/commits",
            data={
                "message": git_message,
                "author": {
                    "name": author_name,
                    "email": author_email,
                },
                "parents": [last_commit_sha],
                "tree": tree_sha,
            },
        )
        new_commit_sha = jeez["sha"]
    
        resp, jeez = github_request(
            "PATCH",
            f"/repos/{repository}/git/refs/{branch}",
            data={"sha": new_commit_sha},
        )
        return (resp, jeez)

1

I have created an example for committing with multiple files using Python:

import datetime
import os
import github
   
# If you run this example using your personal token the commit is not going to be verified.
# It only works for commits made using a token generated for a bot/app 
# during the workflow job execution.

def main(repo_token, branch):

    gh = github.Github(repo_token)

    repository = "josecelano/pygithub"

    remote_repo = gh.get_repo(repository)

    # Update files:
    #   data/example-04/latest_datetime_01.txt
    #   data/example-04/latest_datetime_02.txt
    # with the current date.

    file_to_update_01 = "data/example-04/latest_datetime_01.txt"
    file_to_update_02 = "data/example-04/latest_datetime_02.txt"

    now = datetime.datetime.now()

    file_to_update_01_content = str(now)
    file_to_update_02_content = str(now)

    blob1 = remote_repo.create_git_blob(file_to_update_01_content, "utf-8")
    element1 = github.InputGitTreeElement(
        path=file_to_update_01, mode='100644', type='blob', sha=blob1.sha)

    blob2 = remote_repo.create_git_blob(file_to_update_02_content, "utf-8")
    element2 = github.InputGitTreeElement(
        path=file_to_update_02, mode='100644', type='blob', sha=blob2.sha)

    commit_message = f'Example 04: update datetime to {now}'

    branch_sha = remote_repo.get_branch(branch).commit.sha
   
    base_tree = remote_repo.get_git_tree(sha=branch_sha)
 
    tree = remote_repo.create_git_tree([element1, element2], base_tree)

    parent = remote_repo.get_git_commit(sha=branch_sha)

    commit = remote_repo.create_git_commit(commit_message, tree, [parent])

    branch_refs = remote_repo.get_git_ref(f'heads/{branch}')

    branch_refs.edit(sha=commit.sha)
0

I'm on Google App Engine (GAE) so beside of python, I can create a new file, update it, even delete it via a commit and push into my repo in GitHub with GitHub API v3 in php, java and go.

Checking and reviewing some of the available third party libraries to create like the example script that presented in perl, I would recommend to use the following:

As you aware, you can get one site per GitHub account and organization, and unlimited project sites where the websites are hosted directly from your repo and powered by Jekyll as default.

Combining Jekyll, Webhooks, and GitHub API Script on GAE, along with an appropriate GAE Setting, it will give you a wide possibility like calling external script and create a dynamic page on GitHub.

Other than GAE, there is also an option run it on Heroku. Use JekyllBot that lives on a (free) Heroku instance to silently generates JSON files for each post and pushing the changes back to GitHub.

0

Solution using Github API and requests package

You can commit files to a repository by Create or update file contents endpoint.

PUT https://api.github.com/repos/{owner}/{repo}/contents/{path}
{
   "message": "<commit_message>",
   "committer": {
       "name": "<username>",
       "email": "<email>"
   },
   "content": <content-encoded-in-base64>
}

Sharing the python snippet

import base64
import requests


class GitHubRepo:
    def __init__(self):
        self.base_url = "https://api.github.com/repos"
        self.repo = "GITHUB_REPOSITORY_NAME"
        self.owner = "GITHUB_USERNAME"
        self.token = "GITHUB_ACCESS_TOKEN"
        self.branch = "BRANCH_NAME"
        self.email = "EMAIL"

    def get_headers(self):
        headers = {
            'Authorization': 'Bearer {}'.format(self.token),
            'User-Agent': (
                'Mozilla/5.0 (Windows NT 6.1; Win64; '
                'x64; rv:47.0) Gecko/20100101 Firefox/47.0'
            ),
            'X-GitHub-Api-Version': '2022-11-28',
            'Accept': 'application/vnd.github+json'
        }
        return headers

    def put(self, url, data):
        try:
            response = requests.put(
                url,
                headers=self.get_headers(),
                json=data
            )
            return response.json()
        except requests.exceptions.RequestException:
            return None

    def create_file(self, content_in_bytes, path, commit_message):
        encoded = str(base64.b64encode(content_in_bytes).decode("utf-8"))
        data = {
            "message": commit_message,
            "committer": {
                "name": self.owner,
                "email": self.email
            },
            "content": encoded
        }
        url = "{}/{}/{}/contents/{}".format(
            self.base_url,
            self.owner,
            self.repo,
            path
        )
        response = self.put(url, data)
        return response
New contributor
sreehari.codes is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
1
  • Hello, please don't post code only and add an explantation as to why you think that this is the optimal solution. People are supposed to learn from your answer, which might not occur if they just copy paste code without knowing why it should be used.
    – Destroy666
    20 hours ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.