I want to understand Github Checks API so that I can use them to retrieve data. On following Github documentation https://docs.github.com/en/rest/guides/getting-started-with-the-checks-api I am able to derive that check runs are associated with the sha of the change and at each commit on branch check suite is created. Checks API helps in getting all this information. But I want more clarity on three of them in terms of differences. Can anyone please explain these three terms using simple example and terms?
-
What three terms exactly? Are you asking about the difference between the status and checks APIs?– NickofthymeMay 16, 2022 at 14:19
-
@Nickofthyme check API, check runs and check suites.. sort of difference in three of these and if any example to differentiate them.– Stack_IQMay 18, 2022 at 9:30
1 Answer
So, first of all, the GitHub Commit Statuses API is separate from the GitHub Checks API (includes suites & runs), so let's look at them individually first then I'll explain the differences.
Before we get into it, I want to differentiate a PR Check from a Check Run, to avoid confusion. A PR Check describes the current state (trying not to say status here 😉) of a given job or task running in CI or elsewhere on a specific PR commit. These can be created via either a Commit Status or a Check Run. All the items in the pink box below are PR Checks, notice the Hide all checks button.
GitHub statuses - API
I see this as the simple, all-purpose API for reporting PR Checks for a given commit. It's easy and doesn't require jumping through hoops to just display a simple result of a PR Check for a given commit. This API also came before the check API so it's a little less powerful.
Pros
- Simple and easy API
- Simple relation with
context
as the identifier. - Can have a fully customizable text description on the PR Checks UI.
- You can create statuses as a user with a Personal Access Token (aka PAT) without needing to create a GitHub App, though does work with GitHub Apps too!
Cons
- Limited
status
options, only allowerror
,failure
,pending
,success
, noconclusion
subset option to define completed jobs like Check Runs. - No concept of job timing or duration.
- No grouping of statuses, like Check Runs are grouped into Check Suites by their GitHub App
- No annotations
- No detailed
output
logs. This is not too important as you could just link to the URL where the actual PR Check was run such as in CircleCI, Jenkins, etc. But the user is not always authorized to view these runs so the output could be helpful for open source repos that have non-public CI.
GitHub Checks - API
The Checks API is the latest and greatest tool for displaying task results on commits, which can essentially do everything the Commit Statuses API can do and more. Check Runs belong to one Check Suite, one Check Suite can have many Check Runs. You can only have one Check suite per commit (i.e. head_sha
) per GitHub App, attempting to create another for a given App will just return the previously created Check Suite. Thus, a new Check Run is automatically assigned to the Check Suite based on the authenticated GitHub App, you cannot manually assign Run to Suites.
Contrary to statuses, Check runs are identified by an auto-generated check_run_id
and not a context
string.
I haven't touched too much on the Check Suites API because they are really just a grouping of Check Runs which is pretty self-explanatory and they don't affect any of the PR Checks UI, only the grouping of Check Runs in the checks tab. One thing to note is that by default you can create a Check Run without having to first create a Check suite and GitHub will just create a new Check Suite for you.
Pros
- Greater granularity of
status
/conclusion
for a run. - A lot of power to display the result of a PR Check.
- Can provide run context via output summary in Markdown.
- Can create annotations for specific lines of code to add information about the analysis performed, like linting error for a given line of code. These will show up in the PR files tab UI similar to PR code comments as well as in the PR Checks tab with any other Check Run output.
- Has time awareness to report on durations automatically with little effort.
Cons
- A little more complicated API and relationships to manage.
- Part of the description in the PR UI is auto-generated based on the
status
conclusion
and duration of the check run task. - Cannot be created via a user PAT, must be created from an authenticated GitHub App. Read access to this API does not require authenticating as a GitHub App.
One edge case you likely won't come across but is good to know: If you are creating a new Check Run with the exact same
name
of an existing check run under the same authenticated app. The resulting behavior is a little strange but doing this will create the new Check Run under the same name, but will not delete the existing Check Run. However, the Check Suite will not see or link to the existing Check Run, even in the PR UI. BUT!! If you change thename
of either such that the runs now have uniquename
s, it will be linked up again. It seems GitHub just does a sort by date and then filters by unique names when looking up Check Runs in a Check Suite. This does not apply with identicalnames
from different authenticated apps.
Comparisons
Below is a mapping of sorts to compare similar options between the Commit Statuses API and the Check Runs API. They are not exactly 1:1 but similar.
Commit Status | Check Run | Option Desciption |
---|---|---|
sha |
head_sha |
These are equivalent with the minor exception that the Commit Status is linked to the sha directly, whereas the head_sha is linked to the Check Suite that the Check Run belongs to. |
context |
name |
The context is used as an identifier but both define the title of the PR Check in the PR UI. Because Check Runs are tracked by the check_run_id the name option may change without creating a new Check Run. This is not the case for context , changing the context will always create a new Commit Status. You may not have duplicate names for Check Runs created with the same GitHub App, see note above. |
context * |
external_id |
The external_id is meant for keeping track of Check Runs without having to store ids or always keep the name constant. This is only somewhat similar to the context option but only for the purpose of identifying the Check Run. |
description |
output.title |
The main difference here is that description gives you the full space to work with, where output.title will be displayed after the auto-generated status string. |
target_url |
details_url |
These are somewhat equivalent, the first difference is that target_url will not show unless defined whereas details_url will default to the check run URL in the PR UI. The other difference is the Check Runs Details button on the PR Checks UI will always link the the Checks page which will present a link to the details_url if defined. |
state |
status |
These are very similar but have slightly different allowed values but effectively appear the same in the PR UI. |
N/A | conclusion |
This just shows the increased power of the PR Checks API where you have more granular control over the PR Check status, though not a big difference in the UI, see example variations below. |
N/A | started_at |
No comparable option for Commit Statuses. |
N/A | completed_at |
No comparable option for Commit Statuses. |
N/A | actions |
No comparable option for Commit Statuses. |
N/A | output |
No comparable option for Commit Statuses. |
N/A | output.annotations |
No comparable option for Commit Statuses. |
* Only somewhat similar, not a direct equivalent.
PR Checks UI Component mappings
I've taken a simple PR Check and highlighted the differences between the elements of the UI between the Commit Status API and the Check Runs API. Very similar but slight differences.
Below are example variations to relate the options to their impact on the PR Checks UI.
Commit status variations
Check Run variations
If a check run is in an incomplete state for more than 14 days, then the Check Run's
conclusion
becomesstale
. Only GitHub can mark Check Runs asstale
.
Checks Tab in PR UI
The Checks Tab in the PR UI will display all created Check Suites and their child Check Runs and allows you to set the output
option to control the content of this page. This page would also show images
and annotations
if you defined those too.
One final note: You cannot DELETE any Commit Status, Check Run or Check Suite once created, only updating is allowed. With these statuses being so ephemeral this is not that big of a deal but in case you were wondering.
Update (6/21/2022)
I found two more quirks to explain. First, the details_url
for Check Runs does not set the value of the Details
button, instead the Details
button actually always redirects to the Checks page which has a link to the details_url
if defined.
Second, once a Check Run status
is set to 'completed'
, there is not way to un-complete the Check run. But you can get around this by creating a new Check Run with the same name
and set that status
to something other than 'completed'
. See this edge case with duplicate-name
d Check Runs explained above in detail.
Update (6/29/2022)
As mentioned above, the Check Runs API keeps track of the duration of the run. It is important to note that the started_at
time is set only once whenever a check run is created, regardless of the defined status
. For example, if I trigger a CI build and set all jobs status
to queued
, the jobs run duration will be inaccurate. A good way to fix this is to always set the started_at
time whenever you set the status
to in_progress
. Something as simple as const started_at = new Date().toISOString()
(javascript) will do the trick.
-
1Thanks @Nickofthyme for the answer. This made the comparison quite simple.– Stack_IQNov 9, 2022 at 11:00