315

I'm searching for the way to get $ go get work with private repository, after many google try.

The first try:

$ go get -v gitlab.com/secmask/awserver-go
Fetching https://gitlab.com/secmask/awserver-go?go-get=1
https fetch failed.
Fetching http://gitlab.com/secmask/awserver-go?go-get=1
Parsing meta tags from http://gitlab.com/secmask/awserver-go?go-get=1 (status code 200)
import "gitlab.com/secmask/awserver-go": parse http://gitlab.com/secmask/awserver-go?go-get=1: no go-import meta tags
package gitlab.com/secmask/awserver-go: unrecognized import path "gitlab.com/secmask/awserver-go

Yep, it did not see the meta tags because I could not know how to provide login information.

The second try:

Follow https://gist.github.com/shurcooL/6927554. Add config to .gitconfig.

[url "ssh://[email protected]/"]
    insteadOf = https://gitlab.com/
$ go get -v gitlab.com/secmask/awserver-go --> not work
$ go get -v gitlab.com/secmask/awserver-go.git --> work but I got src/gitlab.com/secmask/awserer-go.git

Yes it work but with .git extension with my project name, I can rename it to original but do it everytime $ go get is not so good, is there an otherway?

0

19 Answers 19

257

You have one thing to configure. The example is based on GitHub but this shouldn't change the process:

$ git config --global [email protected]:.insteadOf https://github.com/
$ cat ~/.gitconfig
[url "[email protected]:"]
    insteadOf = https://github.com/
$ go get github.com/private/repo

For Go modules to work (with Go 1.11 or newer), you'll also need to set the GOPRIVATE variable, to avoid using the public servers to fetch the code:

export GOPRIVATE=github.com/private/repo
9
  • 1
    The only drawback with this is that you cant have different config for each host (e.g if you're using multiple providers) without altering the global git configuration for each time. Then it's much better to specify this on «ssh-level» as described her: stackoverflow.com/questions/27500861/…
    – Joachim
    Commented Feb 23, 2017 at 10:21
  • 13
    go env -w GOPRIVATE=github.com/<OrgNameHere>/*as described here stackoverflow.com/questions/58305567/…
    – huggie
    Commented Oct 15, 2020 at 3:35
  • 1
    There can be cache messing things up. Probably from attempting to get some repo without changing the git config as described above. What solved it for me : rm -rf $GOPATH/pkg/mod/cache/vcs Commented Mar 31, 2021 at 21:08
  • 2
    Is anyone able to get this working with gitlab? I keep getting the error The project you were looking for could not be found or you don't have permission to view it. when I try. I can run git commands via ssh just fine. This only works for me with the token based solution.
    – Solidus
    Commented Aug 25, 2021 at 9:45
  • 2
    This appears to be broken in 2021 with go 1.16. Git clone works with public and private repos on github but go get (and go mod tidy) fail on the private repo. Oddly enough, the go commands work correctly with private repos on my own gitea server.
    – Kevin Shea
    Commented Nov 25, 2021 at 17:43
60

I had a problem with go get using private repository on gitlab from our company. I lost a few minutes trying to find a solution. And I did find this one:

  1. You need to get a private token at:
    https://gitlab.mycompany.com/profile/account

  2. Configure you git to add extra header with your private token:

     $ git config --global http.extraheader "PRIVATE-TOKEN: YOUR_PRIVATE_TOKEN"
    
  3. Configure your git to convert requests from http to ssh:

     $ git config --global url."[email protected]:".insteadOf "https://gitlab.mycompany.com/"
    
  4. Finally you can use your go get normally:

     $ go get gitlab.com/company/private_repo
    
7
  • 6
    Interesting use of http.extraheader. +1
    – VonC
    Commented Aug 29, 2017 at 10:45
  • 67
    will --global also send your private token to other git server in case you using many repo? any secure risk?
    – secmask
    Commented Aug 29, 2017 at 15:06
  • Anyone any review on that? I think it will send it to public github repo when we do go get on them
    – hemu
    Commented May 30, 2020 at 11:45
  • 9
    There is another option where you send token only to your gitlab - but I am not sure this is encoded. git config --global url."https://${user}:${personal_access_token}@mygitlab.com".insteadOf "https://mygitlab.com"
    – S.R
    Commented Sep 10, 2020 at 15:25
  • 1
    I think @S.R solution is best (even though I had to add a trailing slash to the first url to make it work for my case => [...] url."https://${user}:${personal_access_token}@mygitlab.com/".insteadOf [...] Commented Apr 16, 2021 at 15:10
56

The proper way is to manually put the repository in the right place. Once the repository is there, you can use go get -u to update the package and go install to install it. A package named

github.com/secmask/awserver-go

goes into

$GOPATH/src/github.com/secmask/awserver-go

The commands you type are:

cd $GOPATH/src/github.com/secmask
git clone [email protected]:secmask/awserver-go.git
4
  • 11
    @secmask go get is designed as a tool for the common case. The Go team explicitly decided against adding configurability so people adhere to standards instead of rolling out their own cruft. It was never made for the case you have (i.e. private repositories).
    – fuz
    Commented Dec 16, 2014 at 11:27
  • 1
    This is an interesting perspective, but the big limitation of this approach is the handling of recursive, private dependencies. Handling this in a scalable way would essentially require re-implementing go get, if SSH access couldn't be handled as described in other answers. Commented Mar 1, 2023 at 12:09
  • "manually put the repository in the right place" isn't a solution in a Go modules world.
    – dolmen
    Commented Jun 28 at 13:57
  • @dolmen Correct. This answer predates Go modules.
    – fuz
    Commented Jun 28 at 15:48
53

For people using private GitLabs, here's a snippet that may help: https://gist.github.com/MicahParks/1ba2b19c39d1e5fccc3e892837b10e21

Also pasted below:

Problem

The go command line tool needs to be able to fetch dependencies from your private GitLab, but authenticaiton is required.

This assumes your private GitLab is hosted at privategitlab.company.com.

Environment variables

The following environment variables are recommended:

export GO111MODULE=on
export GOPRIVATE=privategitlab.company.com # this is comma delimited if using multiple private repos

The above lines might fit best in your shell startup, like a ~/.bashrc.

Explanation

GO111MODULE=on tells Golang command line tools you are using modules. I have not tested this with projects not using Golang modules on a private GitLab.

GOPRIVATE=privategitlab.company.com tells Golang command line tools to not use public internet resources for the hostnames listed (like the public module proxy).

Get a personal access token from your private GitLab

To future proof these instructions, please follow this guide from the GitLab docs. I know that the read_api scope is required for Golang command line tools to work, and I may suspect read_repository as well, but have not confirmed this.

Set up the ~/.netrc

In order for the Golang command line tools to authenticate to GitLab, a ~/.netrc file is best to use.

To create the file if it does not exist, run the following commands:

touch ~/.netrc
chmod 600 ~/.netrc

Now edit the contents of the file to match the following:

machine privategitlab.company.com login USERNAME_HERE password TOKEN_HERE

Where USERNAME_HERE is replaced with your GitLab username and TOKEN_HERE is replaced with the access token aquired in the previous section.

Common mistakes

Do not set up a global git configuration with something along the lines of this:

git config --global url."[email protected]:".insteadOf "https://privategitlab.company.com"

I beleive at the time of writing this, the SSH git is not fully supported by Golang command line tools and this may cause conflicts with the ~/.netrc.

Bonus: SSH config file

For regular use of the git tool, not the Golang command line tools, it's convient to have a ~/.ssh/config file set up. In order to do this, run the following commands:

mkdir ~/.ssh
chmod 700 ~/.ssh
touch ~/.ssh/config
chmod 600 ~/.ssh/config

Please note the permissions on the files and directory above are essentail for SSH to work in it's default configuration on most Linux systems.

Then, edit the ~/.ssh/config file to match the following:

Host privategitlab.company.com
  Hostname privategitlab.company.com
  User USERNAME_HERE
  IdentityFile ~/.ssh/id_rsa

Please note the spacing in the above file matters and will invalidate the file if it is incorrect.

Where USERNAME_HERE is your GitLab username and ~/.ssh/id_rsa is the path to your SSH private key in your file system. You've already uploaded its public key to GitLab. Here are some instructions.

4
  • 3
    There's something that always rubbed me the wrong way about configuring git globally just so that Go dependencies can be downloaded. But doesn't this have that same problem? You're configuring a tool used by your entire system (in this case, if I understand .netrc properly, the way your machine performs every HTTP request) just so that Go dependencies can be downloaded. In fact, this way seems risky, since wouldn't it include that token with every request you send to that host?
    – Matt Welke
    Commented Feb 2, 2021 at 16:42
  • 1
    Thank you, For gitlab the following worked for me echo "machine gitlab.example.com login gitlab-ci-token password ${CI_JOB_TOKEN}" > ~/.netrc && chmod 600 ~/.netrc
    – spuder
    Commented Oct 26, 2021 at 19:33
  • 1
    Thanks! Little note for windows: %HOME%\_netrc file there.
    – Elena
    Commented Dec 1, 2022 at 7:01
  • Using netrc is a bad advice as your password is stored in clear. Use instead the git credentials helper for you password store. See git help credentials.
    – dolmen
    Commented Jun 28 at 14:00
28

All of the above did not work for me. Cloning the repo was working correctly but I was still getting an unrecognized import error.

As it stands for Go v1.13, I found in the doc that we should use the GOPRIVATE env variable like so:

$ GOPRIVATE=github.com/ORGANISATION_OR_USER_NAME go get -u github.com/ORGANISATION_OR_USER_NAME/REPO_NAME
1
  • 2
    That was so simple... made my day! Commented Oct 11, 2021 at 16:56
19

Generate a github oauth token here and export your github token as an environment variable:

export GITHUB_TOKEN=123

Set git config to use the basic auth url:

git config --global url."https://$GITHUB_TOKEN:[email protected]/".insteadOf "https://github.com/"

Now you can go get your private repo.

4
  • 1
    best answer for Dockerfile, thank you Miguel Mota Commented Feb 6, 2022 at 10:21
  • This works as of today. Other solutions does not provide option to provide token or username/token Commented Jun 18, 2022 at 18:39
  • Now your GitHub token is stored in clear text in ~/.gitconfig. Not a good security advice. Instead use the git credential helper adapted for your password store. See git help credentials.
    – dolmen
    Commented Jun 28 at 14:02
  • @dolmen it depends, git config --global credential.helper store would also store the token in plaintext in ~/.git-credentials
    – Miguel
    Commented Jul 1 at 19:24
15

If you've already got git using SSH, this answer by Ammar Bandukwala is a simple workaround:


$ go get uses git internally. The following one liners will make git and consequently $ go get clone your package via SSH.

Github:

$ git config --global url."[email protected]:".insteadOf "https://github.com/"

BitBucket:

$ git config --global url."[email protected]:".insteadOf "https://bitbucket.org/"
12

I came across .netrc and found it relevant to this.

Create a file ~/.netrc with the following content:

machine github.com
    login <github username>
    password <github password or Personal access tokens >

Done!

Additionally, for latest GO versions, you might need to add this to the environment variables GOPRIVATE=github.com/<orgname|username> (I've added it to my .zshrc)

netrc also makes my development environment setup better as my personal github access for HTTPS is been configured now to be used across the machine (just like my SSH configuration).

Generate GitHub personal access tokens: https://github.com/settings/tokens

See this answer for its use with Git on Windows specifically

Ref: netrc man page


If you want to stick with the SSH authentication, then mask the request to use ssh forcefully

git config --global url."[email protected]:<orgname|username>".insteadOf "https://github.com/<orgname|username>"

Replace <orgname|username> with your GitHub org name or username.


Issue is that, go tries to access the dependencies using https protocol and lot of us use ssh instead to authenticate on git. So we either need to setup a https authentication using ~/.netrc or force using ssh on all request to a specific git location using git config

More methods for setting up git access: https://gist.github.com/technoweenie/1072829#gistcomment-2979908

4
  • 1
    This requires you to store your login credentials in a plain text file on your machine. That's not good.
    – Joachim
    Commented Aug 19, 2020 at 8:25
  • You are probably right. For me, it looks similar to how AWS credentials are stored in ~/.aws. And with GitHub's personal access token, I can fine grain the access according to requirement and control rotations. There's SSH way as well 👆 Commented Aug 20, 2020 at 9:58
  • Storing credentials in ~/.aws is bad as well. That's a bad justification for incorrect security practices. Instead use the Git credentials helper appropriate for your password store. See git help credentials.
    – dolmen
    Commented Jun 28 at 14:06
  • It's all about tradeoffs, I didn't say one is better than the other. There are two default ways of doing this - SSH public-private key pair & GitHub PAT. And I've shown both. Github is also providing fine-grained access token which can be used instead of the password (as mentioned). Storing these fine-grain access tokens into netrc is undeniably a viable option for many. Storing the web login password in a file would defiantly be bad and I didn't suggest that I believe, at-least for GitHub & AWS. Commented Jul 12 at 9:32
10

After trying multiple solutions my problem still persisted. The final solution after setting up the ~/.netrc and SSH config file, was to add the following line to my ~/.bash_profile

export GOPRIVATE="github.com/[organization]"

1
  • Using netrc is a bad advice as your password is stored in clear. Use instead the git credentials helper for you password store. See git help credentials.
    – dolmen
    Commented Jun 28 at 14:07
10

2023: .git is no longer required on GitHub repos access over ssh.
The same should apply to GitLab.


2014: That looks like the GitLab issue 5769.

In GitLab, since the repositories always end in .git, I must specify .git at the end of the repository name to make it work, for example:

import "example.org/myuser/mygorepo.git"

And:

$ go get example.org/myuser/mygorepo.git

Looks like GitHub solves this by appending ".git".

It is supposed to be resolved in “Added support for Go's repository retrieval. #5958”, provided the right meta tags are in place.
Although there is still an issue for Go itself: “cmd/go: go get cannot discover meta tag in HTML5 documents”.

5
  • Actually they (and others) has already supported meta tags, but it just works for public repo (where go get can see meta tags without login)
    – secmask
    Commented Dec 16, 2014 at 10:17
  • @secmask which is why you are using ssh: to provide the credentials that way.
    – VonC
    Commented Dec 16, 2014 at 10:23
  • oh, that just option, I can use http,https to but go get command will look like go get gitlab.com/secmask/awserver-go.git.git (will ask http basic authenticate), which look not good too.
    – secmask
    Commented Dec 16, 2014 at 10:30
  • I suspect this answer is out of date now, as .git is no longer required on github repos access over ssh
    – erik258
    Commented Sep 25, 2023 at 14:58
  • @erik258 True, and the same should apply for GitLab, I presume.
    – VonC
    Commented Sep 25, 2023 at 18:39
6

I have created a user specific ssh-config, so my user automatically logs in with the correct credentials and key.

First I needed to generate an key-pair

ssh-keygen -t rsa -b 4096 -C "[email protected]"

and saved it to e.g ~/.ssh/id_my_domain. Note that this is also the keypair (private and public) I've connected to my Github account, so mine is stored in~/.ssh/id_github_com.

I have then created (or altered) a file called ~/.ssh/config with an entry:

Host github.com
    HostName github.com
    User git
    IdentityFile ~/.ssh/id_github_com

On another server, the "ssh-url" is [email protected]:username/private-repo.git and the entry for this server would have been:

Host domain.com
    HostName domain.com
    User admin
    IdentityFile ~/.ssh/id_domain_com

Just to clarify that you need ensure that the User, Host and HostName is set correctly.

Now I can just browse into the go path and then go get <package>, e.g go get main where the file main/main.go includes the package (from last example above) domain.com:username/private-repo.git.

3
  • You can also import a package directly with: go get hostname.com/username/repo.git (.git extension is crucial).
    – Joachim
    Commented Apr 11, 2017 at 11:05
  • is there any way to do it without the extension .git and why happen it?
    – Cristian
    Commented Apr 25, 2017 at 16:19
  • 1
    Interesting, the reason cause I don't be able to get the repo, without the .git extension it was that on my local git configuration, git config --global url."[email protected]:".insteadOf "https://gitlab.myserver.com/" , but the subdomain gitlab.myserver.com doesn't have the ssl certification, so I use http instead of https , by now
    – Cristian
    Commented Apr 25, 2017 at 19:13
4

For me, the solutions offered by others still gave the following error during go get

[email protected]: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

What this solution required

  1. As stated by others:

    git config --global url."[email protected]:".insteadOf "https://github.com/"

  2. Removing the passphrase from my ./ssh/id_rsa key which was used for authenticating the connection to the repository. This can be done by entering an empty password when prompted as a response to:

    ssh-keygen -p

Why this works

This is not a pretty workaround as it is always better to have a passphrase on your private key, but it was causing issues somewhere inside OpenSSH.

go get uses internally git, which uses openssh to open the connection. OpenSSH takes the certs necessary for authentication from .ssh/id_rsa. When executing git commands from the command line an agent can take care of opening the id_rsa file for you so that you do not have to specify the passphrase every time, but when executed in the belly of go get, this did not work somewhy in my case. OpenSSH wants to prompt you then for a password but since it is not possible due to how it was called, it prints to its debug log:

read_passphrase: can't open /dev/tty: No such device or address

And just fails. If you remove the passphrase from the key file, OpenSSH will get to your key without that prompt and it works

This might be caused by Go fetching modules concurrently and opening multiple SSH connections to Github at the same time (as described in this article). This is somewhat supported by the fact that OpenSSH debug log showed the initial connection to the repository succeed, but later tried it again for some reason and this time opted to ask for a passphrase.

However the solution of using SSH connection multiplexing as put forward in the mentioned article did not work for me. For the record, the author suggested adding the collowing conf to the ssh config file for the affected host:

  ControlMaster auto
  ControlPersist 3600
  ControlPath ~/.ssh/%r@%h:%p

But as stated, for me it did not work, maybe I did it wrong

3
  • I think you can get this to work without removing the passphrase on your SSH key by using ssh-agent before running go-get to pre-authenticate your ssh-key. Commented Jun 7, 2021 at 21:17
  • In my case using an agent beforehand did not work unfortunately
    – Tarmo
    Commented Jun 8, 2021 at 6:29
  • To ease SSH setup for GitHub, you could use my tool github-keygen.
    – dolmen
    Commented Jun 28 at 14:10
4

After setting up GOPRIVATE and git config ...

People may still meeting problems like this when fetching private source:

https fetch: Get "https://private/user/repo?go-get=1": EOF

They can't use private repo without .git extension.

The reason is the go tool has no idea about the VCS protocol of this repo, git or svn or any other, unlike github.com or golang.org them are hardcoded into go's source.

Then the go tool will do a https query before fetching your private repo:

https://private/user/repo?go-get=1

If your private repo has no support for https request, please use replace to tell it directly :

require private/user/repo v1.0.0

...

replace private/user/repo => private.server/user/repo.git v1.0.0

https://golang.org/cmd/go/#hdr-Remote_import_paths

3

first I tried

[url "ssh://[email protected]/"]
    insteadOf = https://github.com/

but it didn't worked for my local.

I tried

ssh -t [email protected]

and it shows my ssh is fine.

finally, I fix the problem to tell the go get to consider all as private and use ssh instead of HTTPS.

adding export GOPRIVATE=*

1
  • To ease SSH setup for GitHub, you could use my tool github-keygen.
    – dolmen
    Commented Jun 28 at 14:11
2

I have tried all the existing answers with GitLab Enterprise Edition 15.11.11-ee and none worked.

With our instance of Gitlab I found go get only works if your repository (repo) is not in a subgroup

Imagine the domain name is https://gitlab.company-name.com

If the path of the repo URL is /repository or group/repository.

go get will work.

However, if you have a subgroup i.e the repo URL path is group/subgroup/repository go get will not work at all.

2
  • I got same problem now, it used to be working for a while. I didn't upgrade my gitlab instance, so I can be sure, it is not the gitlab. Latest working go version is also the one i am currently use. The only thing might be changes, I try to update my system, it failed in the middle, something might be updated (git, network etc). so now netrc, gitconfig nothing works anymore. the go get always try first group and then exit because eventually the subgroup is not a git repository. Do you find any solution? Commented Jun 10 at 11:52
  • 1
    this costs me a day to figure out why. for those who have the same problem, this is indeed gitlab changes. but, I can't say why this was working before. might be related to cached modules. who knows. if you use netrc with private access token, regenerate new one with additional right "api". this is required, so that the subgroups are visible. more context here. stackoverflow.com/a/57099566/3392555 and for the issue (might not be fixed, as I read. it was intended) gitlab.com/gitlab-org/gitlab-foss/-/issues/30785 Commented Jun 10 at 12:14
1

Make sure you remove your previous gitconfigs, I had the same issue.

Previously I executed gitconfig whose token was expired, when you execute the command next time with new token make sure to delete previous one.

0

For standalone/final repos, an as a quick fix, why don't just to name the module within the go.mod as a package using your company's domain ... ?

module go.yourcompany.tld/the_repo

go.yourcompany.tld don't even have to exist as a valid (sub)domain...

Also, in the same go.mod you can use replacement block/lines to use private repos previously cloned the same way (within a respective folder cloned also in $GOPATH/src/go.yourcompany.tld) (why do we have to depend too much in GitHub?)

Edit

  1. Needless to say that a private repo usually shall be a private repo, typically a standard git repo, right? With that, why not just to git clone and then go get within the cloned folder?
0

Run followings in the terminal

export GOPRIVATE=*

Then make sure to use ssh instead of https and use the protocol ssh://

Either by running followings in the terminal:

$ git config --global url."ssh://[email protected]".insteadOf "https://github.com"

or by adding followings to the .gitconfig file in $HOME folder:

[url "ssh://[email protected]"]
    insteadOf = https://github.com
-2

It's Hard Code In Go Get. Not The Git Reason. So Modify Go Source.
Reason:
repoRootForImportDynamic Will Request: https://....go-get

// RepoRootForImportPath analyzes importPath to determine the
// version control system, and code repository to use.
func RepoRootForImportPath(importPath string, mod ModuleMode, security web.SecurityMode) (*RepoRoot, error) {
    rr, err := repoRootFromVCSPaths(importPath, security, vcsPaths)
    if err == errUnknownSite {
        rr, err = repoRootForImportDynamic(importPath, mod, security)
        if err != nil {
            err = importErrorf(importPath, "unrecognized import path %q: %v", importPath, err)
        }
    }
    if err != nil {
        rr1, err1 := repoRootFromVCSPaths(importPath, security, vcsPathsAfterDynamic)
        if err1 == nil {
            rr = rr1
            err = nil
        }
    }

So add gitlab domain to vcsPaths will ok.
Download go source code:

vi ./src/cmd/go/internal/vcs/vcs.go    

Look for code below:

var vcsPaths = []*vcsPath{
    // GitHub
    {
        pathPrefix: "github.com",
        regexp:     lazyregexp.New(`^(?P<root>github\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/[A-Za-z0-9_.\-]+)*$`),
        vcs:        "git",
        repo:       "https://{root}",
        check:      noVCSSuffix,
    },

add Code As Follow,XXXX Is Your Domain:

    // GitLab
    {
        pathPrefix: "gitlab.xxxx.com",
        regexp:     lazyregexp.New(`^(?P<root>gitlab.xxxx\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/[A-Za-z0-9_.\-]+)*$`),
        vcs:        "git",
        repo:       "https://{root}",
        check:      noVCSSuffix,
    },

compile and replace go.

Not the answer you're looking for? Browse other questions tagged or ask your own question.