37

I'm trying to debug the following build error in our CI where "A depends on B which can't build because it depends on C." I'm building my data service which doesn't directly depend on kafkaAvailMonitor.go which makes this error hard to trace. In other words:

data (what I'm building) depends on (?) which depends on kafkaAvailMonitor.go

It may seem trivial to fix for a developer they just do "go get whatever" but I can't do that as part of the release process - I have to find the person that added the dependency and ask them to fix it.

I'm aware that there are tools to visualize the dependency tree and other more sophisticated build systems, but this seems like a pretty basic issue: is there any way I can view the full dependency tree to see what's causing the build issue?

go build -a -v

../../../msgq/kafkaAvailMonitor.go:8:2: cannot find package 
  "github.com/Shopify/sarama/tz/breaker" in any of:
  /usr/lib/go-1.6/src/github.com/Shopify/sarama/tz/breaker (from $GOROOT)
  /home/jenkins/go/src/github.com/Shopify/sarama/tz/breaker (from $GOPATH)
  /home/jenkins/vendor-library/src/github.com/Shopify/sarama/tz/breaker
  /home/jenkins/go/src/github.com/Shopify/sarama/tz/breaker
  /home/jenkins/vendor-library/src/github.com/Shopify/sarama/tz/breaker
4
  • 2
    Or are you looking for go list ? Take a look at Imports and Deps.
    – Volker
    Jun 13, 2017 at 12:30
  • 2
    I'm not sure why this is diverging into a discussion about the technical meaning of stack trace - this always happens to me on SO. I have a CI job building dozen of services (that I didn't write) and this is what I see in the logs but that doesn't help me. I have no idea who is importing kafkaAvailMonitor.go i.e. who is breaking the build. go list was a helpful suggestion but didn't show anything - would love to know more about that.
    – Charlie
    Jun 13, 2017 at 16:08
  • 1
    go list -f '{{ .GoFiles }}' github.com/Shopify/sarama/tz/breaker
    – RickyA
    Jun 27, 2017 at 13:44
  • 1
    The OP isn't looking for a stack trace, he wants the dependency tree. Jul 1, 2017 at 15:24

5 Answers 5

48

When using modules you may be able to get what you need from go mod graph.

usage: go mod graph

Graph prints the module requirement graph (with replacements applied)
in text form. Each line in the output has two space-separated fields: a module
and one of its requirements. Each module is identified as a string of the form
path@version, except for the main module, which has no @version suffix.

I.e., for the original question, run go mod graph | grep github.com/Shopify/sarama then look more closely at each entry on the left-hand side.

23
+25

if the following isn't a stack trace what is it?

It is the list of path where Go is looking for your missing package.

I have no idea who is importing kafkaAvailMonitor.go

It is not "imported", just part of your sources and compiled.
Except it cannot compile, because it needs github.com/Shopify/sarama/tz/breaker, which is not in GOROOT or GOPATH.

Still, check what go list would return on your direct package, to see if kafkaAvailMonitor is mentioned.

go list can show both the packages that your package directly depends, or its complete set of transitive dependencies.

% go list -f '{{ .Imports }}' github.com/davecheney/profile
[io/ioutil log os os/signal path/filepath runtime runtime/pprof]
% go list -f '{{ .Deps }}' github.com/davecheney/profile
[bufio bytes errors fmt io io/ioutil log math os os/signal path/filepath reflect run

You can then script go list in order to list all dependencies.
See this bash script for instance, by Noel Cower (nilium)

#!/usr/bin/env bash
# Usage: lsdep [PACKAGE...]
#
# Example (list github.com/foo/bar and package dir deps [the . argument])
# $ lsdep github.com/foo/bar .
#
# By default, this will list dependencies (imports), test imports, and test
# dependencies (imports made by test imports).  You can recurse further by
# setting TESTIMPORTS to an integer greater than one, or to skip test
# dependencies, set TESTIMPORTS to 0 or a negative integer.

: "${TESTIMPORTS:=1}"

lsdep_impl__ () {
    local txtestimps='{{range $v := .TestImports}}{{print . "\n"}}{{end}}'
    local txdeps='{{range $v := .Deps}}{{print . "\n"}}{{end}}'

    {
        go list -f "${txtestimps}${txdeps}" "$@"
        if [[ -n "${TESTIMPORTS}" ]] && [[ "${TESTIMPORTS:-1}" -gt 0 ]]
        then
            go list -f "${txtestimps}" "$@" |
            sort | uniq |
            comm -23 - <(go list std | sort) |
                TESTIMPORTS=$((TESTIMPORTS - 1)) xargs bash -c 'lsdep_impl__ "$@"' "$0"
        fi
    } |
    sort | uniq |
    comm -23 - <(go list std | sort)
}
export -f lsdep_impl__

lsdep_impl__ "$@"
5
  • Are you familiar with the build "output" of other language like Java/C#/Python etc? Where you build something and it tells you the dependency chain e.g. A import B at line 3 which imports C at line 2? The issue with your answer is that I have to run "go list" on every dependency so if A depends on B depends on C etc. I have go all the way down each one which on a large project is impossible. The whole point is that I don't want to add tz/breaker - someone is adding it by mistake but I can't figure out who because it's buried under several other dependencies.
    – Charlie
    Jun 26, 2017 at 22:14
  • 1
    "The issue with your answer is that I have to run "go list" on every dependency so if A depends on B depends on C etc": that is easily scriptable, I have edited the answer accordingly.
    – VonC
    Jun 27, 2017 at 4:34
  • 1
    "The whole point is that I don't want to add tz/breaker - someone is adding it by mistake". Someone? That "someone" is kafkaAvailMonitor.go, line 8. The question becomes: do you need kafkaAvailMonitor.go
    – VonC
    Jun 27, 2017 at 4:35
  • 1
    I think the point is: the OP doesn't know what is depending on kafkaAvailMonitor.go. Jul 1, 2017 at 15:27
  • 2
    @Fli hence the script, in order to list all dependencies on all packages, in order to find the one importing kafkaAvailMonitor
    – VonC
    Jul 1, 2017 at 17:10
9

I just want to mention here that go mod why can also help. Anyway you cannot get and display the whole tree. But you can trace back one single branch of a child dependency until its parent root.

Example:

$ go mod why github.com/childdep
# github.com/childdep
github.com/arepo.git/service
github.com/arepo.git/service.test
github.com/anotherrepo.git/mocks
github.com/childdep

That means, you have imported 'childdep' finally in 'anotherrepo.git/mocks'.

8

can try this https://github.com/vc60er/deptree

 redis git:(master) go mod graph | deptree -d 3
package: github.com/go-redis/redis/v9
dependence tree:

┌── github.com/cespare/xxhash/[email protected]
├── github.com/dgryski/[email protected]
├── github.com/fsnotify/[email protected]
│    └── golang.org/x/[email protected]
├── github.com/nxadm/[email protected]
│    ├── github.com/fsnotify/[email protected]
│    │    └── golang.org/x/[email protected]
│    └── gopkg.in/[email protected]
├── github.com/onsi/[email protected]
│    ├── github.com/go-task/[email protected]
│    │    ├── github.com/davecgh/[email protected]
│    │    └── github.com/stretchr/[email protected]
│    │         └── ...
1
  • Excellent, Go built-in feature :)
    – nitram509
    Nov 22, 2022 at 22:06
2

The above answer still doesn't show me a dependency tree so I've taken the time to write a Python script to do what I need - hopefully that helps other people.

The issue with the above solution (the others proposed like go list) is that it only tells me the top level. They don't "traverse the tree." This is the output I get - which doesn't help any more than what go build gives me.

.../npd/auth/
   .../mon/mlog
   .../auth/service

This is what I'm trying to get - I know that auth is broken (top) and that breaker is broken (bottom) from go build but I have no idea what's in between - my script below gives me this output.

.../npd/auth/
    .../npd/auth/service                
        .../npd/auth/resource
            .../npd/auth/storage
               .../npd/middleware
                  .../npd/metrics/persist
                    .../npd/kafka
                        .../vendor-library/src/github.com/Shopify/sarama
                            .../vendor-library/src/github.com/Shopify/sarama/vz/breaker

My Python script:

import subprocess
import os

folder_locations=['.../go/src','.../vendor-library/src']

def getImports(_cwd):
    #When the commands were combined they overflowed the bugger and I couldn't find a workaround
    cmd1 = ["go", "list", "-f", " {{.ImportPath}}","./..."]
    cmd2 = ["go", "list", "-f", " {{.Imports}}","./..."]

    process = subprocess.Popen(' '.join(cmd1), cwd=_cwd,shell=True,
                               stdout=subprocess.PIPE,
                               stderr=subprocess.PIPE)

    out1, err = process.communicate()

    process = subprocess.Popen(' '.join(cmd2), cwd=_cwd,shell=True,
                               stdout=subprocess.PIPE,
                               stderr=subprocess.PIPE)

    out2, err = process.communicate()
    out2clean=str(out2).replace("b'",'').replace('[','').replace(']','').replace("'",'')

    return str(out1).split('\\n'),out2clean.split('\\n')

def getFullPath(rel_path):
    for i in folder_locations:
        if os.path.exists(i+'/'+rel_path):
            return i+'/'+rel_path
    return None

def getNextImports(start,depth):

    depth=depth+1
    indent = '\t'*(depth+1)

    for i,val in enumerate(start.keys()):

        if depth==1:
            print (val)

        out1,out2=getImports(val)

        noDeps=True
        for j in out2[i].split(' '):
            noDeps=False
    
            _cwd2=getFullPath(j)
            new_tree = {_cwd2:[]}
            not_exists = (not _cwd2 in alltmp)
    
            if not_exists:
                print(indent+_cwd2)
                start[val].append(new_tree)
                getNextImports(new_tree,depth)
                alltmp.append(_cwd2)

        if noDeps:
            print(indent+'No deps')

_cwd = '/Users/.../npd/auth'

alltmp=[]
start_root={_cwd:[]}
getNextImports(start_root,0)
2
  • 1
    Yes - the the real disappointment is that go list has a recursive option but it flattens the tree - so you can see all the dependencies but not the order. go list -f "{{.ImportPath}} {{.Deps}}" ./...
    – Charlie
    Jul 7, 2017 at 12:41
  • not required, since there's a Go built-in feature (see answer from @Gordon). but for the record: good job on the Pythonscript
    – nitram509
    Nov 22, 2022 at 22:08

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.