11

So I'm trying to crate a tar.gz file file from multiple directories and files. Something with the same usage as:

tar -cvzf sometarfile.tar.gz somedir/ someotherdir/ somefile.json somefile.xml

Assuming the directories have other directories inside of them. I have this as an input:

    paths := []string{
      "somedir/",
      "someotherdir/",
      "somefile.json",
      "somefile.xml",
    }

and using these:

    func TarFilesDirs(paths []string, tarFilePath string ) error {
       // set up the output file
       file, err := os.Create(tarFilePath)
       if err != nil {
           return err
       }

       defer file.Close()
       // set up the gzip writer
       gz := gzip.NewWriter(file)
       defer gz.Close()

       tw := tar.NewWriter(gz)
       defer tw.Close()

       // add each file/dir as needed into the current tar archive
       for _,i := range paths {
          if err := tarit(i, tw); err != nil {
               return err
          }
       }

       return nil
   }

func tarit(source string, tw *tar.Writer) error {
    info, err := os.Stat(source)
    if err != nil {
        return nil
    }

    var baseDir string
    if info.IsDir() {
        baseDir = filepath.Base(source)
    }

    return filepath.Walk(source,
        func(path string, info os.FileInfo, err error) error {
            if err != nil {
                return err
            }

            header, err := tar.FileInfoHeader(info, info.Name())
            if err != nil {
                return err
            }

            if baseDir != "" {
                header.Name = filepath.Join(baseDir, strings.TrimPrefix(path, source))
            }

            if err := tw.WriteHeader(header); err != nil {
                return err
            }

            if info.IsDir() {
                return nil
            }

            file, err := os.Open(path)
            if err != nil {
                return err
            }

            defer file.Close()

            _, err = io.Copy(tw, file)
            if err != nil {
                log.Println("failing here")
                return err
            }

            return err
        })
}

Problem: if a directory is large I'm getting:

archive/tar: write too long

error, when I remove it everything works.

Ran out of ideas and wasted many hours on this trying to find a solution...

Any ideas?

Thanks

3 Answers 3

15

I was having a similar issue until I looked more closely at the tar.FileInfoHeader doc:

FileInfoHeader creates a partially-populated Header from fi. If fi describes a symlink, FileInfoHeader records link as the link target. If fi describes a directory, a slash is appended to the name. Because os.FileInfo's Name method returns only the base name of the file it describes, it may be necessary to modify the Name field of the returned header to provide the full path name of the file.

Essentially, FileInfoHeader isn't guaranteed to fill out all the header fields before you write it with WriteHeader, and if you look at the implementation the Size field is only set on regular files. Your code snippet seems to only handle directories, this means if you come across any other non regular file, you write the header with a size of zero then attempt to copy a potentially non-zero sized special file on disk into the tar. Go returns ErrWriteTooLong to stop you from creating a broken tar.

I came up with this and haven't had the issue since.

    if err := filepath.Walk(directory, func(path string, info os.FileInfo, err error) error {
        if err != nil {
            return check(err)
        }

        var link string
        if info.Mode()&os.ModeSymlink == os.ModeSymlink {
            if link, err = os.Readlink(path); err != nil {
                return check(err)
            }
        }

        header, err := tar.FileInfoHeader(info, link)
        if err != nil {
            return check(err)
        }

        header.Name = filepath.Join(baseDir, strings.TrimPrefix(path, directory))
        if err = tw.WriteHeader(header); err != nil {
            return check(err)
        }

        if !info.Mode().IsRegular() { //nothing more to do for non-regular
            return nil
        }

        fh, err := os.Open(path)
        if err != nil {
            return check(err)
        }
        defer fh.Close()

        if _, err = io.CopyBuffer(tw, fh, buf); err != nil {
            return check(err)
        }
        return nil
})
2
  • 1
    It is also possible that the file has changed on disk and has became longer.
    – dpington
    Commented Apr 9, 2017 at 14:02
  • 1
    Heads up on this one - some filesystems don't accurate list file size in attrs, such as /proc. I've seen this happen when a rogue archive tries to include /proc/PID/attr/current which is SELinux context. The file attr will always report size zero, but a read finds data. In GoLang it invariably generates a "write too long" failure, whereas GNU tar/libz will have no problem. Be careful what filesystem you're using to archive and whether it accurately reports lengths.
    – BoeroBoy
    Commented Dec 17, 2018 at 17:36
2

Since you are seeing this issue with a large directory only, I think the following fix might not help, but this will address the issue of creating a tar from files that might be continuously growing.

In my case the issue was that when we were creating the tar header, the header.Size (inside tar.FileInfoHeader) was getting setting to the file size (info.Size()) at that instant of time.

When we later in the code, try to open the concerned file (os.Open) and copy its contents (io.Copy) we risk copying more data than what we earlier set the tar header size to because the file could have grown in the meantime.

This piece of code will ensure we ONLY copy that much data as we set the tar header size to:

_, err = io.**CopyN**(tw, file, info.Size())
if err != nil {
    log.Println("failing here")
    return err
}
0

Write writes to the current entry in the tar archive. Write returns the error ErrWriteTooLong if more than hdr.Size bytes are written after WriteHeader.

There is a Size option you can add to the Header. Haven't tried it but maybe that helps...

See also https://golang.org/pkg/archive/tar/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.