1

I cannot extract all of the contents of a gzipped vcf file. The file is part of an encrypted tarball I downloaded from dbGaP. After decryption, I was able to extract a directory of files with this command:

   tar -xvf phg001.tar

When I used Plink to convert one of the extracted vcf files to a bed file, I got an error message: Error: Line 20 of .vcf file has fewer tokens than expected.

I counted the number of lines in the files with the help of zcat.

    zcat chr22-filtered.dose.vcf.gz | wc -l

Output:

    gzip: chr22-filtered.dose.vcf.gz: decompression OK, trailing garbage ignored
    19

And if I try to unzip the file, I get a similar message about trailing garbage.

gzip: test22.vcf.gz: decompression OK, trailing garbage ignored

The file is too large to have only 20 lines, and if I count the number of lines without using zcat, there is indeed more to the file.

    wc -l chr22-filtered.dose.vcf.gz
    3632730 chr22-filtered.dose.vcf.gz

How can I extract all of the contents of the zipped file.

All advice is appreciated.
Paul

Updated: Here are the results from pigz:

pigz -ltv chr22-filtered.dose.vcf.gz
method    check    timestamp    compressed   original reduced  name
gzip 8  3388a535  ------ -----       17965      65280   72.5%  chr22-filtere...
gzip 8  50734e24  ------ -----        1356      39654   96.6%  <...>
pigz: warning: chr22-filtered.dose.vcf.gz: trailing junk was ignored

ls-vcf.c1$ pigz -ltv chr13-filtered.dose.vcf.gz
method    check    timestamp    compressed   original reduced  name
gzip 8  15c3d4c1  ------ -----       18431      65280   71.8%  chr13-filtere...
gzip 8  e17a4674  ------ -----        3221      39654   91.9%  <...>
pigz: warning: chr13-filtered.dose.vcf.gz: trailing junk was ignored

pigz -ltv chr2-filtered.dose.vcf.gz
method    check    timestamp    compressed   original reduced  name
gzip 8  f4cfe9f0  ------ -----       18245      65280   72.1%  chr2-filtered...
gzip 8  e8370ff3  ------ -----        2345      39654   94.1%  <...>
pigz: warning: chr2-filtered.dose.vcf.gz: trailing junk was ignored

pigz -ltv chr21-filtered.dose.vcf.gz
method    check    timestamp    compressed   original reduced  name
gzip 8  967d168f  ------ -----       19756      65280   69.7%  chr21-filtere...
gzip 8  9e27cda1  ------ -----        8588      39654   78.3%  <...>
pigz: warning: chr21-filtered.dose.vcf.gz: trailing junk was ignored

The output for all the files is similar. And all the files have the same error when I try to extract with gunzip.

6
  • Counting "lines" in a compressed file has no meaning. You will encounter new line characters, on average, every 256 bytes. So if you take the length and divide by 256, you will get approximately the same answer.
    – Mark Adler
    Apr 26 at 21:57
  • Can you provide a link to an example .gz file?
    – Mark Adler
    Apr 26 at 21:58
  • @MarkAdler Thanks for the correction on counting lines of a compressed file. Unfortunately, I cannot link to one of the files since they contain protected health information. But there 22 of these .gz files that vary in size from 2G to 10G, and they all only show 20 lines when viewed with zcat.
    – PaulJ2.0
    Apr 26 at 22:15
  • If you download and compile pigz, you can use it to get a more detailed view of the .gz file, using -ltv. Try that on your examples, and add the results to your question.
    – Mark Adler
    Apr 26 at 22:29
  • Do your other examples show something similar?
    – Mark Adler
    Apr 26 at 23:39

1 Answer 1

0

I'm guessing that chr22-filtered.dose.vcf.gz is a significant fraction of a gigabyte, but pigz only managed to find good compressed data in the first 19 K! The file is corrupted somehow, right at the header of third gzip member.

You should try to download the file again. If it looks exactly the same, then contact whoever made it and let them know there was an issue in the generation of those files.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.