0

I'm using wget to read a batch of urls from an input file and download everything to a single output file, and I'd like to append each url before its downloaded content, anyone knows how to do that?

Thanks!

1
  • 1
    please show the code which you are using so far so we cann add the needed code
    – user753676
    Sep 11, 2013 at 16:04

1 Answer 1

0

afaik wget does not directly support the use case you are envisioning. however, using standard tools, you can emulate this feature.

we will proceed as follows:

  • call wget with logging enabled
  • let sed process the log executing the script detailed below
  • execute the transformation result as a shell/batch script

conventions: use the following filenames:

  • wgetin.txt: the file with the urls to fetch using wget
  • wgetout.sed: sed script
  • wgetout.final: the final result
  • wgetass.sh/.cmd: shell/batch script to assemble the downloaded files weaving in the url data
  • wget.log: the log file of the wget call

Linux

the sed script (linux):

# delete lines _not_ matching the regex
/^\(Saving to: .\|--[0-9: \-]\+--  \)/! { d; }

# turn remaining content into something else
s/^--[0-9: \-]\+--  \(.*\)$/echo '\1\n' >>wgetout.final/
s/^Saving to: .\(.*\).$/cat '\1' >>wgetout.final/

the command line (linux):

rm wgetout.final | rm wgetass.sh | wget -i wgetin.txt -o wget.log | sed -f wgetout.sed -r  wget.log >wgetass.sh | chmod 755 wgetass.sh | ./wgetass.sh

Windows

the syntax for windows batch scripts is slightly different. of course, the windows ports of wget and sed have to be installed first.

the sed script (windows):

# delete lines _not_ matching the regex
/^\(Saving to: .\|--[0-9: \-]\+--  \)/! { d; }

# turn remaining content into something else
s/^--[0-9: \-]\+--  \(.*\)$/echo "\1" >>wgetout.final/
s/^Saving to: .\(.*\).$/type "\1" >>wgetout.final/

the command line (windows):

del wgetout.final && del wgetass.cmd && wget -i wgetin.txt -o wget.log && sed -f wgetout.sed -r  wget.log >wgetass.cmd && wgetass.cmd

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.