52

I cannot get wget to mirror a section of a website (a folder path below root) - it only seems to work from the website homepage.

I've tried many options - here is one example

wget -rkp -l3 -np  http://somewebsite/subpath/down/here/

While I only want to mirror the content links below that URL - I also need to download all the page assets which are not in that path.

It seems to work fine for the homepage (/) but I can't get it going for any sub folders.

5 Answers 5

89

Use the --mirror (-m) and --no-parent (-np) options, plus a few of cool ones, like in this example:

wget --mirror --page-requisites --adjust-extension --no-parent --convert-links
     --directory-prefix=sousers http://stackoverflow.com/users
2
  • 25
    To save anyone else searching the wget manual, -p is --page-requisites and -P is --directory-prefix
    – Alf Eaton
    Oct 3, 2012 at 10:11
  • 4
    Just as a note for others who might bump into this issue where the most commonly downloaded wget binary for Windows 7 seems to be gnuwin32 packages from sourceforge.net, but those are wget-1.11 which do not have the --adjust-extension functionality. It was apparently added only in wget-1.12. So Windows 7 users can get a much newer and self-contained binary from here instead (eternallybored.org/misc/wget)
    – bdutta74
    Mar 5, 2014 at 6:56
19

I usually use:

wget -m -np -p $url
2
  • 2
    -p to download everything required to display a page is useful. Does that override -np for those elements necessary to display a page?
    – Geremia
    Feb 11, 2016 at 16:46
  • For informational purposes only: -m = mirror, -np = no parent (don't retrieve files higher up in the hierarchy when recursing), -p = page requisites or all items necessary to appropriately display the web page.
    – Shrout1
    Mar 16, 2017 at 20:07
3

I use pavuk to accomplish mirrors, as it seemed much better for this purpose just from the beginning. You can use something like this:

/usr/bin/pavuk -enable_js -fnrules F '*.php?*' '%o.php' -tr_str_str '?' '_questionmark_' \
               -norobots -dont_limit_inlines -dont_leave_dir \
               http://www.example.com/some_directory/ >OUT 2>ERR
2

For my use case the no parent option didn't quite work.

I was trying to mirror https://www.example.com/section and URLs under it like https://www.example.com/section/subsection. This can't be done with --no-parent because if you start at /section then it'll download the entire site if you start at /section/ then the site redirects to /section and now it's at parent so wget stops. Fun.

Instead, I am using --acept-regex 'https://www.example.com/(section|assets/).*'. This worked. (Although it would download sectionfoobar but this was acceptable for me and now we are wandering into regexp territory which is amply covered elsewhere on SO.)

0

Check out archivebox.io, it's an open-source, self-hosted tool that creates a local, static, browsable HTML clone of websites (it saves HTML, JS, media files, PDFs, screenshot, static assets and more).

By default, it only archives the URL you specify, but we're adding a --depth=n flag soon that will let you recursively archive links from the given URL.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.