24

I'm trying to retrieve working webpages with wget and this goes well for most sites with the following command:

wget -p -k http://www.example.com

In these cases I will end up with index.html and the needed CSS/JS etc.

HOWEVER, in certain situations the url will have a query string and in those cases I get an index.html with the query string appended.

Example

www.onlinetechvision.com/?p=566

Combined with the above wget command will result in:

index.html?page=566

I have tried using the --restrict-file-names=windows option, but that only gets me to

index.html@page=566

Can anyone explain why this is needed and how I can end up with a regular index.html file?

UPDATE: I'm sort of on the fence on taking a different approach. I found out I can take the first filename that wget saves by parsing the output. So the name that appears after Saving to: is the one I need.

However, this is wrapped by this strange character â - rather than just removing that hardcoded - where does this come from?

1
  • Have you tried that syntax already: curl http://{site,host}.host[1-5].com -o "#1_#2". It might be possible to adjust this to your needs.
    – Marged
    Nov 5, 2015 at 0:04

5 Answers 5

7
+50

If you try with parameter "--adjust-extension"

wget -p -k --adjust-extension  www.onlinetechvision.com/?p=566 

you come closer. In www.onlinetechvision.com folder there will be file with corrected extension: index.html@p=566.html or index.html?p=566.html on *NiX systems. It is simple now to change that file to index.html even with script.

If you are on Microsoft OS make sure you have latter version of wget - it is also available here: https://eternallybored.org/misc/wget/

1
  • I ended up doing this, but I don't really wholeheartedly like this. It feels a bit hacky but it works for now. I combined it with a directory scan after the wget so that I can find the .html page and rename it or use it. Nov 6, 2015 at 11:36
1

To answer your question about why this is needed, remember that the web server is likely to return different results based on the parameters in the query string. If a query for index.html?page=52 returns different results from index.html?page=53, you probably wouldn't want both pages to be saved in the same file.

Each HTTP request that uses a different set of query parameters is quite literally a request for a distinct resource. wget can't predict which of these changes is and isn't going to be significant, so it's doing the conservative thing and preserving the query parameter URLs in the filename of the local document.

0

My solution is to do recursive crawling outside wget:

  1. get directory structure with wget (no file)
  2. loop to get main entry file (index.html) from each dir

This works well with wordpress sites. Could miss some pages tho.

#!/bin/bash

#
# get directory structure
#
wget --spider -r --no-parent  http://<site>/

#
# loop through each dir
#
find . -mindepth 1 -maxdepth 10 -type d | cut -c 3- > ./dir_list.txt

while read line;do
        wget --wait=5  --tries=20 --page-requisites --html-extension --convert-links --execute=robots=off --domain=<domain> --strict-comments http://${line}/

done < ./dir_list.txt
1
  • 1
    the idea is to use -p -k to get a local working copy of the page, so including CSS and JavaScript - not to get a whole website. Plus - I need to know which main file is the one that I need to "view" - now I get index.html?page=xxx and sometimes just index.html and sometimes something crazy like Report123 Nov 2, 2015 at 10:18
0

The query string is required because of the website design what the site is doing is using the same standard index.html for all content and then using the querystring to pull in the content from another page like with script on the server side. (it may be client side if you look in the JavaScript).

Have you tried using --no-cookies it could be storing this information via cookie and pulling it when you hit the page. also this could be caused by URL rewrite logic which you will have little control over from the client side.

-2

use -O or --output-document options. see http://www.electrictoolbox.com/wget-save-different-filename/

1
  • 2
    that doesn't combine with the -k option though Nov 8, 2013 at 17:46

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.