24

I am trying to download the contents of a website using wget tool. I used -R option to reject some file types. but there are some other files which I don't want to download. These files are named as follows, and don't have any extensions.

string-ID

for example:

newsbrief-02

How I can tell wget not to download these files (the files which their names start with specified string)?

2 Answers 2

46

Since (apparently) v1.14 wget accepts regular expressions : --reject-regex and --accept-regex (with --regex-type posix by default, can be set to pcre if compiled with libpcre support).

Beware that it seems you can use --reject-regex only once per wget call. That is, you have to use | in a single regex if you want to select on several regex :

wget --reject-regex 'expr1|expr2|…' http://example.com
5
  • 5
    Thanks for the example with several regex.
    – GregHNZ
    May 1, 2014 at 8:38
  • 1
    Does reject-regex work with things like . or *, what kind of regex is it, extended regex or PCRE regex? Feb 16, 2017 at 6:17
  • 2
    @CMCDragonkai The second part of your question is already addressed in my answer. It does work with . and *, e.g. wget -r --reject-regex ".*foobar.*" http://example.com will discard http://example.com/foobar/. Feb 18, 2017 at 15:32
  • Note that it seems regex only works on recursive downloads, though. Feb 18, 2017 at 15:45
  • 1
    I used it with --spider, and it I had to escape [], so that means [] is also special. I only asked about what kind of regex it is because GNU grep also supports different kinds of regex, like extended regex and pcre regex. Feb 19, 2017 at 6:03
11

You can not specify a regular expression in the wget -R key, but you can specify a template (like file template in a shell).

The answer looks like:

$ wget -R 'newsbrief-*' ...

You can also use ? and symbol classes [].

For more information see info wget.

2
  • 1
    It may have been true at the date of the answer, but now You can use regular expressions to accept/reject patterns. See the other answer below. May 21, 2019 at 10:48
  • You can also reject multiple patterns at once with commas wget -R 'newsbrief-*,string-*,*.txt' Mar 26, 2020 at 9:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.