130

Is it possible to find all the pages and links on ANY given website? I'd like to enter a URL and produce a directory tree of all links from that site?

I've looked at HTTrack but that downloads the whole site and I simply need the directory tree.

1
  • 2
    crawlmysite.in - site not exists Oct 20, 2015 at 7:40

5 Answers 5

88

Check out linkchecker—it will crawl the site (while obeying robots.txt) and generate a report. From there, you can script up a solution for creating the directory tree.

10
  • 2
    A nice tool. I was using "XENU link sleuth before". Linkchecker is far more verbose.
    – Mateng
    Nov 14, 2011 at 20:42
  • 1
    @MariusKavansky How do you manually crawl a website? Or how do you build a crawler? I'm not sure I understand your question. If there is no robots.txt file, that just means you can crawl to your heart's content.
    – Hank Gay
    Jul 31, 2013 at 15:14
  • 1
    Such a great little program! Nov 29, 2014 at 19:24
  • 8
    hi guys, linkchecker has not worked for me when I scan the site it only returns a report of broken links. Very small report. while it does they it checked thousands of links but I can't see where those are reported. Using version 9.3 can you please help?
    – Jay
    Nov 5, 2015 at 10:33
  • 3
    Usual command to write to file is linkchecker https://example.com --file-output=csv --verbose . Different formats can be chosen too.
    – laimison
    Jan 12, 2021 at 23:53
64

If you have the developer console (JavaScript) in your browser, you can type this code in:

urls = document.querySelectorAll('a'); for (url in urls) console.log(urls[url].href);

Shortened:

n=$$('a');for(u in n)console.log(n[u].href)
7
  • 1
    What about "Javascript-ed" urls?
    – Pacerier
    Feb 25, 2015 at 0:56
  • Like what? What do you mean?
    – ElectroBit
    Apr 3, 2015 at 20:53
  • 2
    I mean a link done using Javascript. Your solution wouldn't show it.
    – Pacerier
    Apr 6, 2015 at 13:45
  • 2
    @ElectroBit I really like it, but I'm not sure what I'm looking at? What is the $$ operator? Or is that just an arbitrary function name, same as n=ABC(''a'); I'm not understanding how urls gets all the 'a' tagged elements. Can you explain? I'm assuming its not jQuery. What prototype library function are we talking?
    – zipzit
    May 28, 2016 at 17:32
  • 1
    @zipzit In a handful of browsers, $$() is basically shorthand for document.querySelectorAll(). More info at this link: developer.mozilla.org/en-US/docs/Web/API/Document/…
    – ElectroBit
    May 28, 2016 at 17:54
6

Another alternative might be

Array.from(document.querySelectorAll("a")).map(x => x.href)

With your $$( its even shorter

Array.from($$("a")).map(x => x.href)
1
  • plus 1 - like that you are using modern JS. I ran this program, and while it returned a few links, it didn't return all of the .html pages that are on the top level. Is there a reason why all the pages don't return in the array list? Thanks
    – Chris22
    May 23, 2020 at 20:47
0

If this is a programming question, then I would suggest you write your own regular expression to parse all the retrieved contents. Target tags are IMG and A for standard HTML. For JAVA,

final String openingTags = "(<a [^>]*href=['\"]?|<img[^> ]* src=['\"]?)";

this along with Pattern and Matcher classes should detect the beginning of the tags. Add LINK tag if you also want CSS.

However, it is not as easy as you may have intially thought. Many web pages are not well-formed. Extracting all the links programmatically that human being can "recognize" is really difficult if you need to take into account all the irregular expressions.

Good luck!

1
-2
function getalllinks($url) {
    $links = array();
    if ($fp = fopen($url, 'r')) {
        $content = '';
        while ($line = fread($fp, 1024)) {
            $content. = $line;
        }
    }
    $textLen = strlen($content);
    if ($textLen > 10) {
        $startPos = 0;
        $valid = true;
        while ($valid) {
            $spos = strpos($content, '<a ', $startPos);
            if ($spos < $startPos) $valid = false;
            $spos = strpos($content, 'href', $spos);
            $spos = strpos($content, '"', $spos) + 1;
            $epos = strpos($content, '"', $spos);
            $startPos = $epos;
            $link = substr($content, $spos, $epos - $spos);
            if (strpos($link, 'http://') !== false) $links[] = $link;
        }
    }
    return $links;
}

try this code....

3
  • 10
    While this answer is probably correct and useful, it is preferred if you include some explanation along with it to explain how it helps to solve the problem. This becomes especially useful in the future, if there is a change (possibly unrelated) that causes it to stop working and users need to understand how it once worked. Mar 6, 2015 at 0:12
  • 2
    Eh, it's a little long.
    – ElectroBit
    May 3, 2015 at 18:29
  • 1
    Completely unnecessary to parse the html in this manner in php. php.net/manual/en/class.domdocument.php PHP does have the ability to understand the DOM!
    – JamesH
    Jun 26, 2015 at 12:30

Not the answer you're looking for? Browse other questions tagged or ask your own question.