8

All of the information I can find online is about writing web servers, but there seems to be very little about functions useful for web clients. Ideally, I would like the function to look something like this:

(website "http://www.google.com")

And return a string containing the entire web page, but I would be happy with anything that works.

1

1 Answer 1

10

Here's a simple program that looks like it does what you want:

#lang racket

(require net/url)

(port->bytes
 (get-pure-port (string->url "http://www.google.com")))

If you're like me, you probably also want to parse it into an s-expression. Neil Van Dyke's neil/html-parsing does this:

#lang racket

(require (planet neil/html-parsing:2:0)
         net/url)

(html->xexp
 (get-pure-port (string->url "http://www.google.com")))

Note that since this program refers to a planet package, running this program for the first time will download and install the htmlprag package. Building the documentation could take quite a while. That's an one-time cost, though, and running the program again shouldn't take more than a few seconds.

EDIT: In 2023, this code still works fine, but PLaneT is not widely used at this point, and it would probably be more idiomatic at this point to suggest installing the html-parsing package using raco install html-parsing or with the File>>Package Manager... menu, and then running

#lang racket

(require html-parsing
         net/url)

(html->xexp
 (get-pure-port (string->url "http://www.google.com")))
9
  • Should have clarified; if all you want is the raw text, you don't need the call to html->sxml, you can just use a (regexp-match #px#".*" ...) to suck the chars out of the pipe. Dec 24, 2012 at 4:26
  • 2
    port->string is probably what you'll see when pulling all the content from a port: docs.racket-lang.org/reference/…
    – dyoo
    Dec 24, 2012 at 5:20
  • @JohnClements Perfect, thanks! I used port->string and it gave me the web page as plain text!
    – Alex V
    Dec 24, 2012 at 7:33
  • Is there an obvious way built-in? It seems odd that there's no simple way without requiring a third-party library.
    – JasonFruit
    Dec 24, 2012 at 16:20
  • 1
    Sure thing; let me know what you think. Also, I used port->bytes; I'm guessing that the relevant RFCs are specified using bytes rather than utf-8. Dec 27, 2012 at 6:00

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.