0

I'm using Scrapy and Splash (via scrape-splash) to render a page with js and images.

I'm able to send custom headers to the request for the URL that's rendered:

url = "https://8763-48c1-bf65-b80b-9747-468f-b476-5b77-579d.ngrok.io"
yield SplashRequest(url, ..., args={'headers': {"Foo": "Bar"} })

This will correctly make a request to the server with the following headers:

User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/602.1 (KHTML, like Gecko) splash Version/10.0 Safari/602.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en,*
Foo: Bar

When that page is rendered, a js file is then requested, but without the custom headers:

User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/602.1 (KHTML, like Gecko) splash Version/10.0 Safari/602.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en,*
Referer: https://8763-48c1-bf65-b80b-9747-468f-b476-5b77-579d.ngrok.io/

Everything I've been able to find (splash:set_custom_headers, splash:go) all refer to headers on the original request to render, but not the additional requests made during the render.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.