0

I try to use SplashRequest with endpoint='execute' with below code to send a post request,but the result shows I didn't succeed.

import re
import sys
import os
import scrapy
from scrapy_splash import SplashRequest
from crawler.items import CrawlerItem


class Exp10itSpider(scrapy.Spider):
    name = "test"
    lua_script = """
    function main(splash, args)
      assert(splash:go(args.url))
      assert(splash:wait(0.5))
      return splash:html()
    end
    """

    def start_requests(self):
        urls = [
            'https://httpbin.org/post^sss=lalala'
        ]
        for url in urls:
            if "^" in url:
                post_url_list = url.split("^")
                post_url = post_url_list[0]
                post_data = post_url_list[1]
                yield SplashRequest(post_url, self.parse, endpoint='execute', magic_response=True, meta={'handle_httpstatus_all': True}, args={'lua_source': self.lua_script, 'http_method': 'POST', 'body': post_data})

    def parse(self, response):
        input("body is:")
        print(response.body)

However,I can not post it successully,the return content shows me I don't post it well.Can you help me on post request with SplashRequest in scrapy_splash?

1 Answer 1

3

From the documentation:

Splash 1.8+ is required to handle POST requests; in earlier Splash versions http_method and body arguments are ignored. If you work with /execute endpoint and want to support POST requests you have to handle http_method and body arguments in your Lua script manually.

Also in the documentation, there's an example of POST request. Adapted for a minimal ready-to-run example:

import scrapy
from scrapy_splash import SplashRequest

class SplashPostSpider(scrapy.Spider):
    name = "splash_post"

    lua_script = """
    function main(splash, args)
      assert(splash:go{
        splash.args.url,
        http_method=splash.args.http_method,
        body=splash.args.body,
      })
      assert(splash:wait(0.5))
      return {
        html = splash:html(),
      }
    end
    """

    def start_requests(self):
        post_url = 'https://httpbin.org/post'
        post_data = 'foo=bar'
        yield SplashRequest(post_url, self.parse, endpoint='execute',
                            magic_response=True, meta={'handle_httpstatus_all': True},
                            args={'lua_source': self.lua_script, 'http_method': 'POST', 'body': post_data})

    def parse(self, response):
        print(response.body)
0

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.