8

Currently I have two small projects using Scrapy. One project is basically to scrape URL's, while the other is only to scrape products of the scrape URL's. The directory structure is this:

.
├── requirements.txt
├── .venv
├── url
|   ├── geckodriver
|   ├── scrapy.cfg
|   ├── url
|   |   ├── items.py
|   |   ├── middlewares.py
|   |   ├── pipelines.py
|   |   ├── settings.py
|   |   ├── spiders
|   |   |    ├── store1.py
|   |   |    ├── store2.py
|   |   |    ├── ...
├── product
|   ├── geckodriver
|   ├── scrapy.cfg
|   ├── product
|   |   ├── items.py
|   |   ├── middlewares.py
|   |   ├── ...

When I want to run a spider using the command, I always must follow this path: ~/search/url$ scrapy crawl store1 or ~/search/product$ scrapy crawl store1.

How can I deploy and run this project using AWS lambda functions?

2

1 Answer 1

0

This code is a part of script used in a previous project for a client.

Just replace spider_class_getting_from_spiders with your spider class.

import imp
import sys
from crochet import setup, wait_for
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
from settings import *
from spiders import *


setup()
sys.modules["sqlite"] = imp.new_module("sqlite")
sys.modules["sqlite3.dbapi2"] = imp.new_module("sqlite.dbapi2")



@wait_for(900) # maximum 15 minutes 
def crawl(spider_class_getting_from_spiders):
    '''
    wait_for(Timeout = inseconds)
    change the timeout accordingly
    this function will raise crochet.TimeoutError if more than 900
    seconds elapse without an answer being received

    '''

    configure_logging({'LOG_LEVEL': 'ERROR'})
    process = CrawlerRunner(DOWNLOADER_MIDDLEWARES);
    d = process. Crawl(spider_class_getting_from_spiders);
    return d;


def lambda_handler(event, context):

    crawl(spider_class_getting_from_spiders)
    # it return the whole event instead of return on the response to send the input to next state in the step function
    event['statusCode'] = 200
    return event

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.