Currently I have two small projects using Scrapy. One project is basically to scrape URL's, while the other is only to scrape products of the scrape URL's. The directory structure is this:
.
├── requirements.txt
├── .venv
├── url
| ├── geckodriver
| ├── scrapy.cfg
| ├── url
| | ├── items.py
| | ├── middlewares.py
| | ├── pipelines.py
| | ├── settings.py
| | ├── spiders
| | | ├── store1.py
| | | ├── store2.py
| | | ├── ...
├── product
| ├── geckodriver
| ├── scrapy.cfg
| ├── product
| | ├── items.py
| | ├── middlewares.py
| | ├── ...
When I want to run a spider using the command, I always must follow this path: ~/search/url$ scrapy crawl store1
or ~/search/product$ scrapy crawl store1
.
How can I deploy and run this project using AWS lambda functions?