I'm fairly new to Python and Scrapy and have issues wrapping my head around how to create nested JSON with the help of Scrapy.
Selecting the elements I want from HTML has not been a problem with the help of XPath Helper and some Googling. I am however not quite sure how I’m supposed to get the JSON structure that I want.
The JSON structure I desire would look like:
{"menu": {
"Monday": {
"alt1": "Item 1",
"alt2": "Item 2",
"alt3": "Item 3"
},
"Tuesday": {
"alt1": "Item 1",
"alt2": "Item 2",
"alt3": "Item 3"
}
}}
The HTML looks like:
<ul>
<li class="title"><h2>Monday</h2></li>
<li>Item 1</li>
<li>Item 2</li>
<li>Item 3</li>
</ul>
<ul>
<li class="title"><h2>Tuesday</h2></li>
<li>Item 1</li>
<li>Item 2</li>
<li>Item 3</li>
</ul>
I did find https://stackoverflow.com/a/25096896/6856987, I was however not able to adapt this to fit my needs. I would greatly appreciate a nudge in the right direction on how I would accomplish this.
Edit: With the nudge provided by Padraic I managed to get one step closer to what I want to accomplish. I've come up with the following, which is a slight improvement over my previous situation. The JSON is still not quite where I want it.
Scrapy spider:
import scrapy
from dmoz.items import DmozItem
class DmozSpider(scrapy.Spider):
name = "dmoz"
start_urls = ['http://urlto.com']
def parse(self, response):
uls = response.xpath('//ul[position() >= 1 and position() < 6]')
item = DmozItem()
item['menu'] = {}
item['menu'] = {"restaurant": "name"}
for ul in uls:
item['menu']['restaurant']['dayOfWeek'] = ul.xpath("li/h2/text()").extract()
item['menu']['restaurant']['menuItem'] = ul.xpath("li/text()").extract()
yield item
Resulting JSON:
[
{
"menu":{
"dayOfWeek":[
"Monday"
],
"menuItem":[
"Item 1",
"Item 2",
"Item 3"
]
}
},
{
"menu":{
"dayOfWeek":[
"Tuesday"
],
"menuItem":[
"Item 1",
"Item 2",
"Item 3"
]
}
}
]
It sure feels like I'm doing a thousand and a one things wrong with this, hopefully someone more clever than me can point me the right way.
dmoz.items.DmozItem?