I am trying to scrape API response data from https://prod-mkt-gateway.mindbody.io/v1/search/locations API. It is basically AWS Cloudsearch data.The actual page link is this - https://www.mindbodyonline.com/explore/fitness/studios-san-diego-ca-us
which tells to set a json input as payload. Now, I am not getting the whole data (there is around 45K of it). So, how to get the whole data? Is there any pagination here? Please let me know. Here is my code -
import requests
import json
headers = {
"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0"
}
payload = {
"page":{
"size":50,
"number":9000,
"start":3000
},"filter":{
"radius":-5,
"latitude":45,
"longitude":-99,
"categoryTypes":[
"Fitness",
"Beauty",
"Wellness"
]
}}
url = "https://prod-mkt-gateway.mindbody.io/v1/search/locations"
r = requests.get(url, headers = headers, params=payload).json()
data_cleaned = []
for d in r['data']:
data_cleaned.append({
'location_address': d['attributes']['location_address'],
})
with open("output.json", "w") as outfile:
json.dump(r, outfile)