0

I am trying to scrape API response data from https://prod-mkt-gateway.mindbody.io/v1/search/locations API. It is basically AWS Cloudsearch data.The actual page link is this - https://www.mindbodyonline.com/explore/fitness/studios-san-diego-ca-us

which tells to set a json input as payload. Now, I am not getting the whole data (there is around 45K of it). So, how to get the whole data? Is there any pagination here? Please let me know. Here is my code -

import requests
import json

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0"
}

payload = {
    "page":{
        "size":50,
        "number":9000,
        "start":3000
    },"filter":{
        "radius":-5,
        "latitude":45,
        "longitude":-99,
        "categoryTypes":[
            "Fitness",
            "Beauty",
            "Wellness"
        ]
    }}

url = "https://prod-mkt-gateway.mindbody.io/v1/search/locations"
r = requests.get(url, headers = headers, params=payload).json()

data_cleaned = []
for d in r['data']:
    data_cleaned.append({
       'location_address': d['attributes']['location_address'],
    })

with open("output.json", "w") as outfile:
    json.dump(r, outfile)

1 Answer 1

0

There were a few things wrong with your script, you need to make a post request and increment the pages as you go. Also you need to send the api json data as a payload.

I discovered that you can change the radius (the search area) to -1 to force it to include every location. Also you can get a max of 100 locations per search so that should speed things up, see below:

import requests
import json

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0"
}

data_cleaned = []
for page in range(1,5):
    print(f'Scraping page {page}')

    payload = {"sort":"-_score,distance","page":{"size":100,"number":page},"filter":{"categories":[],"radius":-1,"term":"","cmMembershipBookable":"any","latitude":32.715738,"longitude":-117.1610838,"categoryTypes":[]}}

    url = "https://prod-mkt-gateway.mindbody.io/v1/search/locations"
    r = requests.post(url, headers = headers, data=json.dumps(payload)).json()
    print(r['meta'])
    for d in r['data']:
        if d not in data_cleaned:
            data_cleaned.append(d)

print(f'Scraped {len(data_cleaned)} results')

with open("output.json", "w") as outfile:
    json.dump(data_cleaned, outfile)
1
  • Hello @bushcat69, Thank you so much. After running your code I found it gives me 10K data. Is it possible to get all the 42K data? Please let me know. Commented Feb 3, 2022 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.