Nice you to ask this question and think about how to be as considerate as you can.
Some other thoughts:
- Find the most minimal, least expensive (for you and them both) way to get the data you're looking for. Sometimes you can iterate through search results pages and get all you need from there in bulk, rather than iterating through detail pages one at at a time.
- Even if they don't have an official/documented API, they may very likely have internal JSON routes, or RSS feeds that you can consume directly, which may be easier for them to accommodate.
- Pay attention to response times. If you get your results back in 50ms, it probably was trivially easy for them and you can request a bunch without troubling them too much. On the other hand, if responses are taking 5s to come back, then be gentle. If you are using internal undocumented APIs you may find that you get faster/cheaper cached results if you stick to the same sets of parameters as the site is using on its own (e.g., when the site's front end makes AJAX calls)
Some other thoughts:
- Find the most minimal, least expensive (for you and them both) way to get the data you're looking for. Sometimes you can iterate through search results pages and get all you need from there in bulk, rather than iterating through detail pages one at at a time.
- Even if they don't have an official/documented API, they may very likely have internal JSON routes, or RSS feeds that you can consume directly, which may be easier for them to accommodate.
- Pay attention to response times. If you get your results back in 50ms, it probably was trivially easy for them and you can request a bunch without troubling them too much. On the other hand, if responses are taking 5s to come back, then be gentle. If you are using internal undocumented APIs you may find that you get faster/cheaper cached results if you stick to the same sets of parameters as the site is using on its own (e.g., when the site's front end makes AJAX calls)