I would have some sympathy with this if you made it easy for single people to pay you to be able to use your data in a manner outside of the ToS of the free APIs.
However, for the Elevation API there are two choices:
* free and heavily restricted
* business api, which looks extremely sketchy †
and is blocked to most people
The gaming world has learned this with Steam and friends already. People will pirate the hell out of digital offerings that are not available in a reasonable manner. Steam recognized this and made games available in the most convenient manner humanly possible and are making money hand-over-fist. Sure, there are still pirates in gaming, but those are the people who are either paranoid about Steam or those who truly couldn't afford it anyhow.
You're still in this pre-realization phase and have only one single entity to blame if people "misuse" your APIs: Yourself. Make the APIs available for normal people in a reasonable manner and you will find adherence to your ToS becoming the norm, not the exception.
† By sketchy i mean the facts that there is zero pricing information, it's a buzzword-slaughterhouse, and the only way to get at it is to fill out a form with a lot of information, an indication of what one might want to do and which offers no further information or even actual flesh-and-blood human contact.
I don't understand this feeling of self-entitlement - who are we to judge whether or not an API has been offered "in a reasonable manner?" I'm guessing it's very resource-consuming to collect, process, and maintain the data necessary for geolocation APIs (especially given the upstream providers).
It'd be one thing for Google to be deceptive about it, offer a free service, and try to upsell you every point along the way. It's quite another thing for Google to dedicate an entire section in the API intro page [1] about Usage Limits.
Saying that "Google has nobody to blame but itself" is interesting when the blame is usually from developers to Google, not the other way around.
I agree in part with your point, but I think Google has pulled a bit of a bait and switch here. For many years, they were conspicuously open and friendly, as you would expect from a company run by engineers. That got them a lot of geek love, and a big boost to their early adoption efforts. I also think it had the effect of stunting the growth of alternatives. (E.g., Google Reader.)
Now they are running things more in standard large-business style. There's a natural rebound from that as people adjust. I agree the pragmatic thing is to drop the high expectations; I don't think Google is going back. But I can see why people are going to get all Kubler-Ross on Google while they adjust.
By open and friendly you must be talking about small pet projects released by single or small groups of engineers that didn't take much effort to produce.
If you are expecting unlimited free access to some thing like maps which employs many people and probably millions of dollars to produce, without expecting any type of return to sustain further growth...well it's time to rethink your strategy if you plan on staying in business.
Why does HN believe companies that spend millions of dollars to provide services, many times for free, and for pay with restriction are somehow "evil" (since we're talking about google this feels like an appropriate time to use this irrelevant phrase again).
I'm aware of that. But they also came out of an academic background, used a lot of open source, published many papers, and built an engineering-focused culture, not a business-focused one.
That is, sadly, changing. But it's not a necessary change.
I would offer that any potential or actual user of the API is an apt judge of whether the API has been offered "in a reasonable manner."
As it currently stands, one has the option of 2,500 requests per day for free or a minimum of $10,000 [1] and a limit of 100,000 requests per day.
Is it unreasonable to wish for at least one option between the two current choices of $0 and >=$10,000? The parent isn't being self-entitled, he or she clearly stated, " ... if you made it easy for single people to pay you to be able to use your data ... [emphasis added]"
I actually talked to Google yesterday. Minimum tier is 100k per day at $17.5k per year. Very fair price (If you think that is expensive, then you clearly do not work in commercial GIS, where licensing of software and data is insanely expensive). At the same time, they force you to use the data on a Google Map component (yes, even for the paid accounts), and while their api is acceptable, their js map control is severely lacking in features compared to something like leaflet, openlayers, or esri's js component. This may have very well been simply an arbitrary decision, but it has the effect of snuffing out any small companies or start ups that want to use their apis to build new, innovative applications. Considering the big few companies in this space bought up just about all the data companies that originally compiled the underlying data, this seems completely anti-competitive to me. I am not saying they should give it away for free, but there should definitely be some smaller paid plans with more open terms of use, considering that Google and the couple other competitors basically just purchased a cartel together over the last decade.
Honestly, I had to take a look at in depth to respond, since it has been a while since I looked at the google api in depth. What I still notice is that Openlayers and the esri allow for a lot more service types. You have KML and GeoRSS, which are fine, but do not seem to support many other of the more common formats out of the box such as Arcserver services,WMS, or WFS (which are probably the 3 most common types in the GIS world). I have seen some workarounds for these, but they seem hacky and slow in the implementations I have seen. I do not have a big problem with KML, but I cannot always change what format a 3rd party serves a map as.
The other thing that appears to be lacking is front end geoprocessing. With esri at least, you can do quite a bit of geometry manipulation out of the box. This can be extremely useful for realtime data visualizations, as starting up a new map service is often a bottleneck with any of the backend platforms. If you can just render the new layer on the client, you save loads of time.
My last issue might just be a lack of understanding, but from what I have seen of the backend offering that google maps enterprise offers, it is missing some major features. For one, the documentation is lacking, and it appears that the entire rest api is marked as experimental. Second, although it allows for some basic hosting and whatnot (easily replicated by a free server like geoserver, which is trivial to set up), it once again does not have any real processing engine, so you have to download all of the data to another server you own anyway to do any real work on the data.
That being said, I really do think it is a great product, and I definitely point people in your direction if they are looking for a quick an easy way to add a map to a site with some basic data. At the same time, I am not sure the enterprise plan really makes much sense, since the results can be so easily replicated with geoserver + openlayers and you already need a server solution anyway for running a processing engine like grass or pyqgis.
Also note: I am probably not your typical use case, and I realize that most people just need to throw preprocessed layers on a static map. All I can say is that I look forward to the day when I do not feel like I am the only one around doing dynamic big data GIS on the web, so I don't have to keep building all of this stuff from scratch ;)
I also want to mention that I retract the part about leaflet. I think you guys have wayyy more features than them, but they are the new kids on the block so we can cut them some slack. Also, their controls are the best designed for aesthetics and usability IMHO, which is no small matter.
I do not know if the Crime Doesn't Climb usage was against the ToS.
They could have possibly made all their requests within the rate limit imposed by Google. I'm unsure, however, if their usage of the data was allowed: "The Elevation API may only be used in conjunction with displaying results on a Google map; using elevation data without displaying a map for which elevation data was requested is prohibited." [1]
That said, I certainly think it's reasonable to want Google to allow paid access to their API at a rate between their current free offering and their prohibitively expensive offering for personal and small business usage.
Building your own elevation API is absolutely trivial; I don't know why anyone would even want to pirate Google's API for large-scale usage.
Firstly, the data is freely available. It's NASA's SRTM [1], downloadable from a zillion mirror sites.
Then just take some code to calculate a lat/long offset and find the right position in the right tile. Bob Osola has some easily portable PHP for this if you need it [2].
Free hint: Putting both your tiles and your (Lua) lookup code in Redis, then calling the latter with EVALSHA, is a really neat and fast way to do this.
If you really need ocean depths, higher resolutions than SRTM offers, or stuff above 60 degrees latitude, then it gets a little more complex. But for 90% of cases doing it yourself is eminently practicable.
Building a robust elevation api is definitely not trivial for the average dev using the google api. If someone thought the only way to get elevations was through google then they definitely will not know how to find and compile the data, then write the TIN (triangulated irregular network) algos necessary to appropriately estimate the elevation of an arbitrary point. No, it is not that hard, since tools like Grass can do a lot of the TINing stuff for you, but just running Grass, much less scripting it, is not trivial for someone unfamiliar with GIS. Making this stuff stable and performant on a server (as opposed to a one off calculation on a dataset) is also not trivial in the least.
I would not think interpolation would be that much easier than a creating a TIN. I have not looked at the dataset though, so it might be "pre-interpolated" (probably using a TIN if I had to guess), in which case this would be pretty easy with something like pyqgis on a flask server or whatever. The entire api could basically just be:
interesting. I think of these data are freely available, then there ought to be more people using it and making competing versions so that google will not have a monopoly on provision of such data apis.
Sorry, should have made myself clearer. I'm not saying that Google just uses SRTM - I'm sure they throw lots of other sources and general Google Magick in there. But for most people in most places, SRTM is good enough.
It's not really entitlement. It's part of market forces. People say good things about products that are sold well, and bad things about products that are sold badly. Negative terms create negative buzz and drive people away. This is how it's supposed to work. And 'negative terms' is up to the market to decide. Obviously some people are unreasonable, but they're usually outliers.
There are many problems with the ownership of data, entitlement, etc.
I've argued with the founder of delicious endlessly about the API limits they imposed which ultimately meant that nothing interesting could be done on the outside with delicious. Nothing interesting ever happened on the inside and ultimately it got sold to Yahoo! and destroyed. I guess the founder got some cash, but the data that was contributed by the end users was destroyed.
Nobody asked them for permission to sell to a psychotic company, have the site destroyed, etc.
Towards the end spammers found that they could (within the API terms) get endless amounts of legitimate 'cover traffic' to cover their links.
Really, don't bother trying to contact them, it's not worth it. If anything their business API is less sketchy than their actual sales tactics.
We had two different conversations with them and the first time the rep quoted us a rate that worked out to $72 per 1000 map impressions. The second time we spoke to a different rep who, when asked the differences between the free plan and the business plan, told us that if we were currently on the free plan outside of the ToS there would be "consequences".
Edit: I should add that afterwards we immediately switched to MapBox. Straightforward pricing, and it feels great to know that we're supporting the open source community.
uh, even at its highest it was $4 per 1000 (and now it's 50 cents) and you get 25,000 free per day. And that's through the automated API console, no enterprise license. And the "consequences" are...you get an over quota message. Seriously, search for "OVER_QUERY_LIMIT" and then compare that to all the stories of people being sued and shut down because they went over their maps API quota. I'll wait.
I've heard of terrible sales calls with them, so maybe that's what happened, but this seems a bit of a stretch. If it did, I hope you did some basic research before deciding what to go with.
In any case, MapBox is great and they deserve the support, so at least it worked out.
Yeah, we had done some research before going into the call, which is why we were just floored at what they were saying. "So you're saying that an enterprise license is more expensive than the business rate listed on your website?" ... "yeah."
I think the whole google maps enterprise was just somebody high up saying "hey, maps is popular, let's try monetizing it" and then not actually holding any of their sales people accountable for anything.
"I would have some sympathy with this if you made it easy for single people to pay you to be able to use your data in a manner outside of the ToS of the free APIs.
"
I'm not sure why you assume, among other things, that upstream contracts always make this possible.
An upstream contract that required detailed information about paid users, for instance? I could imagine wanting information like that for auditing if there were complex restrictions on permissible resale of the upstream data.
"Honest curiosity though: What kind of contract would make it impossible and would one make such a contract?
"
Most of them :)
They often have requirements on downstream users, such as requiring we provide info about them, or that they be restricted to certain use cases, etc
There are a lot of ridiculous restrictions i've seen in my days.
Most data brokers don't really want customers to go through you to get "close to bare data".
Thus, they give onerous contracts, and restrict how you can sublicense the data, as well as usually requiring attribution, in the hopes that people will see your restrictions, see the attribution, and buy the data from them instead.
In the specific example of the elevation data set, the data for the United States looks like it's public domain (USGS DEMs). The rest of the world may be more difficult to get high resolution but there is STRM which is reasonable ok for a lot of uses.
About three years ago I got in contact with some very nice people at NASA and/or the USGS (I don't recall which agency ended up being the end-point) for what, at the time, was some of the best data available from the joint ASTER missions they ran with Japan.
The process was essentially to fill out a form and send them a new, unopened hard drive. What I got back a few weeks later was about 125GB of high resolution GeoTIFFs. Along with it were PDF's explaining how the data was rectified and cleaned up as well as information for how they encoded the data.
The process and interactions really impressed me with the quality of people involved in agencies like the USGS and NASA.. just like the people who provide API's in the tech industry, they're usually really in to what they do and interested in hearing how other people are making use.
The data is totally out there and you can get your hands on it without a) breaking terms of service or b) needing any sense of entitlement or self-righteous "ethics". I think the article hit the nail on the head. In that ignoring the data freely available reduces its viability. Projects like GeoNames, OpenStreetMap, the deactivated OpenAerialMap are all incredible sources of information that we can't afford to ignore in favor of the easy way out
I’ve worked with NASA and USGS a fair deal both for fun and professionally, and this is typical.
They are extremely competent and sincerely want you to have good data. They are also hampered by the bureaucratic limits of any large organization. So it’s like working with a large, well-run business that’s hired a lot of the best people in its field and is working on good problems, yet is large enough that it can’t move to meet the exact needs of any one customer.
But on top of that there are political concerns. They have an institutional fear that a congressperson in a budget debate is going to stand up and say something like “And apparently we’re paying the Geological Survey $N million a year to run a web server for something called geotiffs that tell you how tall hills are!” That’s my impression from reading between the lines, anyway; no government employee I know has been indiscreet enough to deliberately hint at such a fear.
For example, the best interface to SRTM isn’t from the agencies that made it, it’s a single-page project from Derek Watkins at the NYT: http://dwtkns.com/srtm/
Working with NASA in particular feels like working with an industry leader that has a mysterious policy against advertising, or even going out of its way to help you find resources. (Individuals do, but not the organization, at least not anywhere near in proportion to the number and value of its resources.)
NOAA too: they have some amazing satellite imagery that’s public domain, but they simply do not have the budget to do anything but the most halfassed job of hosting, publicizing, and documenting it, because from a funding perspective that’s frivolous. They barely archive their images, because no one with budget control gets why a weather agency should save its input data. Look up “VIIRS granule” – that’s technically open data, but yikes.
The resources are there, and if you make the effort to figure it out, the people who manage them are pretty much all a delight to work with. But you have to deal with the damage created by a political culture that too often treats our civilian space and geospatial agencies as afterthoughts rather than as highly multiplied public goods.
With regards to your political argument, its something I've often thought about because it can go either way. There's the loaded language approach as you mention, but there's also the possibility of the hypothetical member of the House or Senate saying "And apparently we’re paying the Geological Survey $N million a year to run a web server that no one is making use of and funding data that no one seems to care about."
This would be a hard argument to defend against as it directly hits the value derived from the service rather than the idea that the service itself exists. In either case, I couldn't agree with you more that the civilian agencies really get a raw deal (and I'm sure its only compounded by the mostly stagnant level of people graduating with STEM degrees[1])
However, for the Elevation API there are two choices:
The gaming world has learned this with Steam and friends already. People will pirate the hell out of digital offerings that are not available in a reasonable manner. Steam recognized this and made games available in the most convenient manner humanly possible and are making money hand-over-fist. Sure, there are still pirates in gaming, but those are the people who are either paranoid about Steam or those who truly couldn't afford it anyhow.You're still in this pre-realization phase and have only one single entity to blame if people "misuse" your APIs: Yourself. Make the APIs available for normal people in a reasonable manner and you will find adherence to your ToS becoming the norm, not the exception.
† By sketchy i mean the facts that there is zero pricing information, it's a buzzword-slaughterhouse, and the only way to get at it is to fill out a form with a lot of information, an indication of what one might want to do and which offers no further information or even actual flesh-and-blood human contact.