I work at Google. We don't typically monitor HN, reddit, twitter, etc but we have plenty of people watching Stack Overflow (the one we direct people to from our docs). The default quotas are conservative during this Beta period but we're happy to increase when needed--the service is certainly not provisioned as high as IMAP at this point. Submitting the quota increase request (from the developers console) will send us an email and we'll either approve it or if we need to, we'll start an email thread to ask for more info/suggest improvements. Hope that helps!
In our tests we did list requests for 100 messages and then bulk requests for each messages' headers (messages.get("me", message.getId).setFields("historyId,payload/headers")).
Did you ever try to contact someone at Google for higher quota? (There's a form linked from the developer site.) I'm not surprised the default quotas were low, it is still Beta...
Our primary issue isn't the quota limits, it's the actual speed of the API. The best we were able to achieve with the Gmail API was 3000 messages/minute whereas IMAP easily lets us get 20k messages/minute.
Are you limiting yourself to one thread or something? If you're not getting limited by quota then why not use batching + more threads until you do get to those limits?
We are getting limited by the quota. As the documentation says, quotas can be temporarily exceeded and in our tests we could fetch about 3000 messages in those bursts. After that, we would be 403'd for one to two minutes before requests would succeed again with quotas strictly enforced for an unknown amount of time past that.
