Hi everyone,
Google AppEngine is now a production service that charges for usage
exceeding minimal resource limits. If you haven't done so already, go to
your appengine dashboard and accept the new service terms. Those are
reachable from the Billing section of any of your applications via the
adminstration console at https://appengine.google.com/
Please monitor the consumption of resources via the administration
console. If your resource limits are exceeded, Aggregate will report
failures and cease to function.
When Aggregate errors, the first thing to check is that you haven't
exceeded your resource limits. Currently, when a resource limit is
encountered, Aggregate's error messages are difficult to interpret (e.g.,
"Error: Server failure"). We are working to improve our messaging.
An idle Aggregate 1.0 is currently hitting the free-usage resource limit on
"Datastore Read Operations", which is set at 50,000 reads/day.
We will be releasing Aggregate 1.0.1 which will lengthen the time between
/gae/watchdog starts (from 3 minutes to 6 minutes), lengthen the intervals
between web page background-refreshes (from 5 seconds to 10 seconds between
refreshes), and we will be caching more-frequently-used information.
However, with these changes, we still expect that an active Aggregate
instance will exceed the free-usage resource limit. We expect that an
idle or very lightly used Aggregate 1.0.1 will not exceed this
free-resource limit.
From Google's billing help pages,
http://code.google.com/appengine/docs/quotas.html , once you exceed any of
the free resource limits, the minimum charge for an AppEngine application
is $2.10 / week, ($109.20 / year).
The primary resource limit we expect Aggregate to exceed is "Datastore Read
Operations." Additional operations cost $0.07 / 100k reads. If the entire
$2.10 / week goes towards this resource, we expect you will be able to use
Aggregate 1.0.1 fairly actively before exceeding this weekly limit.
An alternative hosting model is Amazon AWS; we are investigating the costs
and options for that platform, but expect AppEngine to be lower-cost for
sporadically- or lightly- used instances (the cross-over point is likely
around $350.00 / year).
----Details-----
Q: what is /gae/watchdog and why is it doing so many reads?
Watchdog is a background activity that runs every 3 minutes (to be
increased to 6 minutes in Aggregate 1.0.1). It ensures that all other
background tasks reach completion in the face of sporadic failures; this
includes publishing to external services, generation of csv and kml file
exports, form deletion, and purging of published submissions. Running
every 3 minutes, it is executed 20 times per hour, or 480 times per day.
For a 50,000 read threshold, this means that watchdog would need to execute
fewer than 105 read operations with each run. Watchdog often has to spin
up a new Aggregate instance -- starting from scratch each time. This means
that each execution must read all the security and form definition records
and then scan all the background task queues for stuck tasks; under these
conditions, it would be extremely challenging to do fewer than 105 read
operations. By increasing the interval to 6 minutes, we can halve the
consumption, but, even then, it is still likely to account for nearly 50%
of the free Datastore Read resource limit.