Upcoming changes to Aggregate

Aggregate v2.x is scheduled to be released in early February. This release has major changes that have been discussed by the Technical Steering Committee (TSC) behind Aggregate and I wanted to explain the rationale behind these changes and provide some guidance for Aggregate users.

Summary
Aggregate v2.x will not be installable on Google App Engine (GAE, appspot.com). You will only be able to install Aggregate v2.x on a standard cloud server (Digital Ocean, Vultr, Linode, Amazon EC2, Google CE, Azure VM) or on a local server.

Aggregate v2.x also removes features that are unsupported or deprecated (e.g., Google Fusion Tables, ODK Tables, and Google Accounts).

As far as ongoing development work on ODK servers, our focus will be:

  • Aggregate v1.7.x: Critical bug fixes
  • Aggregate v2.x: Minor features
  • Central: Major features

If you use Google App Engine
You can continue using the Aggregate v1.x series, but we will only ship critical bug fixes. If you depend on Aggregate, upgrade to ODK Aggregate v1.7. Consider migrating to the more maintainable v2.x series on a standard cloud server when it's released. We'll provide more documentation to help in the coming weeks.

If you don't use Google App Engine
You can use the Aggregate v2.x series. We will continue to ship minor features with the goal of making the codebase more maintainable. Consider trying the Central betas to prepare for an eventual switch.

Why we are removing App Engine support
When we launched Aggregate in 2008, we chose GAE as the default platform because it provided easy-to-use, free, secure, and scalable infrastructure for users. The trade-off was that we would have to write custom code to deal with the various restrictions GAE had.

Since then, there have been hundreds of thousands of data collection campaigns that have run on GAE. We are proud of what we've achieved, and in order to achieve more, we have to revisit our assumptions.

Our assumptions in 2008 were that GAE would get easier and more free. That has simply not been the case. Over the last decade, GAE has become more difficult and more expensive. The key facts are:

  • GAE is harder to setup than cloud servers from providers like Digital Ocean, Linode, and Vultr.
  • GAE has a free tier, but Google requires a credit card or bank details on file. It's unlikely GAE will be free forever.
  • GAE's complicated quotas make it hard for users to anticipate costs ahead of time.
  • GAE's restrictions (e.g., killing long-running tasks, non-standard database) make maintaining Aggregate difficult and fixing user problems (especially at scale) very, very painful.

Given our available resources, it's unsustainable to continue GAE support. For that reason, we've removed GAE support in Aggregate v2.x.

Why major features aren't being added to Aggregate
Aggregate is built on technology that makes it hard to evolve. The TSC believes that over time, Central will grow into a better platform for addressing some of the major server features that the community has requested.

We continue to need your help adding documentation and minor features to Aggregate. If you depend on it, please head to GitHub and help. It's never been easier to contribute.

Does Aggregate v2 have anything to do with ODK 2
Nope. The v2 designation for Aggregate is because we use semantic versioning and when we make changes that are not backwards compatible, we must increment the major version number.

Older versions of Aggregate did have ODK 2 support, but all ODK 2 code has been removed from Aggregate. If you use ODK 2, try the Sync Endpoint instead.

I have a question you didn't answer
Ask it below :point_down:

8 Likes

Dear Yaw,

No question! Just compliment from my end for v2.x series. This would be a great turning point for all ODK users and life will be much easier. Would love to move into v2.x series once it will be up. Just a question regarding documentation, will it be available under ODK 1? I believe more user would prefer either to go for Digital Ocean or EC2. More comprehensive documentation targeting less tech savvy users would be helpful.

Thanks,
Imran

2 Likes

@ggalmazor is been working on documentation (and automation so less documentation is needed). My hope is to review and merge that before the release this weekend.

1 Like

Since ODK2 won't be compatible with Aggregate, it would be nice to have the Getting Started Deployment Architect Guide for ODK2 updated to use Sync Endpoint.

A post was split to a new topic: Incomplete submission in Aggregate

A post was split to a new topic: Facing problem with my ODK Aggregate account

Is there an install documentation for Aggregate v2 on Google Compute Engine?

The install documentation for Aggregatev1 on Google App Engine was very good.

@CarlBeltran Please read the first post on this topic. It answers your question.

Hi there - I have been away for this forum for a year or so, so please forgive my lack of understanding. I am using normal ODK tools for simple data collection surveys (not ODK X), so please could you recommend what I should now use to install Aggregate if we can't install it on Google App Engine?

Many thanks!

Hi @Frances
If you want to continue to use Google App Engine then you can deploy https://github.com/opendatakit/aggregate/releases/tag/v1.7.3

1 Like

@Frances For the Aggregate v2.x series, we've also written up instructions at https://docs.opendatakit.org/aggregate-install for installing it on a variety of non-App-Engine platforms. Digital Ocean is probably the easiest.

1 Like

ok thanks!

@yanokwa would you recommend I just go down this route seeing as it's the only system I am familiar with?

Thanks!

@Frances If you have a choice, we recommend you try Digital Ocean.

Ok thanks - I will have a go!