Google AppEngine limits requests to a max size of 32 MB

Source: https://cloud.google.com/appengine/quotas#Requests

This limitation is affecting Collect users that are adding individual binary attachments (image, audio, video, file) larger than 32 MB. Sending those submissions to an Aggregate instance hosted in AppEngine will fail with a Collect upload error: write error: ssl=0x9d03b600: I/O error during system call, broken pipe error and a log stacktrace:

11-26 13:03:25.507 6169-6308/org.odk.collect.android E/HttpClientConnection: javax.net.ssl.SSLException: Write error: ssl=0xa93ec580: I/O error during system call, Broken pipe
        at com.android.org.conscrypt.NativeCrypto.SSL_write(Native Method)
        at com.android.org.conscrypt.OpenSSLSocketImpl$SSLOutputStream.write(OpenSSLSocketImpl.java:839)
        at org.opendatakit.httpclientandroidlib.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:126)
        at org.opendatakit.httpclientandroidlib.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:138)
        at org.opendatakit.httpclientandroidlib.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:169)
        at org.opendatakit.httpclientandroidlib.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:115)
        at org.opendatakit.httpclientandroidlib.entity.mime.content.FileBody.writeTo(FileBody.java:121)
        at org.opendatakit.httpclientandroidlib.entity.mime.AbstractMultipartForm.doWriteTo(AbstractMultipartForm.java:134)
        at org.opendatakit.httpclientandroidlib.entity.mime.AbstractMultipartForm.writeTo(AbstractMultipartForm.java:157)
        at org.opendatakit.httpclientandroidlib.entity.mime.MultipartFormEntity.writeTo(MultipartFormEntity.java:113)
        at org.opendatakit.httpclientandroidlib.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:158)
        at org.opendatakit.httpclientandroidlib.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:162)
        at org.opendatakit.httpclientandroidlib.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:237)
        at org.opendatakit.httpclientandroidlib.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:122)
        at org.opendatakit.httpclientandroidlib.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
        at org.opendatakit.httpclientandroidlib.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.opendatakit.httpclientandroidlib.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at org.opendatakit.httpclientandroidlib.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.opendatakit.httpclientandroidlib.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at org.opendatakit.httpclientandroidlib.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.opendatakit.httpclientandroidlib.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.odk.collect.android.http.HttpClientConnection.uploadSubmissionFile(HttpClientConnection.java:380)
        at org.odk.collect.android.upload.InstanceServerUploader.uploadOneSubmission(InstanceServerUploader.java:183)
        at org.odk.collect.android.tasks.InstanceServerUploaderTask.doInBackground(InstanceServerUploaderTask.java:76)
        at org.odk.collect.android.tasks.InstanceServerUploaderTask.doInBackground(InstanceServerUploaderTask.java:39)
        at android.os.AsyncTask$2.call(AsyncTask.java:304)
        at java.util.concurrent.FutureTask.run(FutureTask.java:237)
        at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:243)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
        at java.lang.Thread.run(Thread.java:761)

(source Collect upload error: write error: ssl=0x9d03b600: I/O error during system call, broken pipe)

Some insights into this issue:

  • This could also happen to submissions without attachments, but it seems very unlikely to have an only text submission larger than 32MB.
  • Aggregate can tell Collect how big requests should be.
    • A user could set this to any value (e.g. 1GB) supported by the infrastructure where Aggregate is deployed
    • Setting this to a value higher than 32MB while serving Aggregate in AppEngine wouldn't make any sense since the infrastructure will enforce a 32MB limit.
  • Collect divides a submission into several smaller requests. If the server doesn't tell how big request should be, it defaults to 10MB requests. An example:
    • Let's say there's a submission with four videos of size 4 MB, 5 MB, 6 MB, and 5MB respectively
    • Collect will make 3 requests:
      1. submission XML + videos #1 & #2 for a total of ~9MB
      2. submission XML + video #3 for a total of ~6MB
      3. submission XML + video #4 for a total of ~5MB
  • Collect will send complete attachments (it won't make smaller chunks of them). This means that when an attachment is 50MB big and the server's max request size is 32MB, it can't make a first 32MB chunk and a second chunk of 18MB to work around the limitation.

As discussed in the @TSC, we want to start an open discussion about this and try to explore the solution space for this issue. This is a non comprehensive list of topics we think we should discuss:

  • Workarounds:
    • How to prevent users from adding too large binary attachments
    • How to design forms that will prevent this situation
  • Alternative hosting providers for Aggregate

So, I'll start with some stuff we talked about in the last TSC meeting:

How to prevent users from adding too large binary attachments

Enketo already does that by preemptively querying the server's limitation before letting the user add attachments and preventing the user of adding attachments that are too big.

Tradeoffs and issues of this approach are:

  • Collect can (and will) run in scenarios without connectivity, which would make it impossible to check server limitations.
  • Once a blank form is pulled from a server, Collect's workflow can be 100% offline (combined with Briefcase). In this scenario, Collect might apply limitations that are not required because submissions won't ever be sent to any server.
  • Telling users that they can't add an attachment can be frustrating, especially if they can't solve the situation. For example, can users cut or compress videos and audios with their devices?

How to design forms that will prevent this situation

If we end up suggesting attaching smaller attachments, instead of having one binary field, forms could be designed with repeat groups of one binary field to let users attach a sequence of audios, videos, photos...

Alternative hosting providers for Aggregate

  • We're already releasing a virtual machine OVA file. How can this be leveraged in hosting providers?
  • We also have Aggregate Docker images and Docker Compose setups. Is there any way to reuse them?
  • AWS (Amazon)
1 Like

Short term

  1. We update the docs and the installer and warn folks that attachments to App Engine must be smaller to 32 MBs. In the docs, we suggest using multiple questions if appropriate.

  2. We update Collect's media/file widgets so that we see an appspot URL (in the settings or in the submission_url) and a user adds an attachment greater than 32 MB, we add a small warning to the widget.

    This attachment is x MB. Attachments greater than 32 MB will be rejected by Google App Engine.

    The above change might miss some GAE servers with custom domains, but I'd expect that number to be small. Moreover, those folks could transparently change their backend if necessary.

  3. We update Collect's error messages to provide a more useful error message if that submission is sent. Worst case scenario, users can get that data using Briefcase.

Long term

After 10 years of making changes to support App Engine, I'm mostly allergic to making any more.

  1. I think we should help people get turnkey setups on Linode/DigitalOcean/Vultr and doing that through Docker seems like the easiest option.
  2. For AWS users, a CloudFormation template seems better in that regard. For both of these, we'd include an SSL setup and default to PostgreSQL.

I'm not as interested in AMIs because it's yet another thing we have to manage.

3 Likes