Source: https://cloud.google.com/appengine/quotas#Requests
This limitation is affecting Collect users that are adding individual binary attachments (image, audio, video, file) larger than 32 MB. Sending those submissions to an Aggregate instance hosted in AppEngine will fail with a Collect upload error: write error: ssl=0x9d03b600: I/O error during system call, broken pipe
error and a log stacktrace:
11-26 13:03:25.507 6169-6308/org.odk.collect.android E/HttpClientConnection: javax.net.ssl.SSLException: Write error: ssl=0xa93ec580: I/O error during system call, Broken pipe
at com.android.org.conscrypt.NativeCrypto.SSL_write(Native Method)
at com.android.org.conscrypt.OpenSSLSocketImpl$SSLOutputStream.write(OpenSSLSocketImpl.java:839)
at org.opendatakit.httpclientandroidlib.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:126)
at org.opendatakit.httpclientandroidlib.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:138)
at org.opendatakit.httpclientandroidlib.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:169)
at org.opendatakit.httpclientandroidlib.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:115)
at org.opendatakit.httpclientandroidlib.entity.mime.content.FileBody.writeTo(FileBody.java:121)
at org.opendatakit.httpclientandroidlib.entity.mime.AbstractMultipartForm.doWriteTo(AbstractMultipartForm.java:134)
at org.opendatakit.httpclientandroidlib.entity.mime.AbstractMultipartForm.writeTo(AbstractMultipartForm.java:157)
at org.opendatakit.httpclientandroidlib.entity.mime.MultipartFormEntity.writeTo(MultipartFormEntity.java:113)
at org.opendatakit.httpclientandroidlib.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:158)
at org.opendatakit.httpclientandroidlib.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:162)
at org.opendatakit.httpclientandroidlib.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:237)
at org.opendatakit.httpclientandroidlib.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:122)
at org.opendatakit.httpclientandroidlib.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
at org.opendatakit.httpclientandroidlib.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.opendatakit.httpclientandroidlib.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.opendatakit.httpclientandroidlib.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.opendatakit.httpclientandroidlib.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.opendatakit.httpclientandroidlib.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.opendatakit.httpclientandroidlib.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.odk.collect.android.http.HttpClientConnection.uploadSubmissionFile(HttpClientConnection.java:380)
at org.odk.collect.android.upload.InstanceServerUploader.uploadOneSubmission(InstanceServerUploader.java:183)
at org.odk.collect.android.tasks.InstanceServerUploaderTask.doInBackground(InstanceServerUploaderTask.java:76)
at org.odk.collect.android.tasks.InstanceServerUploaderTask.doInBackground(InstanceServerUploaderTask.java:39)
at android.os.AsyncTask$2.call(AsyncTask.java:304)
at java.util.concurrent.FutureTask.run(FutureTask.java:237)
at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:243)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
at java.lang.Thread.run(Thread.java:761)
(source Collect upload error: write error: ssl=0x9d03b600: I/O error during system call, broken pipe)
Some insights into this issue:
- This could also happen to submissions without attachments, but it seems very unlikely to have an only text submission larger than 32MB.
- Aggregate can tell Collect how big requests should be.
- A user could set this to any value (e.g. 1GB) supported by the infrastructure where Aggregate is deployed
- Setting this to a value higher than 32MB while serving Aggregate in AppEngine wouldn't make any sense since the infrastructure will enforce a 32MB limit.
- Collect divides a submission into several smaller requests. If the server doesn't tell how big request should be, it defaults to 10MB requests. An example:
- Let's say there's a submission with four videos of size 4 MB, 5 MB, 6 MB, and 5MB respectively
- Collect will make 3 requests:
- submission XML + videos #1 & #2 for a total of ~9MB
- submission XML + video #3 for a total of ~6MB
- submission XML + video #4 for a total of ~5MB
- Collect will send complete attachments (it won't make smaller chunks of them). This means that when an attachment is 50MB big and the server's max request size is 32MB, it can't make a first 32MB chunk and a second chunk of 18MB to work around the limitation.
As discussed in the @TSC, we want to start an open discussion about this and try to explore the solution space for this issue. This is a non comprehensive list of topics we think we should discuss:
- Workarounds:
- How to prevent users from adding too large binary attachments
- How to design forms that will prevent this situation
- Alternative hosting providers for Aggregate