not immediately. sounds like i have some investigation to do. how long does it hang before timing out?
Unqualified comment from the sidelines:
Is the ODK Central that times out behind another proxy with its own timeout settings?
What's the setup and deployment?
Are there logs? (even inside containers e.g. written to
@issa - 60 seconds exactly. Consistently.
@Florian_May Looks like the Nginx proxy is timing out trying to retrieve from Central.
Logs in the service_1 container don't suggest anything wrong -
Nothing in /var/log/odk/stderr.log and only
::ffff:172.19.0.5 - - [11/Sep/2019:10:21:36 +0000] "GET /v1/projects/6/forms/baseline_hh_survey/submissions.csv.zip HTTP/1.0" 200 -
that's really helpful, thanks! i'll take a look.
Some feedback on App User & Collect setup
The QR codes need to have a human readable component
This is really important when a supervisor is setting up lots of devices (especially when doing it remotely in a different country from the main IT support team) so they know each device is being configured with the correct settings.
This will become even more important once it is possible to setup different app users with different form access on a given project.
Related issue is that it's very difficult to look at the subsequent URL in the settings and make sense of it - I.e easily know which server / project you are linking too - this is important when we have field superviaors overseas and we need them to be able to easily check that they are connected to correct project / server
For example in collect I'd like it to then show that my URL was XYZ
My project was ABC
And my app Username was 123
That way I can be confident my remote user has set the device up using the correct QR code/settings
@LN these suggestions would have to be implemented in Collect i guess?
It's not only collect though - the actual QR code image central generates under that would ideally have some text that says
App User Name
Something like this would be ideal
- Also an option in Central to download / email the QR code would be very very valuable to help remote users setup devices
(Ideally automatically naming the QR code file based on Project name / App user-nickname)
If I understand correctly, @dr_michaelmarks, you'd like the image that includes the QR code to also include identifying information because it might be emailed, printed out, etc, is that right? That is, there's sufficient context for the QR code when it's displayed from Central but a supervisor setting up devices would not be logged in to Central.
I also do agree that it should be possible to review the server/project/app user configuration from clients. The project id is listed at the end of the URL but it's just a numeric identifier. Perhaps as a first step the project name could be appended to the URL (sluggified) and ignored by the server? For username, I think that if the QR code json included the app user name associated with the
username key, the username would show up in the server settings.
Correct - this is very common. See also my suggestion above building into central a download/email button for the QR code.
Wanted something like my example above:
Happy to discuss how to improve the display of URL within Collect; like your suggestion @LN to include project name. If we could something similar for App User Nickname that would help.
okay, i understand more about the QR code now. i didn't have the context that people were downloading the image as its own artifact; we can probably come up with a different solution there that achieves your eventual goal.
for background, from our research we'd concluded that most deployments involved the phones being set up ahead of time by a small centralized staff who could just provision the phones directly from the panel. being able to send provisioning information elsewhere was something we'd entertained but eventually ruled out because it sounded like it wouldn't be as useful. we can come up with a solution that's tailored for that approach rather than try to shoehorn things into the existing process.
as for the collect display of information, some of these proposed hacks might work, but i think some of them will cause problems with how collect authenticates with central.
Thanks - yes for us it is pretty common that we are remotely supporting field teams (for example that is how we do all the ebola vaccination work)
okay i looked into the timeout issue. turns out the server was doing the right thing and i neglected to configure nginx to match. silly and simple problem. silly and simple fix:
you can find the diff here if you want to quickly patch your setup.
(the increase to a 2 minute timeout is probably not strictly necessary. it's a per-chunk timeout, so it will only trip if no new data is streamed back to the client within that window. i artificially inflated row-processing time by 50ms per row and the first chunk came back at 58 seconds so i figured 2 minutes wasn't a terrible idea for some diabolically complex forms, possibly. i don't see the harm in increasing by 60 seconds.)
@MatthewMac did we manage to try this solution so we can test if it fixes issue our end?
well, that's extraordinarily strange. how sensitive is the data? it would help if i had it to work with directly.
It seems specifically a problem with a specific survey I think....
I have a different project with 200+ forms running on the same server no problem.
baseline_household_survey.xlsx (16.6 KB)
baseline_household_survey.xml (29.9 KB)
Added points for future (things I mentioned at Convening)
- Turn off decryption for projects (obviously wont decrypt existing data)
- 'Better' control of the decryption (i.e not just weak passphrases)
- Question is how on the fly-decryption will work on when form number is high
- Resume download from last time point
it's interesting to me that it's obvious it wouldn't decrypt existing data. i had planned on automatically (with good messaging of course) decrypting existing data on project encrypt disable. what's the case you envision where you'd want to disable but leave existing records encrypted? it sounded like the main reason you wanted disable was because it made trying out project encryption a lower commitment/barrier, in which case you'd only have test data right?
i think it's actually not too hard to provide the option, but i do want to be sure i understand the reasoning on your side to inform our decision making.
something other than passphrases:
my logic behind starting with passphrases was that pem files are really hard to handle safely and securely, and people are likely to do really insecure things with them (email them around, etc). certainly larger organizations with good infrastructure and practices like your own are more equipped to establish the right procedures.
passphrases at least can be stored in password managers, etc which do a good job of balancing user ease and security.
i think it makes sense to pursue something more secure, though i also wonder whether in this case it makes more sense to support something like a yubikey which is a more automatic and guaranteed-secure process than throwing a pem file around.
in either case, central supports the old aggregate encryption, so you can just continue your current practices for now.
on the fly decryption performance:
i don't really see a reason this would necessarily be bad. if it isn't good now we can improve it.
with the REST api, building a simple tool that does this should be pretty straightforward. briefcase already does it. adding windowing to the odata api (and possibly the csv api) should also not be all that difficult. just a matter of scheduling.