ODK Central v0.4 Beta

[yanokwa]

Central betas are an opportunity to get community feedback on ongoing work, so check out the introduction and follow the instructions to try it out on our sandbox or your own machine!

Projects

The major highlight for this release is the introduction of Projects! Projects allow you to organize related forms and users. At a minimum, this should help those of you who are trying to run multiple projects off the same server, as mobile App Users connected to a project will only see forms from that project. You can find more information about projects here.

And, in the next few releases, we have exciting improvements planned for Projects: you'll be able to assign Managers project-by-project, directly manage the available forms and configurations of the mobile devices connected to each project, and more. We have a rough roadmap of those ideas and features here, and we'd love to hear your thoughts on them.

As a part of this change, all your existing forms will be migrated into a single project. They will continue working as-is for now, but note that if you are upgrading from v0.3 there are steps you will have to perform before you upgrade to v0.5 in order to ensure continued mobile connectivity. Please take a look at these notes if this describes you.

We would like to thank @danbjoseph, @berpita, @Lloyd_Banwart, Guido Pizzini, and certainly not least @dr_michaelmarks for their generous time and insight in helping us to design and plan for this and future releases of Central Projects.

Other Highlights

Projects aren't the only change we have made! Thanks in part to feedback from our users as well as our first community code contributor (notated below), we have made the following improvements:

• Much better alignment of the Central .csv export zipfile with the Briefcase format. In particular, we have improved the naming of repeat data files (#145), and renamed the files/ folder to media/ (#156) to better match Briefcase.
• We now include the App User's display name in the .csv export as well (#132, suggestion by @Felix_Mutua).
• In addition to the App User information, we record the deviceID on submission now (#150, contribution by Alex Patow).
• We now automatically (and invisibly) compress form attachments, both upon upload from your computer to Central, as well as on download to mobile devices, for faster data transfer times (#36/#50/#196, suggestion by @jniles).
• We send an email notification when any account's email or password is changed (#156, also contribution by Alex Patow).
• Error messages during form attachment upload now have explicit dismissal buttons (#176).
• Some error messages regarding backups were appearing when they should not have (#184).
• For API users, you may now upload submission attachments via the RESTful API by POSTing the binary data to /projects/:id/forms/:id/submissions/:id/attacments/:filename (#108, suggestion by @xiphware). You can find full API documentation on that here.

In addition, we have reformatted and refined the installation instructions for deploying Central to DigitalOcean, and moved them from GitHub to the ODK documentation website. If you have thought about deploying your own copy of Central in the past but were put off by the dense instructions, give them another look and please let us know if they seem more approachable.

We have also updated our instructions for additional deployment options, like enabling DKIM to prevent Central's emails from being marked as spam, using your own custom SSL Certificate, and more.

Known Issues

• If you are on using the sandbox, you might have to clear your browser cache to login.

Final Notes

Looking forward, we have our plans for Central v0.5 roughed out, but we would love to hear your thoughts if you have any.

We encourage you to try Central and put your feedback in a comment below We are particularly interested in answers to the following questions.

1. What, if anything, was confusing?
2. What missing feature is preventing you from using this for your next project?

[yanokwa]

This release has been a long time coming, so I wanted to say a special Thank You to @issa, @Matthew_White, and @LN for their work getting this shipped!

[Felix_Mutua]

This is great. @yanokwa . I love the direction this is heading... I'll soon be putting it in to some production work!

[issa]

hello! just so that everybody knows, there is an issue with the installation process right now, for anybody who is trying to install Central from scratch.

if you already have Central installed and are trying to upgrade, or if you are using the sandbox installation we offer as a playground, you should be fine.

if you are trying to install from scratch, then please wait a few days and we'll post an update here when we manage to talk to the right people to get the problem resolved.

[Phil_Burapha]

Hi @yanokwa Sanbox access DM said feedback on the v0.4 Beta should be left here.

I'm testing a form with the file-upload-widget (trying Central as I had a problem #455 with Aggregate v2.0.2).

The problem seems to be present in Central. The form works fine, but when I upload a .gpx file it is not included in the zipped file download. If I do the same with a .jpg file it is included.

My form and submissions can be seen here:

https://sandbox.central.opendatakit.org/#/projects/9/forms/mach_test_v0.4/submissions

[issa]

hey phil:

sorry it took some time to respond to you. i've taken a look and as far as i can tell Central never receives the data from Collect at all. reading over the issue #455 you linked, it seems the Aggregate folks have came to the same conclusion: the problem is with Collect, and therefore using Central instead of Aggregate will not solve it.

i'm sorry for the troubles this is causing!

[Absolomon_Kihara]

Installation issue:

I am trying to install ODK Central and so far I have been able to:

1. Installed ODK Central. I have other docker containers running, so ODK central was installed alongside the other containers
2. Used custom SSL certificates for ODK central installation
3. Modified the docker-compose.yml to use other free ports
4. Set-up nginx config
5. Started the containers and I can see they are running and they are healthy, see attached screenshot
   
   

However it seems that I cannot connect to the odkcentral_service which is running on port 8383. Trying to run curl http://127.0.0.1:8383 when logged in the docker host returns a connection refused error.

When I connect from the browser it gives me a 502 error.

The nginx logs shows that nginx cannot redirect traffic to the upstream application, see below:

2019/04/29 11:59:53 [error] 30202#30202: *389 no live upstreams while connecting to upstream, client: 162.158.42.50, server: odk-central.badili.co.ke, request: "GET /favicon.ico HTTP/1.1", upstream: "https://odk-central/favicon.ico", host: "odk-central.badili.co.ke", referrer: "http://odk-central.badili.co.ke/"
2019/04/29 12:00:04 [error] 30202#30202: *396 connect() failed (111: Connection refused) while connecting to upstream, client: 162.158.42.50, server: odk-central.badili.co.ke, request: "GET /favicon.ico HTTP/1.1", upstream: "https://[::1]:8383/favicon.ico", host: "odk-central.badili.co.ke", referrer: "http://odk-central.badili.co.ke/"
2019/04/29 12:00:04 [warn] 30202#30202: *396 upstream server temporarily disabled while connecting to upstream, client: 162.158.42.50, server: odk-central.badili.co.ke, request: "GET /favicon.ico HTTP/1.1", upstream: "https://[::1]:8383/favicon.ico", host: "odk-central.badili.co.ke", referrer: "http://odk-central.badili.co.ke/"
2019/04/29 12:00:04 [error] 30202#30202: *396 connect() failed (111: Connection refused) while connecting to upstream, client: 162.158.42.50, server: odk-central.badili.co.ke, request: "GET /favicon.ico HTTP/1.1", upstream: "https://127.0.0.1:8383/favicon.ico", host: "odk-central.badili.co.ke", referrer: "http://odk-central.badili.co.ke/"
2019/04/29 12:00:04 [warn] 30202#30202: *396 upstream server temporarily disabled while connecting to upstream, client: 162.158.42.50, server: odk-central.badili.co.ke, request: "GET /favicon.ico HTTP/1.1", upstream: "https://127.0.0.1:8383/favicon.ico", host: "odk-central.badili.co.ke", referrer: "http://odk-central.badili.co.ke/"

I think it is something to do with networks but I cannot seem to figure it out.

Anybody who knows what is going on?

[issa]

hey @Absolomon_Kihara:

it sounds like you know what you're doing, but unfortunately you're describing things a little too quickly for me to follow. some questions:

1. what modification did you make to docker-compose.yml?
2. what setup did you do on the nginx config?
3. where are you making the :8383 requests from?

[Absolomon_Kihara]

Thanks Clint for the follow up. I am not a noob with docker. I have a server which docker is already setup and running and serving a number of applications. Some of these applications were using the ports which odk-central wanted to use.

I mainly modified the ports which are being used. For nginx, I exposed different ports which are meant to forward the traffic from the docker host to the nginx container. See a screenshot of a section of the docker-compose.yml

For the nginx config, I re-used a config from one of my docker containers, see attached screenshot too.

I am making the :8383 requests from the docker-host.

[Odil]

Hello,
I was planning extensive testing the Central and therefore needed to install it locally. I'm running apache server in my Ubuntu 18.04 and have several websites using 80 as well as 443 ports, (I'm using Let’s Encrypt SSL Cert for Apache on Ubuntu) therefore I also changed ports in docker-compose.yml :

ports:
  - "8080:80"
  - "444:443"

When I run

docker-compose up

I can access Central, however running

sudo systemctl status docker-compose@central

shows that

• docker-compose@central.service - central via docker-compose
  Loaded: loaded (/etc/systemd/system/docker-compose@.service; indirect; vendor preset: enabled)
  Active: failed (Result: exit-code) since Thu 2019-05-02 14:04:12 UTC; 1min 43s ago
  Process: 17132 ExecStart=/usr/local/bin/docker-compose up (code=exited, status=200/CHDIR)
  Main PID: 17132 (code=exited, status=200/CHDIR)

I'm new to docker. Can anyone give me a hint how to correctly install Central to Ubuntu when I have running Apache?

Thank you!

[issa]

it certainly didn't sound like you were a noob!

your nginx is insufficient; please see the original configuration for details, but nginx must perform two tasks:

1. it must serve the local files that are built when the nginx container is built. if you do not do this the web interface will not work. those files are served under the root /
2. it must also proxy-forward only /v1 requests to the service on port :8383.

to be clear, the service container that starts up on port 8383 is only the backend API. it will also need access to the postgres container and the mail container under those hostnames (unless otherwise configured; see the files/ directory) to work correctly, and it will not on its own serve the web interface.

as for making the 8383 requests from the docker host, i know that can be done but i, unlike you, am a docker noob so i'm not sure how that works and i'm not sure how to help you with it.
all i know is that the docker-compose orchestration we ship with the software works for me every time as-is.

[issa]

hey!

from what i understand you are trying to run the docker-compose package we ship with Central exactly as-is, but your problem is that you already have Apache on port 80 on the host machine so you've remapped the nginx host ports onto 8080 and 444; is this correct?

it looks to me like you've done that correctly. i'm not entirely sure what is wrong, actually; is there an error log with a message you can post?

[Odil]

Thanks @issa

@issa

from what i understand you are trying to run the docker-compose package we ship with Central exactly as-is, but your problem is that you already have Apache on port 80 on the host machine so you've remapped the nginx host ports onto 8080 and 444 ; is this correct?

Yes that's what I did. I changed ports in order to be able to start docker-compose, otherwise when running docker-compose up I'm getting:

Starting central_mail_1 ... done
Starting central_postgres_1 ... done
Starting central_service_1 ... done
Recreating central_nginx_1 ... error

ERROR: for central_nginx_1 Cannot start service nginx: driver failed programming external connectivity on endpoint central_nginx_1 (08b69e8e174dff4f4ac427b0f5e7c038a662155c900c3aac8620a0e631ba3fa3): Error starting userland proxy: listen tcp 0.0.0.0:443: bind: address already in use

ERROR: for nginx Cannot start service nginx: driver failed programming external connectivity on endpoint central_nginx_1 (08b69e8e174dff4f4ac427b0f5e7c038a662155c900c3aac8620a0e631ba3fa3): Error starting userland proxy: listen tcp 0.0.0.0:443: bind: address already in use
ERROR: Encountered errors while bringing up the project.

However, even after changing the ports trying start docker-compose@central is failed, so I'm getting

docker-compose@central.service - central via docker-compose
Loaded: loaded (/etc/systemd/system/docker-compose@.service; indirect; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2019-05-04 09:37:53 UTC; 3s ago
Process: 27490 ExecStart=/usr/local/bin/docker-compose up (code=exited, status=200/CHDIR)
Main PID: 27490 (code=exited, status=200/CHDIR)

May 04 09:37:53 udaserver systemd[1]: docker-compose@central.service: Service hold-off time over, scheduling restart.
May 04 09:37:53 udaserver systemd[1]: docker-compose@central.service: Scheduled restart job, restart counter is at 5.
May 04 09:37:53 udaserver systemd[1]: Stopped central via docker-compose.
May 04 09:37:53 udaserver systemd[1]: docker-compose@central.service: Start request repeated too quickly.
May 04 09:37:53 udaserver systemd[1]: docker-compose@central.service: Failed with result 'exit-code'.
May 04 09:37:53 udaserver systemd[1]: Failed to start central via docker-compose.

What else can I share with you? I could not find log files.

[issa]

oh hmm, can you try doing docker-compose build again then run it? given the changes you made i'm not sure why it would still be trying to bind to 443, which it clearly is.

[Odil]

I ran:

docker-compose build
docker-compose create
docker-compose up

Now I can access Central in my server with port 444 https://[my server address]:444.

docker-compose ps returns that

However, commands:

systemctl start docker-compose@central
systemctl status docker-compose@central

shows that:

docker-compose@central.service - central via docker-compose
Loaded: loaded (/etc/systemd/system/docker-compose@.service; indirect; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2019-05-07 12:10:27 UTC; 5s ago
Process: 4685 ExecStart=/usr/local/bin/docker-compose up (code=exited, status=200/CHDIR)
Main PID: 4685 (code=exited, status=200/CHDIR)

In the docker-compose.yml file I changed following:

ports:
- "8001:80"
- "444:443"

[Stephen_K_ojwang]

Hi Team, I have been trying out odk central --Odata feed connection with power bi. I have been encountering this querry error. I don't know what could be the problem. I suggest the query doesn't accept variable names with "." dot as one of my variable reads "q1.8". How can I go around this?? If this is the case. Can ODK central accept replace of xforms without losing the already submitted data. This is a nice feature for odk central to have coz many a times form update is necessary as data collection is in progress. I would really appreciate your responses on this. Thanks.

[issa]

if you have run docker-compose up, then the whole system is already being run by your manual startup. in that case, the systemctl runner won't be able to work, because again you're running into conflicts. so you'll have to docker-compose stop and then try systemctl start docker-compose@central again, i think.

[issa]

hey: this sounds like a bug. i will take a look at it in the next few days.

[issa]

i started working on a fix for this but i didn't get it quite right. but i still think you're right that the dot is the problem. i'll have an update for you soon.

[issa]

okay! i have fixed the bug in question and the data loads correctly into power bi from the sandbox server. two notes:

1. if you are going to install/use your own server, you'll have to apply the patch. if so, let me know and i'll give you some instructions; it's very easy to do once you have the server set up.
2. it looks like there's still an issue where power bi expects all fields to be present in the data for every row even if they are empty. so right now it shows "Error" in each empty cell rather than an empty cell. i will work on this problem as well, but i think the data should be usable in the meantime.