Unit testing for ODK Collect/XLSForms

Ideas
1

What is the general goal of the feature?

One thing I tend to come across when working on ODK Collect forms is that comprehensive testing becomes very difficult. It's also time-consuming, particularly when all answers are required and there are a lot of dependencies and skip logic - we have to go through the form filling out all of the previous items again in order to get to the piece we're interested in. Similarly, when items are changed slightly or the form is restructured (adding or changing the scope of relevant fields, groups and repeats in particular) it can have unanticipated knock-on effects.

It seems to me that this would be a case for unit testing. It would be great to see some version of unit testing incorporated for ODK Collect forms.

What are some example use cases for this feature?

I'd envision for example a case where the skip logic/visibility for a particular question p depends on the input to a previous question q where the answers can be 1-10. Say I change how q works - I expand the constraint so that now 0 is allowed, but I haven't remembered that p, way down at the end the form, doesn't make sense now if ${q}=0. I'm envisioning a scenario where I can specify unit tests that show what would happen in each case, which questions would be displayed and what the results would be. Then when I add my new expanded q, I can just modify an existing set of tests with q set to 0, and get a report to see which questions/groups are displayed and what the results are. I don't have to do a full manual re-test of the form following all the branches of concern to their various ends.

In a slightly different version, say I already allow 0 as an answer to q (say this is the number of eligible respondents in the house), but I have the rest of the questions are in repeat called rGroup, set to repeat ${q} number of times - so each eligible person will do the full survey. Having ${q}=0 here just means that the survey finishes immediately with not further questions asked. Then say I change my whole structure so that I now select one person randomly from eligible respondents. Now I only have a few questions in that rGroup for everyone (getting their names for example), but the rest of the questions have been moved outside the repeat so they only appear once, for the selected person - however they still shouldn't appear if there is no one eligible in the household. I should add a relevant around the non-repeated questions to keep them from appearing if ${q}=0, but say I forget - if I had a previous test from first version with the result of q set to 0, it would show me that all of my non-repeated questions are now showing where they wouldn't have before. (It would presumably come up with a lot of other changes too since the structure's been changed so much!)

What can you contribute to making this feature a reality?

I don't know if there is already a way to do this using existing unit testing tools? It would be great though to have the ability to do it with the same ease as writing an XLSForm.

I probably wouldn't be able to help build the feature unfortunately as I lack the coding skills, but I'd be excited to help test it.

2

I would love for this to exist, but I don't know a good way to make it happen. I've invited some of cleverest people in the community to this topic to see if they have some ideas.

3

If I understand correctly, the goal would be able to check how some particular question would turn our after filling out a form up to a point, and we want to somehow feed predefined answers to avoid entering them manually in Collect.

It seems like this would be needed in a context where a form maintainer is creating or updating a form, and that it would be used to check some question and maybe fix the form when problems are detected. After a form has been reviewed and released, this tool wouldn't be used until the next update.

@Bramleyl, @yanokwa, could you tell me if I got the goal and context right?

1 Like
4

Hi @yanokwa and @ggalmazor, thanks for picking up on this!

I was thinking of it being used when the form is being written or updated by the maintainer, yes. Ideally (in my view), it wouldn't just be used to check a particular question, but the whole form "state" if you like - which questions appear, any constraints that pop up, and the result of each question - given a particular set of inputs. Then every time you add a new question or change the logic, you would just need to add a couple tests and/or re-run the old ones to make sure any state changes for each one are as expected. If so, you could update your old tests to accept the new state. If not, you would know that you have a problem to fix.

Something like this would really help me as a form designer - it would make the testing of new forms or updates much quicker, and it would be more likely to catch the kind of errors that sneak in when trying to add new options, cleverer skip logic, constraints, etc.

1 Like
5

What I would do to meet the requirements you describe is adapt some of the existing Espresso tests from Collect to your particular form(s). This will require installing the free Android Studio environment but should be approachable to fearless non-programmers.

The tests to look at are in AllWidgetsFormTest. That test file opens the All Widgets form and swipes through it question by question, verifying that values are set and that the user interface updates as expected. One possible barrier is that not all tests are populated yet but that's something that can be remedied.

What I would do is identify the part(s) of my form that are complex and that I'd like to have some verification on and have the test jump to the first question of that part and then swipe though, verifying things along the way. I could then change the form in other parts as I wish with the confidence that the tested behavior remains the same.

The test structure will be tightly coupled with the structure of the form but I think that will always be the case with this kind of test.

4 Likes