1. What is the problem? Be very detailed.
I want a Dynamic selects from pre-loaded data using ODK Central. As a test, I used the example files given in here http://xlsform.org/en/#dynamic-selects-from-pre-loaded-data but when I load the XML in Central, it does not ask me for any media file as it should. However, the same set of XML and CSV files work perfectly when I upload them on ODK Aggregate and use ODK Collect to fill them.
2. What app or server are you using and on what device and operating system? Include version numbers.
ODK Central v0.4 Beta
3. What you have you tried to fix the problem?
I tried using the pulldata() function and it is working fine. Having trouble with the search() function only.
4. What steps can we take to reproduce the problem?
Use the example files provided here: Form definition Media file CSV 5. Anything else we should know or have? If you have a test form or screenshots or logs, attach below.
This is my first post here, so apologies if I missed to mention anything!
Thanks for the detailed report, @saif2khan. I'm pretty sure this is because pyxform doesn't put an instance declaration in the XForm when search() is used. In the short term, I'd recommend adding a calculate row with a simple pulldata(). This won't be visible to the user and will force pyxform to add the instance declaration. It will be part of the submission data so I'd just put it at the end so it's easy to ignore in analysis.
@issa@Matthew_White, I'm guessing Central doesn't have a special case for identifying media from search(). It's a function that goes in an appearance attribute for a select and for which the first parameter is the name of a CSV file with or without the .csv extension. That is, if the appearance is search('fruits'), a fruits.csv file is expected to be attached. Just to make things more fun, the dataset name could be specified by an XPath expression. Sorry. The hope is to fully deprecate it eventually but there needs to be a performant alternative in place first.
The options I see for Central are:
do nothing and document that users should include a pulldata call in their XLSForms
notice that the search() appearance/function is used and try to get the CSV name from the form. If it's dynamically determined, ???
notice that the search() appearance/function is used and let the user add some number of CSVs of their choice
Ah very interesting! If the feature is going to be deprecated before too long, maybe it makes sense to lean toward a low-cost option for now?
Is search() widely used? If not, this sounds like it could be a good option.
One possibility might be a middle-of-the-road option whereby if a CSV name is specified to search(), we add that to the list of form attachments; but if it is dynamically determined, we recommend just adding a pulldata call to the XLSForm. (Do we know how often it is dynamically determined?) That said, if that would require Backend to start parsing XPath expressions, maybe that would be getting to be fairly complex?
I think this is likely the most complex option from the Frontend perspective. (The form attachments page is already quite complex.) But when search() is replaced, do you think the replacement will also have an option for dynamically determining the CSV name, such that it will not be possible to determine from the form alone what the full list of form attachments is? If that's something we will likely need to support eventually anyway, maybe it makes sense to implement it now.
Hey @LN , adding a calculate row with a pulldata() works! Thank you so much Have to say- you people are doing amazing work. ODK is core to many of the applications that we are creating for governments in India, to be used by millions!
Cheers,
Saif
That's wonderful, @saif2khan! It's always great to hear how the tools are used so please consider sharing what you're working on in the Showcase! There are several Indian developers who contribute to the project and I bet they'd be particularly excited to learn about how their government uses the tools.
To answer @Matthew_White's questions regarding next steps -- while search() is fairly widely used (see External data - current state of affairs - #8 by yanokwa for stats), I think letting users know to include a dummy pulldata call is a fine approach now that I've thought more about it. In fact, the two often work hand-in-hand so it will be very common for a form that uses search() to need pulldata for something as well. Let's do nothing for now and consider adding a warning on form upload in Central if a lot of people run into the issue.
Yes, definitely, though there are more parameters possible after the filename. Collect grabs everything between search( and ), splits it on commas and uses the first element as the filename expression. I do imagine it's rare to have something other than a set filename there (should have answered that when @Matthew_White asked).
That's right. Double quotes are allowed too for the literal value case.
I think that's ok because the comma would be in quotes somewhere either as the filename itself or as a string that is part of an expression that evaluates to a filename. So I believe you can grab something after search( that is in a pair of ' or " before a , or ). The Collect implementation does a match on search(.+) and then puts that through the XPath parser to get the values for the different parameters so that's why I can't give you an absolutely definitive answer without diving deeper (I don't know the ins and outs of the parser).
okay so it's not as simple as just splitting on commas, and if the xpath parser is involved i assume quotes may be escaped as well etc. so, all this adds time and complexity but obviously it's not impossible to do.
Escapes are done with XML entities (") exclusively. I think doing something fairly naive and refining if needed would be a great start. (e.g. something like search\(['"](.+(\.csv)?)['"][,\)] or separating the ' and " cases for a little less crazy)
but if other parameters are given, then because regex is greedy this won't work. i'd rather just build the thing that works (likely a rudimentary lexer) than have to spend a bunch of releases firefighting user support issues.
The hope is to fully deprecate it eventually but there needs to be a performant alternative in place first.
Have to say- you people are doing amazing work. ODK is core to many of the applications that we are creating for governments in India, to be used by millions!