ODK Meta - Repeat Groups not merging

@Francisco_Carballo, @aurdipas, have you been able to use odkmeta successfully in the past? Can you tell whether these issues have only recently started occuring?

It'd be quite useful if you're able to upload an example form that demonstrates these two issues (the merge error and the filename issue). From there, I'll see if I can figure out what the source of the issues is.

We're still investigating, but one possibility is that this is related to changes made for Briefcase version 1.11.0. (The current version of Briefcase is 1.12.0.) I've created a GitHub issue about PARENT_KEY and one about filenames.

While those discussions are ongoing, one option is to use the latest version of Briefcase before 1.11.0, which is 1.10.1. When I use that version, the odkmeta do-file doesn't issue any error messages about KEY. The filenames will still differ slightly — Briefcase 1.10.1 exports filenames with underscores, while the odkmeta do-file uses hyphens — but otherwise they should be the same.

Hello Matthew,

Thanks for investigating this issue.

It is the first time using ODK meta, I do not know when they ocurred first.

Regarding file names, in my case the names differ more than only underscores. In the repeat group files, an extra word is included in the Do-File (namely the name of the group), while the file exported from Briefcase does not include the name of the group.

Best regards,
Francisco

Do you still see this if you export the CSV files using Briefcase 1.10.1? If you use that version of Briefcase, I think the only difference should be whether the filenames include hyphens or underscores.

1 Like

Hello Matthew,

Yes, you are right.

With Briefcase 1.10.1 the only difference is hyphens/underscores.

Best,
Francisco

Hi, @Francisco_Carballo, @Matthew_White, @aurdipas !

We've released Briefcase v1.12.1 which includes a fix for some issues related to what you're discussing here :slight_smile:

2 Likes

Thank you for the quick release, @ggalmazor! @Francisco_Carballo, @aurdipas, this should resolve the two issues you were seeing. Filenames are also exported with hyphens between group names, not underscores, which means that the odkmeta do-file should work without you having to modify filenames.

1 Like

Hello @Matthew_White @ggalmazor,

Thanks for the quick fix. File names are good now, and PARENT_KEY and KEY match in the parent and child files.

However, okdmeta still does not run properly. In this case, a problem with the device_id is issued:

  • device_id
    char device_iddevice_id[Odk_name] device_id
    ERROR: variable device_iddevice_id not found

I think it could be related to the changes in hyphens/underscores. I am sure there are easy solutions to it, by tailoring the code to my needs. Unfortunately I have never worked with some of the commands used in the Do-File that "odkmeta" exports.

1 Like

Thanks for reporting this, @Francisco_Carballo. Other than having to modify filenames, were you able to run the odkmeta do-file successfully for exports from Briefcase 1.10.1? Or is this error message also issued for exports from Briefcase 1.10.1?

The latest version of Briefcase changed the delimiter in filenames back from an underscore to a hyphen, but it did not change the delimiter in column headers, which I believe has always been a hyphen. The Stata insheet command removes hyphens from column headers, but odkmeta knows to account for that behavior.

It'd be very helpful if you could share the form you're using or another form that results in the same error message. From there, I can do more investigation into the error message.

8 posts were split to a new topic: Naming of repeat CSVs in Briefcase export in v1.12.1

Hello @Matthew_White and @ggalmazor,

I have to note that this error is only issued when I generate a new odkmeta do-file on my survey and try to run it.. So it might be more related to odkmeta than to a Briefcase version. To be honest, I am a bit confused now.

The code generated by "odkmeta" is somehow different than the code I used couple of days ago when I reported the first issue. I do not know if something has been changed in the user-written odkmeta command, but I am unable to export exactly the same Do-File as I had it before. I did modify something manually myself, but not in that chunk of the code. Anyway, a new-generated code is issuing the error, so it should be taken care of.

As for the questions:

  1. The error is also issued using Briefcase v1.10.1.

  2. I have privately sent you a copy of the form I am using. Please, let me know if there is anything else I can help with.

Best,
Francisco

1 Like

Thanks for sharing your form, @Francisco_Carballo! When I ran odkmeta for the form and ran the resulting do-file, no error message was issued. I also don't see this command in the do-file, which you mentioned above:

char device_iddevice_id[Odk_name] device_id

Instead I see:

char device_id[Odk_name] device_id

@ggalmazor released a new version of Briefcase last week (version 1.12.1), but there's been no recent update to odkmeta.

I would first try re-exporting the survey and choices sheets of your XLSForm and rerunning odkmeta using those. It seems odd that you can't reproduce the do-file that you originally generated.

There's been no recent update to odkmeta, but you can check that you have the correct version by typing which odkmeta in Stata. You should see a file path followed by:

*! version 1.2.0 Matthew White 02sep2014

Hello @Matthew_White,

I am sorry for the confusion. You were right. I re-exported new survey and choices sheets and it runs the code with no issues.

From my side, we can consider the thread closed.

Thanks everyone for your help.

Best.

2 Likes

Hello @Matthew_White and other collegaues here,

I collected my data and now need help merging it using ODK Meta in preparation for analysis in stata. The data has two repeat groups (each with several questions) and on exporting it using briefcase, i get three three separate csv files i.e the parent file and the two (one for each repeat group). When I have had one csv file, the following command has helped with minimal challenges

odkmeta using import.do, csv("ODKexample.csv") survey("survey.csv") choices("choices.csv")

Now that I have three files which need to be merged together into one single dataset, which command do I use or how do I mordify the command above to get me what I need. Friends, I would like to put a disclaimer that I am not tech-savvy and would appreciate keeping the guidelines really simple. Much appreciated

Hello guys,
Without being too demanding, but any help with my issue above? @Matthew_White you have always helped me out in such situations

Hi @Musinguzi_Polycarp!

odkmeta will automatically detect repeat groups in your XLSForm and will add merge and reshape commands to the do-file. In other words, you don't need to specify anything extra for odkmeta to work with repeat groups. If you do run into issues, there are some threads on the forum related to odkmeta and merging that may be helpful. (odkmeta certainly does not require you to be a Stata expert, but it may require some troubleshooting.)

Hope that helps!

Hi @Matthew_White ,

Thanks a lot for your guidance. However, I have run into another problem (see attachment below).


I have read through some threads and they seem to indicate that this issue could be related to having duplicate variable names as a result of splitting multiple choice questions. However looking at that Q112a, it wouldn't result into any duplicates since there isn't any other variable named Q112a1. In fact, there's another multiple choice question in that repeat group (Q112d) that triggers the same error.

Any thoughts on what's causing this and how to make it right?. Thanks

1 Like

I'm not sure I've ever seen this error message outside the context of duplicate variable names. I'm not sure what the issue is, but looking at the output, it seems that Q112a may have all missing values. Maybe split can't be run on such a variable? (Is it expected that no data was collected for that variable?)

If that's the case about split, then the logic in the do-file will need to change slightly. I'll think more about how to modify that logic. I think a workaround for now would be to remove Q112a from your XLSForm before exporting the survey sheet to CSV. Then the do-file won't recognize Q112a as select_multiple and won't try to split it.

Hello Mathew,

Thanks for your quick response. Actually, as per the data that has already been submitted, Q112a and Q112d have a few responses. However, if both Questions are removed from the CSV, odkmeta runs successfully. Not sure what could be causing this

Hello,
What do you mean to re export the survey and choices. I have an issue of assertion is false
When I run the do file from odkmeta. for me I have just saved survey sheet as CSV and choices sheet as CSV. Am I using the right methods or there is a link that explain how to create survey and choices when you are going to use odkmeta.

Thank you!