ODK Briefcase v1.16 Beta

Hi @ggalmazor

one more thing, I have an issue in Arabic characters in the exported csv file, I've tried it in both beta and release of ODK-Briefcase v 1.16.0.

Note: I changed character encoding to 65001 Unicode (UTF-8), but its not working.

Thanks

To import CSVs into Excel, you cannot download and open in one step; nor can you double-click on the CSV. You must open Excel and choose Import. If you are asked, the file origin or encoding is UTF-8. Can you try that and see if that works?

I've sent in a pull request to add this to the docs. https://github.com/opendatakit/docs/pull/1067

1 Like

Thanks @yanokwa for your rapid response.

I always did the same steps as you explained and it works for all releases of Briefcase prior to v 1.16. But in ODK-Briefcase v1.16 it doesn't work.

Many thanks

Hi, @rassam! Could you send me a form and some submissions to test it?

Hi, @ggalmazor Attached is the form and ODK Briefcase Storage containing submissions.

Many Thanks.

KII.zip (957.8 KB)

Thanks! I'll try to reproduce the issue

Hi, @rassam!

I would like to confirm what I'm seeing in the submissions you've sent me. Take the submission with UID uuid:029be172-443e-4c5f-8c42-744a4f14de44, for example. I have this XML file:

<KII_C5 id="KII_C5" instanceID="uuid:029be172-443e-4c5f-8c42-744a4f14de44" version="1906251040" submissionDate="2019-07-01T19:49:09.956Z" isComplete="true" markedAsCompleteDate="2019-07-01T19:49:09.956Z" xmlns="http://opendatakit.org/submissions">
  <start>2019-06-30T18:33:24.934Z</start>
  <end>2019-07-01T19:48:06.678Z</end>
  <deviceid>353754070200048</deviceid>
  <subscriberid/>
  <simserial/>
  <username>YE1046</username>
  <KII_note/>
  <A>
    <KII_Name>&#216;&#174;&#216;&#167;&#217;&#8222;&#216;&#175; &#217;&#8230;&#216;&#173;&#217;&#8230;&#216;&#175; &#216;&#185;&#217;&#8222;&#217;&#352; &#216;&#167;&#217;&#8222;&#217;&#8230;&#216;&#177;&#217;&#352;</KII_Name>
    <Position>&#216;&#185;&#216;&#167;&#217;&#8218;&#217;&#8222; &#216;&#173;&#216;&#167;&#216;&#177;&#217;&#8225;</Position>
    <Phone_no>777208029</Phone_no>
    <Governorate>Aden</Governorate>
    <District>Al_Buraiqeh</District>
    <Ozla/>
    <Village/>
    <Area>Urban</Area>
    <Date_of_Interview>2019-07-01</Date_of_Interview>
    <Time_of_Interview>21:33:24</Time_of_Interview>
    <U_think_UR_eligible>No</U_think_UR_eligible>
  </A>
  <interview>
    <B>
      <Age>54</Age>
      <Gender>Male</Gender>
      <Educational_Status>Secondary</Educational_Status>
      <Years_of_education>9.0</Years_of_education>
      <KII_Profession>&#216;&#185;&#217;&#8230;&#217;&#8222; &#216;&#174;&#216;&#167;&#216;&#181;</KII_Profession>
    </B>
    <C>
      <C.Q1>Saw_the_poster</C.Q1>
      <C.Q1.1/>
      <C.Q2>More_than_one_month</C.Q2>
      <C.Q3>
        <generated_table_list_label_35/>
        <C.Q3.1>&#217;&#8222;&#216;&#167;&#217;&#352;&#217;&#710;&#216;&#172;&#216;&#175;</C.Q3.1>
        <C.Q3.2>&#217;&#8222;&#216;&#167;&#217;&#352;&#217;&#710;&#216;&#172;&#216;&#175;</C.Q3.2>
      </C.Q3>
      <C.Q4>Help_the_people_in_Yemen</C.Q4>
      <C.Q4.1/>
      <C.Q5>Former_SWF_beneficiaries</C.Q5>
      <C.Q5.1/>
      <C.Q6>This_is_a_new_UNICEF_project</C.Q6>
      <C.Q6.1/>
      <C.Q7>UNICEF</C.Q7>
      <C.Q7.1/>
      <C.Q8>To_buy_food To_pay_for_health_costs</C.Q8>
      <C.Q8.1/>
      <C.Q9>They_are_happy_to_receive_cash</C.Q9>
      <C.Q9.1/>
      <C.Q10>No</C.Q10>
      <C.Q10.1/>
      <C.Q10.1.1/>
      <C.Q11>No</C.Q11>
      <C.Q11.1/>
      <C.Q11.1.1/>
      <C.Q12>I_dont_know_enough_about_the_payment_process_to_allow_myself_any_judgment_on_it</C.Q12>
      <C.Q13>No</C.Q13>
      <C.Q13.1/>
      <C.Q13.1.1/>
      <C.Q14>Yes</C.Q14>
      <C.Q15>SMS</C.Q15>
      <C.Q15.1/>
    </C>
    <D>
      <D.Q16/>
      <D.Q16.1>No</D.Q16.1>
      <D.Q16.1.1/>
      <D.Q16.2>No</D.Q16.2>
      <D.Q16.2.1/>
      <D.Q16.3>No</D.Q16.3>
      <D.Q16.3.1/>
      <D.Q16.4>No</D.Q16.4>
      <D.Q16.4.1/>
      <D.Q16.5>No</D.Q16.5>
      <D.Q16.5.1/>
      <D.Q16.6>No</D.Q16.6>
      <D.Q16.6.1/>
      <D.Q16.7>No</D.Q16.7>
      <D.Q16.7.1/>
      <D.Q17>&#217;&#8222;&#216;&#167;&#217;&#352;&#217;&#710;&#216;&#172;&#216;&#175;</D.Q17>
      <D.Q18>&#217;&#8222;&#216;&#167;&#217;&#352;&#217;&#710;&#216;&#172;&#216;&#175;</D.Q18>
    </D>
    <E>&#217;&#8222;&#216;&#167;&#217;&#352;&#217;&#710;&#216;&#172;&#216;&#175;</E>
  </interview>
  <Supervisor_name>Asel_Ali_Mohammed_Soliman</Supervisor_name>
  <Enumerator_ID>YE1046</Enumerator_ID>
  <n0:meta xmlns:n0="http://openrosa.org/xforms">
    <n0:audit>audit.csv</n0:audit>
    <n0:instanceID>uuid:029be172-443e-4c5f-8c42-744a4f14de44</n0:instanceID>
  </n0:meta>
</KII_C5>

Take a closer look to the answer for the KII_Name field: <KII_Name>&#216;&#174;&#216;&#167;&#217;&#8222;&#216;&#175; &#217;&#8230;&#216;&#173;&#217;&#8230;&#216;&#175; &#216;&#185;&#217;&#8222;&#217;&#352; &#216;&#167;&#217;&#8222;&#217;&#8230;&#216;&#177;&#217;&#352;</KII_Name>

As you can see, all the characters are encoded as their escaped UTF-8 entity codes. Using an online decoder, I can translate that list of characters to this: خالد محمد علي المري. This matches what Briefcase v1.16 exports in my case. I've verified that Briefcase v1.15 produces the same file.

Taking this results into account, my guess at this moment is that there's something wrong in the source submissions. @rassam, could you double check the ZIP file you've sent me? Maybe you could push the form and submissions to the sandbox Aggregate server at https://sandbox.aggregate.opendatakit.org so that I can pull it with older versions of Briefcase to see if I get different submission contents.

Hi, @ggalmazor

I guess the issue is in the pull process using ODK-Briefcase v1.16.0, when I pulled the same instance using ODK-Briefcase v1.15.0 it produces a a submission file with correct encoding.
Attached is the same submission with UID uuid:029be172-443e-4c5f-8c42-744a4f14de44 that I pulled using ODK-Briefcase v1.15.0.

Many thanks.submission.xml (2.6 KB)

1 Like

Hi, @ggalmazor I,ve uploaded the form and two submissions https://sandbox.aggregate.opendatakit.org. so that you can test it.

Many thanks.

2 Likes

That would make sense.

Testing it, thanks! :slight_smile:

OK, this is what I've done (all from a clean state: no prefs, empty storage directory):

  • First, pull the form with v1.15 and then export it with v1.16 works
  • Second, pull and export the form with v1.16

Both scenarios worked OK. My computer uses es_ES.UTF-8 locale

I'm guessing that your default encoding wasn't UTF-8 when you pulled the form with v1.16. Could you confirm this?

Hi, @ggalmazor Thanks for your valuable support.

I think that the issue is in my computer, windows OS, I've tried it in Ubuntu and it worked correctly.

Thanks! :slightly_smiling_face:

1 Like

I tried to use this briefcase version on a 1.4.12 ODK Aggregate server to download a form that has a couple of repeats.
This works perfectly with the previous version of Briefcase.

Now with this version when I try the export I get an error (see attached log).
briefcase.zip (1.0 KB)
If you need I can forward you the pulled submissions privately.

Best,
Aurelio

1 Like

I'll try to reproduce the issue on Windows, just in case :slight_smile:

Thanks, @aurdipas! I'll take a look to the issue you're reporting today :slight_smile:

1 Like

I've filed a fix for what @aurdipas is reporting https://github.com/opendatakit/briefcase/pull/760

We had an issue under particular encofings of time zone offsets in time field values.

Hi, @rassam!

I've reproduced the problem you're reporting on a Windows 10. I'll try to do something about it.

Hi, @rassam!

It's kind of hard to track how encoding works when dealing with Briefcase and other apps in Windows, but I wanted to try forcing Briefcase to read the pulled submissions using UTF-8 and I got some confusing results. Here's a screenshot:

You can see here:

  • The Windows notepad app can open the exported file and shows (hopefully correct arabic characters.
  • The powershell terminal doesn't show arabic characters
  • The bash (emulated Linux on Windows) doesn't show arabic characters
    • This one is particularly confusing because I verified that I was using the same locale as in my main Ubuntu machine
    • The blanks might be telling us that the arabic characters are there but the font I'm using doesn't have them, although I can't be sure.

OK, false alarm: I copied the exported file to my Ubuntu machine and now the arabic characters are there.

This proves that we can improve Briefcase. I'll file an issue and, hopefully, it should be released in v1.16.1 tomorrow

1 Like

Hi, @ggalmazor

Great!

Thanks for your valuable support :slightly_smiling_face:

2 Likes

Heads up, @rassam, @aurdipas! We've just released Briefcase v1.16.1 with the fix for the issues you have reported. Thanks for your help!!

1 Like