Pushing/pulling forms to/from android devices directly from a pc

Hi there,

Although I have published this as part of an answer to https://forum.getodk.org/t/pushing-form-from-computer-to-tablet/6991, I think it is useful to repeat it here as all of us have been confronted to these issues, especially when not using the Internet/ODK Aggregate.

I have successfully tested the following procedure to push a new form to my Android MashMallow smartphone:

  1. Install usb driver for android from the app store in your smartphone, The "usb driver for android" app icon appears on the phone screen
  2. Connect your phone with a usb cable to your pc
  3. tap on the "usb driver for android" app's icon's which shall lead you to the setup screen. Do not forget to put USB Debugging on "ON". This will allow your pc to detect your phone and it will install a driver for it (takes a few minutes on my Windows 10 64 bit pc)
  4. Install google's Android Studio app in your pc. It will create a "C:\Users\yourusername\AppData\Local\Android\sdk\platform-tools" folder. I created a link to this folder on my desktop not to have to remember the place!
  5. Click on the link to get to this folder where adb.exe is located,
  6. Shift-right mouse button click to open a command line in this folder
  7. copy adb push your-form.xml /sdcard/odk/forms in the command line then click Enter

The same procedure works if you want to push the form to a simulator if you have created one with Android Studio. I always do this during the debugging/testing phase instead of using my real phone.

To pull (download) the odk directory from your phone (or simulator) use:

adb pull /sdcard/odk/ yourpreferred_location_in_your_pc_

Please note that for pushing as well as for pulling, the phone should not be connected to pc while you're working with a simulator and vice-versa.

regards

1 Like

Thanks for writing this up, Souirji! I had tips I wanted to add:

  1. Unless you are a software developer, installing Android Studio isn't needed. Also, it's a big download (450 MB). You can instead download the ADB tools as a standalone download in SDK Platform Tool (8 MB). I've also had great luck with Minimal ADB and Fastboot (2 MB).

  2. If you can't find a USB driver for your device, you can try this Universal ADB driver.

1 Like

hi @yanokwa,

Thank you very much for adding these useful tips to make the procedure lighter.

Is it possible to build an Android emulator for forms' testing with only the SDK Platform Tools? Of course the emulator is not necessary for pushing/pulling forms from a physical android device but it is handy for rapid form testing.

The Android emulator only comes with Android Studio. GenyMotion has a pretty good alternative, but it's licensed for personal use only.

1 Like

Hi @yanokwa,
Thanks! That's why I have installed it. Good to know about the other tools you mentioned.

2 posts were split to a new topic: Export now to csv the form name is not available

A post was split to a new topic: How can I pull data directly from devices?

Hi Souriji and Yanokwa,

I hope you are good and thank you for writing this guidelines.

I am trying to pull the answer from the surveys in a csv file from ODK collect using ADB. I have downloaded Android studio and I am using the following command line:

adb pull /sdcard/odk/instances/. /Users/Olimpia/Desktop/Rawdata

However, when I got to the folder specified in my directory (Rawdata), only the survey (xml forms) are pulled.

Do you know why this happens? How can I pull the finalised forms instead of the xlm forms?

I am using a mac terminal and I have an ODK collect version v1.28.2. I have attached a screenshot of the code I am using for your reference.


Any help is really appreciate it!
Many many thanks.

Hi @Olimpia

I am trying to pull the answer from the surveys in a csv file from ODK collect using ADB.

we save forms using xml files not csv so if you pull files from instances folder you pull xml files. Csv files are also used to save audit files.

org.odk.collect/android should be org.odk.collect.android. See https://docs.getodk.org/collect-adb/#collect-directory.

To pull just the instances, pull /sdcard/Android/data/org.odk.collect.android/files/instances/*.

To build a CSV from all the submissions pulled from devices, you can use Briefcase.

Hi @Grzesiek2010 @LN

Many thanks for your answers.

I see what you mean. However, I still have a question.

At the moment, I am doing the following:

  1. Plugging in my Android phone to my laptop where adb studio is installed

  2. Use the adb pull command to transfer the forms to a designated folder in my laptop
    (code: $ adb pull /instances)

  3. Use the pull forms from collect command
    (code: $ java -jar {path/to/briefcase-jar-file} --pull_collect --storage_directory {path/to/briefcase-storage-location} --odk_directory {path/to/unzipped-odk-file})

  4. Convert forms in csv format using the following command
    (code: $ java -jar {path/to/briefcase-jar-file} --export --form_id {form-id} --storage_directory {path/to/briefcase-storage-location} --export_directory {path/to/output-directory} --export_filename {output-file-name.csv})

However, this creates a csv file for each completed survey. When I am doing field work, I would like to have one csv file for all the answers collected in one day. Do you know if there is a way to combine the forms for the different survey completed?

Thank you in advance for your help!

Best,

Olimpia

1 Like

The answer is here above:

So if you copy the instances to your laptop first, you can then use Briefcase to pull and export them into a single CSV (and export any media files).

To be honest, I am much more Luddite - I don't use adb, I just plug my phone in to the laptop and copy the instances folder to a drive on the laptop then use Briefcase...

It would all be much easier if Briefcase was allowed direct access to the phone (i.e. if it were recognised as a drive) - but this is a very old problem. And there is then the issue of managing that space on your drive:

Hope this helps.

1 Like

Thank you very much for your responses, they have all been very useful inputs.

In the end, have written a shell script that can be run in the terminal to simply pull the forms and export them in Csv. This uses briefcase but in the CLI format so that there is no need to do it manually everytime.

Now that I have exported my data, however, I have another question. I am conducting a household survey were a number of question (e.g. the age, tribe, religion etc) are repeated for all the household's individuals. Thus, my xls form has a number of "begin repeat" functions. When exporting the data into .csv, the file with all the data combined looks like this (see attached). In the file, the answers for the repeated questions are like "uuid:0b78d68e-cadd-4abe-a558-32484237700d/function_repeat_hh_mem_details" instead of being the single questions with the respective answer. Do you know why this is and how to solve this?

Day1_HH_survey.csv (8.1 KB)
Many thanks,

Olimpia

If you were to export the data manually using Briefcase, you would see that it creates a number of csv files - one for each 'level' of repeat. Forgive me if I'm telling you something you already know, but think of the uuid: as a 'foreign key' in a series of relational tables. So I usually build a spreadsheet or database with lookups for those keys - you may need to do the same.
There might be a way to 'flatten' the dataset, but you'd obviously have the 'parent' data repeated multiple times, which might be just as hard to read / analyse.

I haven't used Briefcase in CLI so I'm not sure what script would look like or how to get all the data (i.e. each csv) that way, just explaining the data structure as I think I understand it. This doesn't solve it, sorry, but it might point you towards rewriting your script.