Pulling all data through ODK briefcase


(Hanan Aqilan) #1

Is there a way to pull newly records from online cloud server through ODK
briefcase, like pulling data by date or so, because we want to save our
quota limit as most of time we ran out of quota so please let me know if
there another way to do so without exceeding this?


(Yaw Anokwa) #2

Hanan,

I'm pretty sure Briefcase only pulls records that don't already exist
in the Briefcase storage folder.

Yaw

ยทยทยท -- Need ODK services? http://nafundi.com provides form design, server setup, professional support, and software development for ODK.

On Mon, Jul 27, 2015 at 4:21 AM, Hanan Aqilan hjameelq@gmail.com wrote:

Is there a way to pull newly records from online cloud server through ODK
briefcase, like pulling data by date or so, because we want to save our
quota limit as most of time we ran out of quota so please let me know if
there another way to do so without exceeding this?

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Hanan Aqilan) #3

My problem is not the data we get, it lies in the media folder, the images
are very big, so it is about 100 MB for one form. Frankly speaking we do
really need to keep these images stored on our server, but how can I manage
to download only the new ones, or to not download them at all. Is there an
option to not download media along with the csv file.

Any ideas??

ยทยทยท On Monday, July 27, 2015 at 12:21:05 PM UTC+3, Hanan Aqilan wrote: > > Is there a way to pull newly records from online cloud server through ODK > briefcase, like pulling data by date or so, because we want to save our > quota limit as most of time we ran out of quota so please let me know if > there another way to do so without exceeding this? >

(Hanan Aqilan) #4

For the huge forms, we don't have repeats, but the data is encrypted, I
pass private key files when I export it.

ยทยทยท On Monday, July 27, 2015 at 12:21:05 PM UTC+3, Hanan Aqilan wrote: > > Is there a way to pull newly records from online cloud server through ODK > briefcase, like pulling data by date or so, because we want to save our > quota limit as most of time we ran out of quota so please let me know if > there another way to do so without exceeding this? >

(Hanan Aqilan) #5

I will give it a try right away.

ยทยทยท On Monday, July 27, 2015 at 12:21:05 PM UTC+3, Hanan Aqilan wrote: > > Is there a way to pull newly records from online cloud server through ODK > briefcase, like pulling data by date or so, because we want to save our > quota limit as most of time we ran out of quota so please let me know if > there another way to do so without exceeding this? >

(Hanan Aqilan) #6

Here what I got. I tried to export the form data directly from our
aggregate server, but I got 3 options only CSV, KML, and JSON. When I
select JSON and CSV, the filters option gets grayed, but when I select KML
which I have zero idea about, the filter option gets enabled to filter by
title, and picture: media or encryptedXMLFile.

I'm not sure if there is even an option to manage downloading img to save
some quote bandwidth.

We are on our way to get a dedicated server not one instance as we have
right now, but until that happens, we are still in need to handle this
issue.

I appreciate your input.

ยทยทยท On Monday, July 27, 2015 at 12:21:05 PM UTC+3, Hanan Aqilan wrote: > > Is there a way to pull newly records from online cloud server through ODK > briefcase, like pulling data by date or so, because we want to save our > quota limit as most of time we ran out of quota so please let me know if > there another way to do so without exceeding this? >

(Mitch S) #7

Correct. If the data already exists locally in ODK Briefcase's storage
location, ODK Briefcase will not re-download it.

By default, ODK Briefcase always queries for new data from the beginning of
time, so it does quickly sweep through all the prior submissions before
finding new ones; that can consume a lot of quota.

The two choices are:
(1) Keep all the data on the server, and all the data in the ODK Briefcase
storage location. Issue an ODK Briefcase PUSH after each PULL. The PUSH
will configure ODK Briefcase to begin the next PULL request from the time
of the last PUSHed submission. I.e., if you've pulled 1000 records, by
issuing a PUSH, you configure ODK Briefcase so that it will begin its next
PULL from the 1000'th record, not from the 1st record.

(2) Keep only new data on the server, and all data in the ODK Briefcase
storage location. After completing an ODK Briefcase PULL, use the Purge
Submissions button on the Forms Management / Submission Admin tab. This
will delete data before a certain date from ODK Aggregate. The data will be
completely removed and unrecoverable. In this case, you would not issue an
ODK Briefcase PUSH, as that would simply restore the data on the server,
from the copy held within ODK Briefcase.

ยทยทยท On Mon, Jul 27, 2015 at 5:15 AM, Yaw Anokwa wrote:

Hanan,

I'm pretty sure Briefcase only pulls records that don't already exist
in the Briefcase storage folder.

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Mon, Jul 27, 2015 at 4:21 AM, Hanan Aqilan hjameelq@gmail.com wrote:

Is there a way to pull newly records from online cloud server through ODK
briefcase, like pulling data by date or so, because we want to save our
quota limit as most of time we ran out of quota so please let me know if
there another way to do so without exceeding this?

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com


(Yaw Anokwa) #8

Hanan,

https://opendatakit.org/use/aggregate/data-transfer/ has your options.
If you don't have repeats, easiest thing to do is to download the CSV
from the Aggregate server without Briefcase. If you do have repeats,
maybe write a JSON listener and use the JSON publisher to send you the
new text data.

Yaw

ยทยทยท -- Need ODK services? http://nafundi.com provides form design, server setup, professional support, and software development for ODK.

On Tue, Aug 11, 2015 at 7:55 AM, Hanan Aqilan hjameelq@gmail.com wrote:

My problem is not the data we get, it lies in the media folder, the images
are very big, so it is about 100 MB for one form. Frankly speaking we do
really need to keep these images stored on our server, but how can I manage
to download only the new ones, or to not download them at all. Is there an
option to not download media along with the csv file.

Any ideas??

On Monday, July 27, 2015 at 12:21:05 PM UTC+3, Hanan Aqilan wrote:

Is there a way to pull newly records from online cloud server through ODK
briefcase, like pulling data by date or so, because we want to save our
quota limit as most of time we ran out of quota so please let me know if
there another way to do so without exceeding this?

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Yaw Anokwa) #9

Hanan,

I don't know how JSON streaming and CSV export handle encryption. Try
it and see how it goes...

Yaw

ยทยทยท -- Need ODK services? http://nafundi.com provides form design, server setup, professional support, and software development for ODK.

On Tue, Aug 11, 2015 at 8:03 AM, Hanan Aqilan hjameelq@gmail.com wrote:

For the huge forms, we don't have repeats, but the data is encrypted, I pass
private key files when I export it.

On Monday, July 27, 2015 at 12:21:05 PM UTC+3, Hanan Aqilan wrote:

Is there a way to pull newly records from online cloud server through ODK
briefcase, like pulling data by date or so, because we want to save our
quota limit as most of time we ran out of quota so please let me know if
there another way to do so without exceeding this?

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mitch S) #10

If the form is encrypted, you won't be able to use the Export functionality
in Aggregate.

You would need to run ODK Briefcase to download the data to your local
computer.

There is no mechanism to download just the encrypted form. It needs the
form and all of the images in order for the decryption to be successful.

Your local computer will need disk space that can hold 2x the total size of
all your data (once in encrypted form, once in decrypted form).

On some devices, there are settings within your camera app to control the
default resolution of the captured images so that images are appropriately
sized for your use. If you do not need high-resolution images, this may be
a no-coding solution to this problem.

Note that if you are capturing video, there is already a checkbox to enable
or disable high-resolution video capture in ODK Collect's General
Settings. If setting the default image resolution in the camera app is not
possible, adding a similar checkbox for "Limit Image Resolution" would be a
starting point for your changes to ODK Collect (the action taken when this
checkbox is checked would be different from that performed for the
high-resolution video -- see the code in the annotated image widget -- it
does image resizing).

ยทยทยท On Tue, Aug 11, 2015 at 6:28 AM, Hanan Aqilan wrote:

Here what I got. I tried to export the form data directly from our
aggregate server, but I got 3 options only CSV, KML, and JSON. When I
select JSON and CSV, the filters option gets grayed, but when I select KML
which I have zero idea about, the filter option gets enabled to filter by
title, and picture: media or encryptedXMLFile.

I'm not sure if there is even an option to manage downloading img to save
some quote bandwidth.

We are on our way to get a dedicated server not one instance as we have
right now, but until that happens, we are still in need to handle this
issue.

I appreciate your input.

On Monday, July 27, 2015 at 12:21:05 PM UTC+3, Hanan Aqilan wrote:

Is there a way to pull newly records from online cloud server through ODK
briefcase, like pulling data by date or so, because we want to save our
quota limit as most of time we ran out of quota so please let me know if
there another way to do so without exceeding this?

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com


(Yaw Anokwa) #11

Mitch,

Why not change Briefcase to always do a dummy push after each pull so
we save quota?

Yaw

ยทยทยท On Mon, Jul 27, 2015 at 1:39 PM, Mitch Sundt wrote: > Correct. If the data already exists locally in ODK Briefcase's storage > location, ODK Briefcase will not re-download it. > > By default, ODK Briefcase always queries for new data from the beginning of > time, so it does quickly sweep through all the prior submissions before > finding new ones; that can consume a lot of quota. > > The two choices are: > (1) Keep all the data on the server, and all the data in the ODK Briefcase > storage location. Issue an ODK Briefcase PUSH after each PULL. The PUSH will > configure ODK Briefcase to begin the next PULL request from the time of the > last PUSHed submission. I.e., if you've pulled 1000 records, by issuing a > PUSH, you configure ODK Briefcase so that it will begin its next PULL from > the 1000'th record, not from the 1st record. > > (2) Keep only new data on the server, and all data in the ODK Briefcase > storage location. After completing an ODK Briefcase PULL, use the Purge > Submissions button on the Forms Management / Submission Admin tab. This will > delete data before a certain date from ODK Aggregate. The data will be > completely removed and unrecoverable. In this case, you would not issue an > ODK Briefcase PUSH, as that would simply restore the data on the server, > from the copy held within ODK Briefcase. > > > > > > On Mon, Jul 27, 2015 at 5:15 AM, Yaw Anokwa wrote: >> >> Hanan, >> >> I'm pretty sure Briefcase only pulls records that don't already exist >> in the Briefcase storage folder. >> >> Yaw >> -- >> Need ODK services? http://nafundi.com provides form design, server >> setup, professional support, and software development for ODK. >> >> On Mon, Jul 27, 2015 at 4:21 AM, Hanan Aqilan wrote: >> > Is there a way to pull newly records from online cloud server through >> > ODK >> > briefcase, like pulling data by date or so, because we want to save our >> > quota limit as most of time we ran out of quota so please let me know if >> > there another way to do so without exceeding this? >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "ODK Developers" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to opendatakit-developers+unsubscribe@googlegroups.com. >> > For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "ODK Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit-developers+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitchellsundt@gmail.com > > -- > You received this message because you are subscribed to the Google Groups > "ODK Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to opendatakit-developers+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.

(Mitch S) #12

The extra push wasn't automatically done so that ODK Briefcase could
support the purge-submissions usage (2nd option)

A checkbox could be added to offer to do that push, after the pull, in
cases where users are leaving all data on the server.

It has been a long time since I looked at this code; there were a set of
usages that could cause ODK Briefcase to skip some data records; the extra
push eliminated some of them, but I don't remember if there were others.

I decided to simply scan from the beginning of time to eliminate any chance
that we'd get support questions.

ยทยทยท On Mon, Jul 27, 2015 at 12:29 PM, Yaw Anokwa wrote:

Mitch,

Why not change Briefcase to always do a dummy push after each pull so
we save quota?

Yaw

On Mon, Jul 27, 2015 at 1:39 PM, Mitch Sundt mitchellsundt@gmail.com wrote:

Correct. If the data already exists locally in ODK Briefcase's storage
location, ODK Briefcase will not re-download it.

By default, ODK Briefcase always queries for new data from the beginning
of
time, so it does quickly sweep through all the prior submissions before
finding new ones; that can consume a lot of quota.

The two choices are:
(1) Keep all the data on the server, and all the data in the ODK
Briefcase
storage location. Issue an ODK Briefcase PUSH after each PULL. The PUSH
will
configure ODK Briefcase to begin the next PULL request from the time of
the
last PUSHed submission. I.e., if you've pulled 1000 records, by
issuing a
PUSH, you configure ODK Briefcase so that it will begin its next PULL
from
the 1000'th record, not from the 1st record.

(2) Keep only new data on the server, and all data in the ODK Briefcase
storage location. After completing an ODK Briefcase PULL, use the Purge
Submissions button on the Forms Management / Submission Admin tab. This
will
delete data before a certain date from ODK Aggregate. The data will be
completely removed and unrecoverable. In this case, you would not issue
an
ODK Briefcase PUSH, as that would simply restore the data on the server,
from the copy held within ODK Briefcase.

On Mon, Jul 27, 2015 at 5:15 AM, Yaw Anokwa yanokwa@nafundi.com wrote:

Hanan,

I'm pretty sure Briefcase only pulls records that don't already exist
in the Briefcase storage folder.

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Mon, Jul 27, 2015 at 4:21 AM, Hanan Aqilan hjameelq@gmail.com wrote:

Is there a way to pull newly records from online cloud server through
ODK
briefcase, like pulling data by date or so, because we want to save
our

quota limit as most of time we ran out of quota so please let me know
if

there another way to do so without exceeding this?

--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups

"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an

email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com


(Guillermo) #13

2 posts were split to a new topic: Pull submissions starting from date