Syncing takes too long

Hi,

It has been a month that our survey has started and since then we have collected thousands of records.
As the time pass by the sync operation gets heavier and heavier.

Now it takes about 1 hour to sync the data to the server, even sync without attachments take this much time.

Early in the survey, we didn’t apply row level access, but now we have it, is this because of that issue or anything else we are missing?

@elmps2018 @linl33

Thanks.

I think the solution here depends a bit on whether you need all the records to remain on the server and tablets. One thing we did when fielding was, after taking careful check that we had downloaded all the data with suitcase, we deleted completed data off the server which reduced the syncing time. Not sure if this would work in your case.

Thanks for the reply, the method you mentioned should surely work, however in our survey, one of our requirements was that the completed instances should be visible as well to the surveyor, so we manually marked the “_DELETED” column to ‘f’ for all the instances, and this was the cause for that long sync time.

We tested it with the ODK-X normal i.e “_DELETED”=‘t’ for “_SAVEPOINT_TYPE”=‘COMPLETE’, and it synced as normal.
We are about to bring some changes in our survey policy to coop with this situation.

Thanks.

Greetings,

@elmps2018, we have faced another serious issue in the sync process.

As I mentioned in the question above that we didn’t implement the row level access in the beginning and updated those rows through custom queries in database: setting "_DEFAULT_ACCESS" = 'HIDDEN'.

But now when we sync a device, all the rows in a table are synced to the tablet, even the COMPLETED and "_DELETED"='t' ones as well, which is taking a lot of time.

We have 14 tables and the average records per table are like 20K - 25K.

Worth to mention, that the first time when there is nothing in the device, the sync takes like a minute or two, but for the second time, it syncs everything that is on the server.

Apart from removing data from the server, is there any other way to solve this issue?

Thanks.

There are several approaches it depends on what is best for your situation. Hidden rows are still sync’d because depending on the permissions of groups or SUPER_USER users can still see the rows.

IF there are attachment like pictures you can also have users sync without attachments unless they are in high connectivity areas.

  1. There are “local” tables that do not sync at all so data you do not want sync’d you could put there
  2. You can store read only data in CSVs, and use the query type CSV
  3. Delete the records you no longer want sync’d

The problem is the protocol is checking to see if there are any changes to the rows since last sync. Without that check it hard to make sure things aren’t missed.

Thanks for the reply.

Actually we have resolved the issue by checking the “TableName_LOG” tables which I guess the http://web-url/odktables/default/tables/table_name/ref/{ data-etag }/diff reads from there.

There were still rows that had _DEFAULT_ACCESS = 'HIDDEN', we have fixed that issue and now everything is back to normal.