ODK Aggregate internal server error 500 and Bad Gateway nginx

Greetings,
I am using ODK1 Tools since 2 years. As ODK updates, I moved ODK aggregate from Google cloud to Digital Ocean.
My problem is related to Server . We had installed ODK Aggregate using cloud-config script below given by ODK itself it in Digital Ocean Droplet:
https://raw.githubusercontent.com/opendatakit/aggregate/master/cloud-config/digital-ocean/cloud-config.yml
The ODK server was running smoothly until 4-5 days back it went down with a warning of "Internal Server Error 500" and then we restarted the server now nginx server is giving "502 Bad gateway". we are unable to login to the server itself and fix it.
we open support ticket in digital ocean and here is what they replied.
"
Hi there,

Thanks for reaching out! For third-party software like this, it'll be best to reach out to the person or group who made the software - ODK, it sounds like - they'll be more familiar with it than we are and what troubleshooting steps might be relevant to a given issue or procedure. In general, you would need to check the logs for the upstream application to make sure it's running correctly, but they'll be able to go into more detail.

Some examples of things our Support team are best able to help with include account, billing, or control panel questions, or things like SSH that are the primary interface to a Droplet. Other applications can be a bit difficult for us to troubleshoot since we can't see anything inside customer Droplets, nor do we manage the software inside customers' Droplets.

If there's anything else that we can help with, feel free to let us know!
"
More over,
The odk aggregate contain our house hold survey data more then 10000 including image which makes us worried.
In digital ocean we took 2 core CPU 50GB space.
I am looking for support.

Hi @umesh, it sounds like you aren't able to SSH into the server? If so, the first step here is to reset the root password. https://www.digitalocean.com/docs/droplets/resources/console/ has good instructions on how to do that. Once that's done, then try running the following commands and report what happens.

service tomcat8 restart;
service nginx restart;

Hello @yanokwa,
Actually yes, we are unable to reset password in droplet from above link, so we asked to digitalocean support and they replay as follows"
Thank you for the follow up. If resetting the root password isn't working, there may be issues on the Droplet that are impacting this. In a situation like this, you may need to recover access through other means.

Leveraging the recovery environment to update and troubleshoot issues impacting the root password reset is relatively simple. Starting from the Droplet page, you can power off the Droplet then access the "Recovery" options to "Boot from recovery ISO". Powering on the Droplet after this will run the Droplet from the recovery environment where you can access the menu via the droplet's Web Console, found within your control panel.

Once within the DigitalOcean Recovery ISO, the menu will be available. From this menu, you want to select "Mount your Disk Image" by typing 1 and pushing the Enter/Return key.

After doing this, you will then select "Reset Droplet Root Password" by typing 3 and pushing the Enter/Return key. This will bring you to the Unix passwd prompt, which first asks for your new password. Carefully enter the new password here. After doing so, it will then ask you to re-enter the new password. Once this is completed, you should be brought back to the DigitalOcean Recovery ISO menu.

In some cases, issues such as full inodes or full disk utilization can impact the Droplet root password reset process. To verify this condition, you can access the shell option from the menu and check the /mnt details for both disk space and inode usage:

df -h
df -i

If you see that these are at 100% for the /mnt path ( your Droplet disk ) you will want to explore the file system to clear some space before re-attempting to reset the login. These posts may help outline some steps:


You can also try changing the password via a chroot'd environment. To do this, you will want to select "Attempt to 'chroot' into installed system" by typing 5 and pushing the Enter/Return key. From here, you will be brought into a chroot'd environment of your droplet. You can then use the Linux "passwd" command to change your root password. It will use the same prompts as the "Reset Droplet Root Password" option from the DigitalOcean Recovery ISO.

Once you have completed that, you can boot the Droplet back from the disk. To set this back, simply power off the Droplet once again and use the Droplet > Recovery options to "Boot from Hard Drive"."

and we followed the following steps:

  1. we discovered that the file system /mnt was 100% full i.e 49GB
  2. as all the files were important we did not clear space.
    3.; we resized the droplet to 50GB to 80GB
  3. after the resize was complete we tried to login but could not
  4. we had mounted file system in the recovery ISO and tried to change the password but could not succeed
  5. The ipshows 404 notfound. nginx

So to clarify, you set up a server with 50 GB of space and you sent 10,000 submissions with images to it? At ~4 MB per image, that's 40 GB and the OS is 10 GB. So yeah, running out of space seems very plausible. What do you think we could have added to the documentation to warn you about this?

We are sort of limited in the help we can provide here I'd recommend trying to back up that server and data first before you do anything else.

  1. Turn on Digital Ocean's backups so you have a snapshot of the machine.
  2. Do a PostgreSQL backup and get that off the server. SFTP would help here.
  3. Spin up a new much bigger server, try to restore the backup to that server and see if the data is OK.

thank you for support. My aggregate in running now. The problem was space. With additional space it is running now.

@umesh am getting the same error but cant seem to get around it. how can you assist

I can suggest you that, make back up and try to increase the volume and if it does not work, create new server with high volume space using backup.