monerod crashes after several minutes with no errors

Question

I'm trying to run monerod on a VPS and after several minutes of working correctly the program just stops. There are no logs which indicate any errors or warnings in .bitmonero/bitmonero.log even with log-level 4. I've tried to go through my system logs and there's nothing I can find that indicates what is happening either. The only hints I have is sometimes I get the following message when running "./monerod status" right before the moment it crashes:

bash: fork: Cannot allocate memory

It seems pretty straightforward that there's a memory problem, but there are no logs indicating this or the cause. Ulimit reveals that each process can use unlimited virtual memory, so that's not the problem. I have 3 GB of ram and 2 GB of swap space. When running top or htop neither indicate that much memory is being used at all. The df tool shows that I have 23GB of hard drive space remaining, as well, so it's not as if I've run out of storage. The free tool suggests that I have about 4 GB of memory available even when monerod is running before it crashes. I've updated my operating system and restarted it with no changes in behavior. Changing the number of threads spawned and input/output connections can change the timing but ultimately monerod will still fail. On this VPS I don't have access to logs like /var/log/kern.log, unfortunately.

If it helps, I'm using Ubuntu 16.04.4 LTS.

Anyway, I've really exhausted every option I could find. None of the logs available seems to indicate that there is anything that is failing. This problem is stopping me from running a monero node altogether.

Answer 1

So I figured out what was going on.

My environment was a VPS and the virtualization environment was from OpenVZ, for which all users share a kernel. So access to these kernel logs were disabled for any individual users. I had to contact the VPS administrators and inside the kernel logs monerod was exceeding the limits of kmemsize, so they changed it to unlimited.

It also turned out that the file /proc/user_beancounters had information that would have revealed this. it contains information for container limits, one of which was kmemsize. There's a column showing the number of times a program was killed due to exceeding these limits. Anyway, that number was pretty high, suggesting kmemsize was the problem as well.

Now monerod works fine.

Answer 2

The df tool shows that I have 23GB of hard drive space

That's the hard drive issue,

60+ Gb - ok,

80+ Gb - to be safe.

Synced monerod last week from scratch, more than 50 Gb space used, downloaded more than 100 Gb in process.

Also in a weird way number of virtual cores matter. So, if you can switch hard drive between instance types, and want faster sync, do that, it's possible to sync < 2 hours.