I’m the admin of the lou.lt instance and I’m currently moving all my activities on a new server.
My old server was running Ubuntu Server 16.04 and Mastodon was set up on it using an old version of the Production Guide but everything was running mostly well apart from some issues with updates (haven’t been able to do updates for a little while).
I used the Migration Guide to move everything on the new machine and I had no issues at the time.
The new server
- Ubuntu 18.04
- Node JS v8.10.0
Also, I’m using vmware ESXi on my new server and everything is running inside VMs. I made something like this:
Almost everything seemed to run smoothly at first, I only had to deal with a small issue with the uws node module. At first, the mastodon-streaming service refused to start and I had a “
Error: Compilation of µWebSockets has failed and there is no pre-compiled binary available for your system. Please install a supported C++11 compiler and reinstall the module 'uws'.” in the logs.
After installing gcc-8 and running
node-gyp rebuild by hand in the uws module directory, the service seems to be able to run correctly.
But here start the real issues
When I start the three services
mastodon-streaming, the whole VM randomly crash after a time going from a few minutes to 24 hours (but most of the time, it’s under 5 minutes).
And when I say the whole VM I mean it. The VM won’t answer to ping and is unresponsive in the ESXi admin panel. Memory and CPU usage show no issue and stay linear from the moment the VM has crashed.
I tried to investigate the issue but the only thing I could see is a bunch of “^@” characters in the syslog file at the moment of the crash.
I also have sometimes some
ERR! c9b13871-2815-40ac-928c-ffe15f26cb8a Error: Missing access token in the mastodon-streaming logs. They may be an issue but seems unrelated to the vm crash issue to me.
I did some test and had the following results:
- When the three mastodon services are down, the VM is stable.
- When only the
mastodon-webservice is running, the VM is stable.
mastodon-sidekiqare running, the VM crashes.
mastodon-streamingare running, the VM seems to be stable (but only based on a still running 1 hour long test)
I’m thinking I have an issue with the
mastodon-sidekiq service but nothing in the logs seem bad to me.
I don’t know what to check now or what to do. I did a lot of other test I did not describe here but I can provide any log or test result you need to help me.