I crashed my Mastodon Instance and is unable to recover

I had been using mastodon as my personal front site for about a year. I had set it up and loved how it was working until few days back I crashed it and cant understand how to fix it. Any help would be very much appreciated.

A crashed server - No Disk Space

When I first setup the server I missed a very critical step and forgot to add the cron job to clear the remote media at regular intervals and did not check the situation for long.

Last week I noticed that the server was not responding and when I checked the entire 40 GB of space on my cloud server. All system including nginx, postgres and mastdon had all come to a halt.

I tried clearing the cache with tootctl but it raised a ruby not installed error. Many other commands also refused to work.

As I could not turn of the instance I decided to delete some of the image files in the /public/system/ folder manually.

I managed to free some of the space and tried to fix the ruby problem. However the instance would not wake up and started reporting a 500 error.

Migration Installation

I decided to try a migration installation and carried out all the processes as guided.

I copied the .env.production and /public/system to the new directory setting after I had completed installing.

However when I ran RAILS_ENV=production ./bin/tootctl feeds build I was faced with this

Regenerated feeds for 0 accounts

The running page currently return a 502 Error and the Mastodon Themed error page.

My journalctl -u mastodon-web reads like this

Apr 30 17:56:29 Superdark bundle[11297]:                 LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
Apr 30 17:56:29 Superdark bundle[11297]:                 LEFT JOIN pg_type t ON a.atttypid = t.oid
Apr 30 17:56:29 Superdark bundle[11297]:                 LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation
Apr 30 17:56:29 Superdark bundle[11297]:                WHERE a.attrelid = '"ip_blocks"'::regclass
Apr 30 17:56:29 Superdark bundle[11297]:                  AND a.attnum > 0 AND NOT a.attisdropped
Apr 30 17:56:29 Superdark bundle[11297]:                ORDER BY a.attnum
Apr 30 17:56:29 Superdark bundle[11297]: ):
Apr 30 17:56:29 Superdark bundle[11297]: [20e991cc-98ed-49d4-9d44-33428be6c323]
Apr 30 17:56:29 Superdark bundle[11297]: [20e991cc-98ed-49d4-9d44-33428be6c323] app/models/ip_block.rb:31:in `block in blocked?'
Apr 30 17:56:29 Superdark bundle[11297]: [20e991cc-98ed-49d4-9d44-33428be6c323] app/models/ip_block.rb:31:in `blocked?'
Apr 30 17:56:29 Superdark bundle[11297]: [20e991cc-98ed-49d4-9d44-33428be6c323] config/initializers/rack_attack.rb:46:in `block in <class:Attack>'
Apr 30 17:57:49 Superdark systemd[1]: Stopping mastodon-web...
Apr 30 17:57:49 Superdark bundle[11297]: [11297] - Gracefully shutting down workers...
Apr 30 17:57:51 Superdark systemd[1]: Stopped mastodon-web.
Apr 30 17:57:51 Superdark systemd[1]: Started mastodon-web.
Apr 30 17:57:52 Superdark bundle[11603]: [11603] Puma starting in cluster mode...
Apr 30 17:57:52 Superdark bundle[11603]: [11603] * Version 5.0.4 (ruby 2.7.2-p137), codename: Spoony Bard
Apr 30 17:57:52 Superdark bundle[11603]: [11603] * Min threads: 5, max threads: 5
Apr 30 17:57:52 Superdark bundle[11603]: [11603] * Environment: production
Apr 30 17:57:52 Superdark bundle[11603]: [11603] * Process workers: 2
Apr 30 17:57:52 Superdark bundle[11603]: [11603] * Preloading application
Apr 30 17:58:08 Superdark bundle[11603]: [11603] * Listening on http://127.0.0.1:3000
Apr 30 17:58:08 Superdark bundle[11603]: [11603] ! WARNING: Detected 1 Thread(s) started in app boot:
Apr 30 17:58:08 Superdark bundle[11603]: [11603] ! #<Thread:0x000055bbbf3c8c98 /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:299 sleep> - /home/mastodon
/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:301:in `sleep'
Apr 30 17:58:08 Superdark bundle[11603]: [11603] Use Ctrl-C to stop
Apr 30 17:58:08 Superdark bundle[11603]: [11603] - Worker 0 (pid: 11669) booted, phase: 0
Apr 30 17:58:08 Superdark bundle[11603]: [11603] - Worker 1 (pid: 11673) booted, phase: 0
Apr 30 18:00:48 Superdark bundle[11603]: [8b271ded-51fd-4d47-9346-ddebe5b37a5d] Chewy request strategy is `custom_sidekiq`
Apr 30 18:00:48 Superdark bundle[11603]: [8b271ded-51fd-4d47-9346-ddebe5b37a5d]
Apr 30 17:58:08 Superdark bundle[11603]: [11603] * Listening on http://127.0.0.1:3000
Apr 30 17:58:08 Superdark bundle[11603]: [11603] ! WARNING: Detected 1 Thread(s) started in app boot:
Apr 30 17:58:08 Superdark bundle[11603]: [11603] ! #<Thread:0x000055bbbf3c8c98 /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:299 sleep> - /home/mastodon
/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:301:in `sleep'
Apr 30 17:58:08 Superdark bundle[11603]: [11603] Use Ctrl-C to stop
Apr 30 17:58:08 Superdark bundle[11603]: [11603] - Worker 0 (pid: 11669) booted, phase: 0
Apr 30 17:58:08 Superdark bundle[11603]: [11603] - Worker 1 (pid: 11673) booted, phase: 0
Apr 30 18:00:48 Superdark bundle[11603]: [8b271ded-51fd-4d47-9346-ddebe5b37a5d] Chewy request strategy is `custom_sidekiq`
Apr 30 18:00:48 Superdark bundle[11603]: [8b271ded-51fd-4d47-9346-ddebe5b37a5d]
Apr 30 18:00:48 Superdark bundle[11603]: [8b271ded-51fd-4d47-9346-ddebe5b37a5d] ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR:  relation "ip_blocks" does not exist
Apr 30 18:00:48 Superdark bundle[11603]: LINE 8:                WHERE a.attrelid = '"ip_blocks"'::regclass
Apr 30 18:00:48 Superdark bundle[11603]:                                           ^
Apr 30 18:00:48 Superdark bundle[11603]: :               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
Apr 30 18:00:48 Superdark bundle[11603]:                      pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod,
Apr 30 18:00:48 Superdark bundle[11603]:                      c.collname, col_description(a.attrelid, a.attnum) AS comment
Apr 30 18:00:48 Superdark bundle[11603]:                 FROM pg_attribute a
Apr 30 18:00:48 Superdark bundle[11603]:                 LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
Apr 30 18:00:48 Superdark bundle[11603]:                 LEFT JOIN pg_type t ON a.atttypid = t.oid
Apr 30 18:00:48 Superdark bundle[11603]:                 LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation
Apr 30 18:00:48 Superdark bundle[11603]:                WHERE a.attrelid = '"ip_blocks"'::regclass
Apr 30 18:00:48 Superdark bundle[11603]:                  AND a.attnum > 0 AND NOT a.attisdropped
Apr 30 18:00:48 Superdark bundle[11603]:                ORDER BY a.attnum
Apr 30 18:00:48 Superdark bundle[11603]: ):
Apr 30 18:00:48 Superdark bundle[11603]: [8b271ded-51fd-4d47-9346-ddebe5b37a5d]
Apr 30 18:00:48 Superdark bundle[11603]: [8b271ded-51fd-4d47-9346-ddebe5b37a5d] app/models/ip_block.rb:31:in `block in blocked?'
Apr 30 18:00:48 Superdark bundle[11603]: [8b271ded-51fd-4d47-9346-ddebe5b37a5d] app/models/ip_block.rb:31:in `blocked?'
Apr 30 18:00:48 Superdark bundle[11603]: [8b271ded-51fd-4d47-9346-ddebe5b37a5d] config/initializers/rack_attack.rb:46:in `block in <class:Attack>'
Apr 30 18:02:16 Superdark systemd[1]: Stopping mastodon-web...
Apr 30 18:02:16 Superdark bundle[11603]: [11603] - Gracefully shutting down workers...
Apr 30 18:02:17 Superdark systemd[1]: Stopped mastodon-web

Could someone help me get through this?

Tried running bundle exec rails db:migrate. no effect.

I wonder how would tables disappear? So you have re-installed mastodon from scratch at some point? What tables are still there (psql to Postgres and ask for \dn for your namespaces and then \dt namespace.* to look for your tables… You can dump your data and re-create the database from scratch and restore some data from the dump…

Hi,

Tried it, restored the table and now there seems to be a ip_blocks relation.
However now a new strange issue has taken shape. mastodon service does not seem to be starting up.
output below.

root@Superdark:~# journalctl -u mastodon-web | tail -n 5
Apr 30 18:00:48 Superdark bundle[11603]: [8b271ded-51fd-4d47-9346-ddebe5b37a5d] app/models/ip_block.rb:31:in `blocked?'
Apr 30 18:00:48 Superdark bundle[11603]: [8b271ded-51fd-4d47-9346-ddebe5b37a5d] config/initializers/rack_attack.rb:46:in `block in <class:Attack>'
Apr 30 18:02:16 Superdark systemd[1]: Stopping mastodon-web...
Apr 30 18:02:16 Superdark bundle[11603]: [11603] - Gracefully shutting down workers...
Apr 30 18:02:17 Superdark systemd[1]: Stopped mastodon-web.
root@Superdark:~# systemctl  restart mastodon-*
root@Superdark:~# journalctl -u mastodon-web | tail -n 5
Apr 30 18:00:48 Superdark bundle[11603]: [8b271ded-51fd-4d47-9346-ddebe5b37a5d] app/models/ip_block.rb:31:in `blocked?'
Apr 30 18:00:48 Superdark bundle[11603]: [8b271ded-51fd-4d47-9346-ddebe5b37a5d] config/initializers/rack_attack.rb:46:in `block in <class:Attack>'
Apr 30 18:02:16 Superdark systemd[1]: Stopping mastodon-web...
Apr 30 18:02:16 Superdark bundle[11603]: [11603] - Gracefully shutting down workers...
Apr 30 18:02:17 Superdark systemd[1]: Stopped mastodon-web.
root@Superdark:~# systemctl  start mastodon-*
root@Superdark:~# journalctl -u mastodon-web | tail -n 5
Apr 30 18:00:48 Superdark bundle[11603]: [8b271ded-51fd-4d47-9346-ddebe5b37a5d] app/models/ip_block.rb:31:in `blocked?'
Apr 30 18:00:48 Superdark bundle[11603]: [8b271ded-51fd-4d47-9346-ddebe5b37a5d] config/initializers/rack_attack.rb:46:in `block in <class:Attack>'
Apr 30 18:02:16 Superdark systemd[1]: Stopping mastodon-web...
Apr 30 18:02:16 Superdark bundle[11603]: [11603] - Gracefully shutting down workers...
Apr 30 18:02:17 Superdark systemd[1]: Stopped mastodon-web.

Restarting the system solved it.

Steps followed.

  1. Deleted DB
  2. Reloaded the dumb file
  3. Restarted service.

Did not work.

Restarted the server as such.

NOTE: I noticed the 500 page persisted after many many stop command to systemctl

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.