Problems after db corruption

Hi folks,

I have some problems with my instance, after I mistakenly deleted some rows in the users table. The instance is broken in a way that some remote user can’t be followed and mentions of those users cause errors sending (404 - {“error”:“Record not found”}). Furthermore, I get a lot of errors in the sidekiq log:

https://paste.bka.li/view/c68f6cfd

A fellow admin and I were investigating this problem just now but we didn’t find a solution to repair the instance.

Unfortunally, I don’t have a backup from before the deletion (only after - I noticed the errors too late a: by then, the backups have been overwritten).

when deleting, I assumed that the data records are not used directly and can be loaded from the federation if necessary. In practice, it obviously turns out differently, but of course that depends on the specific implementation. I only deleted users who had nothing else in the system (no pursuits, toots, blocks, nothing). My idea was that these records would also be missing in a fresh instance and could then be fetched from the Federation if necessary.

I honestly didn’t expect these strange side effects. To understand exactly what is going wrong, you first need to know what is going on and how to best debug it. In the best case scenario, the result is somewhat more robust code that could restore a partially corrupt database from the Federation. In the end, that would help everyone, because database errors can occur anytime to anyone.

I have one hint. In the log, the exception is thrown somewher with account_finder_concern::representative in the trace. Here it tries to find the Account with id -99, which never exists and thus the error.

Does that ring a bell to anyone? Or maybe someone has another idea…

I’d be grateful for any help!

My guess is that you deleted the system account that Mastodon uses to sign server-level messages to other users. (Or, you never ran the migration to create that account in the first place).

This isn’t a database corruption issue, you just deleted records that mastodon needs to function. There’s no way this record can be deleted selectively in normal operation, or even in an exceptional error condition, so this isn’t really “something that could happen to anybody”.

1 Like

as @nightpool says, a crucial system-wide user has been deleted.

db/migrate/20190715164535_add_instance_actor.rb is a migration responsible to create it.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.