ERROR: column account_stats.lock_version does not exist

Couka · January 20, 2022, 5:36pm

I have this error in my logs and now I see that I have also an unexpected behavior in web interface : Since yesterday, toots are no longer displayed on Home but they are still visible in Global Public Timeline on my instance.

I have a personal instance with only me as a user. I thought that subscriptions were forbidden but apparently not (I change that since when) and 2 users created an account. They clearly are bots, so I tried to delete their accounts, but I was only able to suspend them. When I try to delete data, nothing is happening.

I recently upgraded from 3.4.0 to 3.4.4 and I am using docker to run my instance.

I have this logs :

# docker-compose logs -f | grep ERROR
db_1         | ERROR:  column account_stats.lock_version does not exist at character 233
sidekiq_1    | 2022-01-20T17:27:18.895Z pid=9 tid=micl WARN: {"context":"Job raised exception","job":{"retry":true,"queue":"pull","class":"Admin::AccountDeletionWorker","args":[107403850632228100],"jid":"f716808ac749904cf9fc5266","created_at":1642695217.1872647,"enqueued_at":1642699440.8429735,"error_message":"PG::UndefinedColumn: ERROR:  column account_stats.lock_version does not exist\nLINE 1: ...ccount_stats.following_count + -1, lock_version = account_st...\n                                                             ^\n","error_class":"ActiveRecord::StatementInvalid","failed_at":1642695400.9203568,"retry_count":6,"retried_at":1642697955.7566504},"jobstr":"{\"retry\":true,\"queue\":\"pull\",\"class\":\"Admin::AccountDeletionWorker\",\"args\":[107403850632228100],\"jid\":\"f716808ac749904cf9fc5266\",\"created_at\":1642695217.1872647,\"enqueued_at\":1642699440.8429735,\"error_message\":\"PG::UndefinedColumn: ERROR:  column account_stats.lock_version does not exist\\nLINE 1: ...ccount_stats.following_count + -1, lock_version = account_st...\\n
  ^\\n\",\"error_class\":\"ActiveRecord::StatementInvalid\",\"failed_at\":1642695400.9203568,\"retry_count\":6,\"retried_at\":1642697955.7566504}"}

If needed I also have other logs as WARN and INFO, might be irrelevant but I have a huge amount of them, still logging, I should not be normal I guess :

2022-01-20T17:32:51.034Z pid=9 tid=rm2x WARN: {"context":"Job raised exception","job":{"retry":8,"queue":"pull","dead":false,"class":"ActivityPub::LowPriorityDeliveryWorker","args":["{\"@context\":\"https://www.w3.org/ns/activitystreams\",\"id\":\"https://pouet.mickaelgillot.xyz/users/travert#delete\",\"type\":\"Delete\",\"actor\":\"https://pouet.mickaelgillot.xyz/users/travert\",\"to\":[\"https://www.w3.org/ns/activitystreams#Public\"],\"object\":\"https://pouet.mickaelgillot.xyz/users/travert\",\"signature\":{\"type\":\"RsaSignature2017\",\"creator\":\"https://pouet.mickaelgillot.xyz/users/travert#main-key\",\"created\":\"2022-01-20T17:27:18Z\",\"signatureValue\":\"mIA1U5Z1a2Axh3LiIfM2J8I4YN9I3paweQ1bBw5rYkMNUrmAhfC/jQcFV6ArPyNbKdP5KtRJ7qQQ4/hKZyGOyNG9NEuibl9UZlkLNNPTvgDmP38CtpGroOv7Ou58x5cUN2A0APDx0A4v4/v8j8NE8wqRt3ETBCsZ38EhF1H1uzwBk/N6JON7wsXNEPkThpdlKJgj6s00BDTT5834BJJ4A9+E1koX/D0Q7QdA+c0HFY7W0V7tsjjCXB1u+83TmMz4wYkXW2V/MZRKIlR7tEq4NZmwdG4SfnPhvpQ/HdksfhZ5tV6t2whrUgJs2ppgZTsO1KHCdsI6Mr+lx4kqmdPQ2g==\"}}",107403850632228100,"https://twitter.activitypub.actor/franceinter/inbox"],"jid":"2aa0e8db7c0a87bb7e9992f0","created_at":1642699638.548503,"enqueued_at":1642699638.579214},"jobstr":"{\"retry\":8,\"queue\":\"pull\",\"dead\":false,\"class\":\"ActivityPub::LowPriorityDeliveryWorker\",\"args\":[\"{\\\"@context\\\":\\\"https://www.w3.org/ns/activitystreams\\\",\\\"id\\\":\\\"https://pouet.mickaelgillot.xyz/users/travert#delete\\\",\\\"type\\\":\\\"Delete\\\",\\\"actor\\\":\\\"https://pouet.mickaelgillot.xyz/users/travert\\\",\\\"to\\\":[\\\"https://www.w3.org/ns/activitystreams#Public\\\"],\\\"object\\\":\\\"https://pouet.mickaelgillot.xyz/users/travert\\\",\\\"signature\\\":{\\\"type\\\":\\\"RsaSignature2017\\\",\\\"creator\\\":\\\"https://pouet.mickaelgillot.xyz/users/travert#main-key\\\",\\\"created\\\":\\\"2022-01-20T17:27:18Z\\\",\\\"signatureValue\\\":\\\"mIA1U5Z1a2Axh3LiIfM2J8I4YN9I3paweQ1bBw5rYkMNUrmAhfC/jQcFV6ArPyNbKdP5KtRJ7qQQ4/hKZyGOyNG9NEuibl9UZlkLNNPTvgDmP38CtpGroOv7Ou58x5cUN2A0APDx0A4v4/v8j8NE8wqRt3ETBCsZ38EhF1H1uzwBk/N6JON7wsXNEPkThpdlKJgj6s00BDTT5834BJJ4A9+E1koX/D0Q7QdA+c0HFY7W0V7tsjjCXB1u+83TmMz4wYkXW2V/MZRKIlR7tEq4NZmwdG4SfnPhvpQ/HdksfhZ5tV6t2whrUgJs2ppgZTsO1KHCdsI6Mr+lx4kqmdPQ2g==\\\"}}\",107403850632228100,\"https://twitter.activitypub.actor/franceinter/inbox\"],\"jid\":\"2aa0e8db7c0a87bb7e9992f0\",\"created_at\":1642699638.548503,\"enqueued_at\":1642699638.579214}"}
sidekiq_1    | 2022-01-20T17:32:51.034Z pid=9 tid=rm2x WARN: HTTP::ConnectionError: failed to connect: No address for twitter.activitypub.actor on https://twitter.activitypub.actor/franceinter/inbox
sidekiq_1    | 2022-01-20T17:32:51.034Z pid=9 tid=rm2x WARN: /opt/mastodon/app/lib/request.rb:249:in `open'
sidekiq_1    | 2022-01-20T17:32:51.044Z pid=9 tid=rn0t class=ActivityPub::LowPriorityDeliveryWorker jid=0665bc7421aa0fd5d1d99dc2 INFO: start
sidekiq_1    | 2022-01-20T17:32:51.053Z pid=9 tid=rmkp class=ActivityPub::LowPriorityDeliveryWorker jid=3835aa2f0390898bd49548bb elapsed=0.421 INFO: done
sidekiq_1    | 2022-01-20T17:32:51.064Z pid=9 tid=rmkp class=ActivityPub::LowPriorityDeliveryWorker jid=a07cd8bc2473ab09ce7c9483 INFO: start
sidekiq_1    | 2022-01-20T17:32:51.084Z pid=9 tid=rlf5 class=ActivityPub::LowPriorityDeliveryWorker jid=e16c2e54b5e5ad93e9e211d5 elapsed=0.067 INFO: fail
sidekiq_1    | 2022-01-20T17:32:51.084Z pid=9 tid=rlf5 WARN: {"context":"Job raised exception","job":{"retry":8,"queue":"pull","dead":false,"class":"ActivityPub::LowPriorityDeliveryWorker","args":["{\"@context\":\"https://www.w3.org/ns/activitystreams\",\"id\":\"https://pouet.mickaelgillot.xyz/users/travert#delete\",\"type\":\"Delete\",\"actor\":\"https://pouet.mickaelgillot.xyz/users/travert\",\"to\":[\"https://www.w3.org/ns/activitystreams#Public\"],\"object\":\"https://pouet.mickaelgillot.xyz/users/travert\",\"signature\":{\"type\":\"RsaSignature2017\",\"creator\":\"https://pouet.mickaelgillot.xyz/users/travert#main-key\",\"created\":\"2022-01-20T17:27:18Z\",\"signatureValue\":\"mIA1U5Z1a2Axh3LiIfM2J8I4YN9I3paweQ1bBw5rYkMNUrmAhfC/jQcFV6ArPyNbKdP5KtRJ7qQQ4/hKZyGOyNG9NEuibl9UZlkLNNPTvgDmP38CtpGroOv7Ou58x5cUN2A0APDx0A4v4/v8j8NE8wqRt3ETBCsZ38EhF1H1uzwBk/N6JON7wsXNEPkThpdlKJgj6s00BDTT5834BJJ4A9+E1koX/D0Q7QdA+c0HFY7W0V7tsjjCXB1u+83TmMz4wYkXW2V/MZRKIlR7tEq4NZmwdG4SfnPhvpQ/HdksfhZ5tV6t2whrUgJs2ppgZTsO1KHCdsI6Mr+lx4kqmdPQ2g==\"}}",107403850632228100,"https://twitter.activitypub.actor/streetpress/inbox"],"jid":"e16c2e54b5e5ad93e9e211d5","created_at":1642699638.548503,"enqueued_at":1642699638.579214},"jobstr":"{\"retry\":8,\"queue\":\"pull\",\"dead\":false,\"class\":\"ActivityPub::LowPriorityDeliveryWorker\",\"args\":[\"{\\\"@context\\\":\\\"https://www.w3.org/ns/activitystreams\\\",\\\"id\\\":\\\"https://pouet.mickaelgillot.xyz/users/travert#delete\\\",\\\"type\\\":\\\"Delete\\\",\\\"actor\\\":\\\"https://pouet.mickaelgillot.xyz/users/travert\\\",\\\"to\\\":[\\\"https://www.w3.org/ns/activitystreams#Public\\\"],\\\"object\\\":\\\"https://pouet.mickaelgillot.xyz/users/travert\\\",\\\"signature\\\":{\\\"type\\\":\\\"RsaSignature2017\\\",\\\"creator\\\":\\\"https://pouet.mickaelgillot.xyz/users/travert#main-key\\\",\\\"created\\\":\\\"2022-01-20T17:27:18Z\\\",\\\"signatureValue\\\":\\\"mIA1U5Z1a2Axh3LiIfM2J8I4YN9I3paweQ1bBw5rYkMNUrmAhfC/jQcFV6ArPyNbKdP5KtRJ7qQQ4/hKZyGOyNG9NEuibl9UZlkLNNPTvgDmP38CtpGroOv7Ou58x5cUN2A0APDx0A4v4/v8j8NE8wqRt3ETBCsZ38EhF1H1uzwBk/N6JON7wsXNEPkThpdlKJgj6s00BDTT5834BJJ4A9+E1koX/D0Q7QdA+c0HFY7W0V7tsjjCXB1u+83TmMz4wYkXW2V/MZRKIlR7tEq4NZmwdG4SfnPhvpQ/HdksfhZ5tV6t2whrUgJs2ppgZTsO1KHCdsI6Mr+lx4kqmdPQ2g==\\\"}}\",107403850632228100,\"https://twitter.activitypub.actor/streetpress/inbox\"],\"jid\":\"e16c2e54b5e5ad93e9e211d5\",\"created_at\":1642699638.548503,\"enqueued_at\":1642699638.579214}"}
sidekiq_1    | 2022-01-20T17:32:51.086Z pid=9 tid=rlf5 WARN: HTTP::ConnectionError: failed to connect: No address for twitter.activitypub.actor on https://twitter.activitypub.actor/streetpress/inbox
sidekiq_1    | 2022-01-20T17:32:51.086Z pid=9 tid=rlf5 WARN: /opt/mastodon/app/lib/request.rb:249:in `open'
sidekiq_1    | 2022-01-20T17:32:51.087Z pid=9 tid=rn5d class=ActivityPub::LowPriorityDeliveryWorker jid=0f8e710c383674645dd962ac INFO: start
sidekiq_1    | 2022-01-20T17:32:51.101Z pid=9 tid=rn5d class=ActivityPub::LowPriorityDeliveryWorker jid=0f8e710c383674645dd962ac elapsed=0.014 INFO: fail
sidekiq_1    | 2022-01-20T17:32:51.101Z pid=9 tid=rn5d WARN: {"context":"Job raised exception","job":{"retry":8,"queue":"pull","dead":false,"class":"ActivityPub::LowPriorityDeliveryWorker","args":["{\"@context\":\"https://www.w3.org/ns/activitystreams\",\"id\":\"https://pouet.mickaelgillot.xyz/users/travert#delete\",\"type\":\"Delete\",\"actor\":\"https://pouet.mickaelgillot.xyz/users/travert\",\"to\":[\"https://www.w3.org/ns/activitystreams#Public\"],\"object\":\"https://pouet.mickaelgillot.xyz/users/travert\",\"signature\":{\"type\":\"RsaSignature2017\",\"creator\":\"https://pouet.mickaelgillot.xyz/users/travert#main-key\",\"created\":\"2022-01-20T17:27:18Z\",\"signatureValue\":\"mIA1U5Z1a2Axh3LiIfM2J8I4YN9I3paweQ1bBw5rYkMNUrmAhfC/jQcFV6ArPyNbKdP5KtRJ7qQQ4/hKZyGOyNG9NEuibl9UZlkLNNPTvgDmP38CtpGroOv7Ou58x5cUN2A0APDx0A4v4/v8j8NE8wqRt3ETBCsZ38EhF1H1uzwBk/N6JON7wsXNEPkThpdlKJgj6s00BDTT5834BJJ4A9+E1koX/D0Q7QdA+c0HFY7W0V7tsjjCXB1u+83TmMz4wYkXW2V/MZRKIlR7tEq4NZmwdG4SfnPhvpQ/HdksfhZ5tV6t2whrUgJs2ppgZTsO1KHCdsI6Mr+lx4kqmdPQ2g==\"}}",107403850632228100,"https://emodykes.club/inbox"],"jid":"0f8e710c383674645dd962ac","created_at":1642699638.548503,"enqueued_at":1642699638.579214},"jobstr":"{\"retry\":8,\"queue\":\"pull\",\"dead\":false,\"class\":\"ActivityPub::LowPriorityDeliveryWorker\",\"args\":[\"{\\\"@context\\\":\\\"https://www.w3.org/ns/activitystreams\\\",\\\"id\\\":\\\"https://pouet.mickaelgillot.xyz/users/travert#delete\\\",\\\"type\\\":\\\"Delete\\\",\\\"actor\\\":\\\"https://pouet.mickaelgillot.xyz/users/travert\\\",\\\"to\\\":[\\\"https://www.w3.org/ns/activitystreams#Public\\\"],\\\"object\\\":\\\"https://pouet.mickaelgillot.xyz/users/travert\\\",\\\"signature\\\":{\\\"type\\\":\\\"RsaSignature2017\\\",\\\"creator\\\":\\\"https://pouet.mickaelgillot.xyz/users/travert#main-key\\\",\\\"created\\\":\\\"2022-01-20T17:27:18Z\\\",\\\"signatureValue\\\":\\\"mIA1U5Z1a2Axh3LiIfM2J8I4YN9I3paweQ1bBw5rYkMNUrmAhfC/jQcFV6ArPyNbKdP5KtRJ7qQQ4/hKZyGOyNG9NEuibl9UZlkLNNPTvgDmP38CtpGroOv7Ou58x5cUN2A0APDx0A4v4/v8j8NE8wqRt3ETBCsZ38EhF1H1uzwBk/N6JON7wsXNEPkThpdlKJgj6s00BDTT5834BJJ4A9+E1koX/D0Q7QdA+c0HFY7W0V7tsjjCXB1u+83TmMz4wYkXW2V/MZRKIlR7tEq4NZmwdG4SfnPhvpQ/HdksfhZ5tV6t2whrUgJs2ppgZTsO1KHCdsI6Mr+lx4kqmdPQ2g==\\\"}}\",107403850632228100,\"https://emodykes.club/inbox\"],\"jid\":\"0f8e710c383674645dd962ac\",\"created_at\":1642699638.548503,\"enqueued_at\":1642699638.579214}"}
sidekiq_1    | 2022-01-20T17:32:51.101Z pid=9 tid=rn5d WARN: HTTP::ConnectionError: failed to connect: Connection refused - connect(2) for 217.70.184.38:443 on https://emodykes.club/inbox
sidekiq_1    | 2022-01-20T17:32:51.101Z pid=9 tid=rn5d WARN: /opt/ruby/lib/ruby/2.7.0/socket.rb:1214:in `__connect_nonblock'
sidekiq_1    | 2022-01-20T17:32:51.102Z pid=9 tid=rn9p class=ActivityPub::LowPriorityDeliveryWorker jid=f6ec8c3020db0fdcff80e15e INFO: start
sidekiq_1    | 2022-01-20T17:32:51.126Z pid=9 tid=rmkp class=ActivityPub::LowPriorityDeliveryWorker jid=a07cd8bc2473ab09ce7c9483 elapsed=0.062 INFO: done
sidekiq_1    | 2022-01-20T17:32:51.128Z pid=9 tid=rmkp class=ActivityPub::LowPriorityDeliveryWorker jid=a3aab27be83869a25c056c80 INFO: start
sidekiq_1    | 2022-01-20T17:32:51.151Z pid=9 tid=rn0t class=ActivityPub::LowPriorityDeliveryWorker jid=0665bc7421aa0fd5d1d99dc2 elapsed=0.107 INFO: done
sidekiq_1    | 2022-01-20T17:32:51.153Z pid=9 tid=rn0t class=ActivityPub::LowPriorityDeliveryWorker jid=03f03d3d0828ef32c35d6818 INFO: start
sidekiq_1    | 2022-01-20T17:32:51.191Z pid=9 tid=rn0t class=ActivityPub::LowPriorityDeliveryWorker jid=03f03d3d0828ef32c35d6818 elapsed=0.038 INFO: fail

Claire · January 23, 2022, 10:07pm

Make sure you ran the database migrations, and restart the containers/services, this should fix all your issues.

Couka · January 24, 2022, 8:42am

I ran the data migration, here is the output:

$ sudo docker-compose run --rm web bundle exec rake db:migrate
Starting mastodon_db_1 ... done
Starting mastodon_redis_1 ... done
/opt/ruby/lib/ruby/2.7.0/net/protocol.rb:66: warning: already initialized constant Net::ProtocRetryError
/opt/mastodon/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.0/lib/net/protocol.rb:66: warning: previous definition of ProtocRetryError was here
/opt/ruby/lib/ruby/2.7.0/net/protocol.rb:206: warning: already initialized constant Net::BufferedIO::BUFSIZE
/opt/mastodon/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.0/lib/net/protocol.rb:206: warning: previous definition of BUFSIZE was here
/opt/ruby/lib/ruby/2.7.0/net/protocol.rb:503: warning: already initialized constant Net::NetPrivate::Socket
/opt/mastodon/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.0/lib/net/protocol.rb:503: warning: previous definition of Socket was here
Your database collation is susceptible to index corruption.
  (This warning does not indicate that index corruption has occurred and can be ignored)
  (To learn more, visit: https://docs.joinmastodon.org/admin/troubleshooting/index-corruption/)

Then I restarted the containers with docker-compose (down, then up). I still have the errors in logs, and UI is still broken.

Claire · January 24, 2022, 9:36am

Couka:

I have this logs :

# docker-compose logs -f | grep ERROR
db_1         | ERROR:  column account_stats.lock_version does not exist at character 233
sidekiq_1    | 2022-01-20T17:27:18.895Z pid=9 tid=micl WARN: {"context":"Job raised exception","job":{"retry":true,"queue":"pull","class":"Admin::AccountDeletionWorker","args":[107403850632228100],"jid":"f716808ac749904cf9fc5266","created_at":1642695217.1872647,"enqueued_at":1642699440.8429735,"error_message":"PG::UndefinedColumn: ERROR:  column account_stats.lock_version does not exist\nLINE 1: ...ccount_stats.following_count + -1, lock_version = account_st...\n                                                             ^\n","error_class":"ActiveRecord::StatementInvalid","failed_at":1642695400.9203568,"retry_count":6,"retried_at":1642697955.7566504},"jobstr":"{\"retry\":true,\"queue\":\"pull\",\"class\":\"Admin::AccountDeletionWorker\",\"args\":[107403850632228100],\"jid\":\"f716808ac749904cf9fc5266\",\"created_at\":1642695217.1872647,\"enqueued_at\":1642699440.8429735,\"error_message\":\"PG::UndefinedColumn: ERROR:  column account_stats.lock_version does not exist\\nLINE 1: ...ccount_stats.following_count + -1, lock_version = account_st...\\n
  ^\\n\",\"error_class\":\"ActiveRecord::StatementInvalid\",\"failed_at\":1642695400.9203568,\"retry_count\":6,\"retried_at\":1642697955.7566504}"}

These should be addressed now.

The errors reported in the second part of your original post are delivery failures to dead or temporarily unavailable instances, they are not indicative of an issue with your instance.

The warnings in your last post are known, they’re just warnings and not indicative of an issue.

In which way is the UI broken?

Couka · January 24, 2022, 10:21am

My personal timeline is frozen. I cannot see any new toot since this day. But if I go to Global Public Timeline, I can see toots of people I follow (and should be displayed in my own timeline)
Besides this, I have not any new notification, even though some people have boosted / fav some of my toots (I can see the boost of my toots from their profile)

Claire · January 28, 2022, 7:31pm

Does reloading the page show you new toots in the personal timeline / notifications etc.?
Do you have any different error in the sidekiq logs?
Do you have errors in the browser’s web console?

Couka · January 31, 2022, 1:34pm

Does reloading the page show you new toots in the personal timeline / notifications etc.?

No.

Do you have any different error in the sidekiq logs?

The only logs from sidekiq are those:

sidekiq_1    | 2022-01-31T13:25:18.715Z pid=7 tid=2hbjf WARN: {"context":"Job raised exception","job":{"retry":true,"queue":"push","lock":"until_executed","class":"ActivityPub::FollowersSynchronizationWorker","args":[1195,"https://mastd.racing/users/Torrone/followers_synchronization"],"jid":"f33586008106afe5c6042c82","created_at":1642200226.0419214,"lock_timeout":0,"lock_ttl":null,"lock_prefix":"uniquejobs","lock_args":[1195,"https://mastd.racing/users/Torrone/followers_synchronization"],"lock_digest":"uniquejobs:79cb902b31a527b9227a92ebcdc1ca28","enqueued_at":1643635518.367449,"error_message":"https://mastd.racing/users/Torrone/followers_synchronization returned code 500","error_class":"Mastodon::UnexpectedResponseError","failed_at":1642200226.3901258,"retry_count":23,"retried_at":1643355565.8346672},"jobstr":"{\"retry\":true,\"queue\":\"push\",\"lock\":\"until_executed\",\"class\":\"ActivityPub::FollowersSynchronizationWorker\",\"args\":[1195,\"https://mastd.racing/users/Torrone/followers_synchronization\"],\"jid\":\"f33586008106afe5c6042c82\",\"created_at\":1642200226.0419214,\"lock_timeout\":0,\"lock_ttl\":null,\"lock_prefix\":\"uniquejobs\",\"lock_args\":[1195,\"https://mastd.racing/users/Torrone/followers_synchronization\"],\"lock_digest\":\"uniquejobs:79cb902b31a527b9227a92ebcdc1ca28\",\"enqueued_at\":1643635518.367449,\"error_message\":\"https://mastd.racing/users/Torrone/followers_synchronization returned code 500\",\"error_class\":\"Mastodon::UnexpectedResponseError\",\"failed_at\":1642200226.3901258,\"retry_count\":23,\"retried_at\":1643355565.8346672}"}
sidekiq_1    | 2022-01-31T13:25:18.715Z pid=7 tid=2hbjf WARN: Mastodon::UnexpectedResponseError: https://mastd.racing/users/Torrone/followers_synchronization returned code 500
sidekiq_1    | 2022-01-31T13:25:18.715Z pid=7 tid=2hbjf WARN: /opt/mastodon/app/helpers/jsonld_helper.rb:83:in `block in fetch_resource_without_id_validation'
sidekiq_1    | /opt/mastodon/app/lib/request.rb:75:in `perform'
sidekiq_1    | /opt/mastodon/app/helpers/jsonld_helper.rb:82:in `fetch_resource_without_id_validation'
sidekiq_1    | 2022-01-31T13:27:29.062Z pid=7 tid=2hbln class=ActivityPub::ProcessingWorker jid=ef909639bc45c906e4c30649 INFO: start
sidekiq_1    | [paperclip] Trying to link /tmp/adb5a27c57203a496238e46fd5abe12720220131-7-spouly.jpg to /tmp/4b72ee474124418f2362c2d1e548e23b20220131-7-1u6ar3f.jpg
sidekiq_1    | Command :: file -b --mime '/tmp/4b72ee474124418f2362c2d1e548e23b20220131-7-1u6ar3f.jpg'
sidekiq_1    | Command :: identify -format %m '/tmp/adb5a27c57203a496238e46fd5abe12720220131-7-spouly.jpg[0]'
sidekiq_1    | Command :: convert '/tmp/adb5a27c57203a496238e46fd5abe12720220131-7-spouly.jpg[0]' -auto-orient -resize "240x" -crop "240x240+0+0" +repage -strip '/tmp/96fd542edce13a36379c582646c3812020220131-7-qmzehz'
sidekiq_1    | [paperclip] Trying to link /tmp/96fd542edce13a36379c582646c3812020220131-7-qmzehz to /tmp/a96d114af9c0b0a747517827ad98966f20220131-7-lsvgeb
sidekiq_1    | [paperclip] Trying to link /tmp/df091b5799fa2ffdefa811e8f3894e4920220131-7-1pm24ag.jpg to /tmp/fe15d6a44c33f235f1b1d98395bfe59720220131-7-1cnvm2y.jpg
sidekiq_1    | Command :: file -b --mime '/tmp/fe15d6a44c33f235f1b1d98395bfe59720220131-7-1cnvm2y.jpg'
sidekiq_1    | [paperclip] Trying to link /tmp/df091b5799fa2ffdefa811e8f3894e4920220131-7-1pm24ag.jpg to /tmp/cbcdd688a01fcd2bc21b67972fc6af2520220131-7-19icfzw.jpg
sidekiq_1    | [paperclip] Trying to link /tmp/a96d114af9c0b0a747517827ad98966f20220131-7-lsvgeb to /tmp/2d18959a33648ed32aa4856118a9dd2a20220131-7-s2h9sq.jpg
sidekiq_1    | Command :: file -b --mime '/tmp/2d18959a33648ed32aa4856118a9dd2a20220131-7-s2h9sq.jpg'
sidekiq_1    | [paperclip] Trying to link /tmp/cbcdd688a01fcd2bc21b67972fc6af2520220131-7-19icfzw.jpg to /tmp/fe15d6a44c33f235f1b1d98395bfe59720220131-7-lzatm1.jpg
sidekiq_1    | Command :: file -b --mime '/tmp/fe15d6a44c33f235f1b1d98395bfe59720220131-7-lzatm1.jpg'
sidekiq_1    | 2022-01-31T13:27:29.742Z pid=7 tid=2haeb class=ActivityPub::SynchronizeFeaturedCollectionWorker jid=17965e8594ef4ac0b2f18e49 INFO: start
sidekiq_1    | 2022-01-31T13:27:29.805Z pid=7 tid=2haeb class=ActivityPub::SynchronizeFeaturedCollectionWorker jid=17965e8594ef4ac0b2f18e49 elapsed=0.063 INFO: done
sidekiq_1    | 2022-01-31T13:27:29.827Z pid=7 tid=2hbln class=ActivityPub::ProcessingWorker jid=ef909639bc45c906e4c30649 elapsed=0.765 INFO: fail

Do you have errors in the browser’s web console?

No.

Claire · January 31, 2022, 1:49pm

Hm, I can’t see anything wrong in those logs, try doing journalctl -u mastodon-sidekiq --follow and watch for new errors as you post

Couka · January 31, 2022, 2:03pm

I might not be clear enough about this point but I am running Mastodon with docker containers, so I don’t have any service mastodon-sidekiq on the host machine.

The logs I am showing you are from the following command:

sudo docker-compose logs -f sidekiq

So basically, it’s all the logs output by the service running in docker container, which should be similar to running the service on the host machine and get logs with journalctl command.

I might need to reinstall mastodon from scratch, so I am wondering if there is an option to export/import db (with all data) from an instance to another.

Claire · January 31, 2022, 2:18pm

Oh, sorry, I forgot you were using docker. I can’t see anything obviously wrong in these logs excerpts anyway…

Can you check /admin/dashboard to see if you have any warning? What about queued and running jobs in /sidekiq?

The things you’d have to export are:

the PostgreSQL database
the media files (stored in public/system)

Couka · January 31, 2022, 5:32pm

No warnings in /admin/dashboard and nothing unusual from /sidekiq (see image below).

However I still have the db ERROR:

db_1         | ERROR:  column account_stats.lock_version does not exist at character 233
db_1         | STATEMENT:  INSERT INTO account_stats(account_id, following_count, created_at, updated_at) VALUES (107403850632228100, 0, now(), now()) ON CONFLICT (account_id) DO UPDATE SET following_count = account_stats.following_count + -1, lock_version = account_stats.lock_version + 1, updated_at = now() RETURNING id;

And I still have 2 unwanted users I cannot delete from my instance. Wouldn’t be related to my main issue as it tries to delete them, but fails and so it won’t show any new toots on those views ?

And If I try to export/import my database, I will keep those users I don’t want. Is there another way to remove these users directly from the database ? With PostgreSQL queries ?

Claire · January 31, 2022, 5:36pm

ok, this is weird. can you check that all your containers are updated to the same Mastodon version? and make sure you stop and restart all of them? that error should not occur in any release, and it indeed seems to be the cause of all your issues

Couka · January 31, 2022, 6:36pm

I think it is working now.

I was failing to build mastodon image from sources, so I was using the remote built image from registry. My docker-compose.yml looked like this:

web:
 #     build: .
      image: tootsuite/mastodon

But as I saw that there was a newer version released (v3.4.5 at this time), I pulled the repository and retry to build the image. For the record, I followed this proc: Updating Mastodon Instance (Docker Version)

So to build the image I changed my docker-compose.yml to this:

web:
      build: .
      image: tootsuite/mastodon:v3.4.5

The build was successful this time (for an unknown reason). I completed the rest of the proc and now I see new toots. I cannot see toots and notifications from the period of time between the beginning of my issue to the resolution, but now personal timeline (and I assume notifications, which I cannot check until I get new ones, but I assume it works as well) is updated successfully and I can see new toots since I restarted my services.

Thanks a lot for your help, your support and your advices which lead me to resolve all of my issues. I mark your last message as a solution, as it was the best thing to do.

PS : I was using v3.4.4 of mastodon image, as it was displayed in the footer my homepage.

system · February 14, 2022, 6:37pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.