Most sidekiq jobs are suddenly failing: function now() does not exist

I run an instance, and since a few days ago, sidekiq jobs have started failing. Here is the sidekiq overview:

As you can see, my server was offline for a few days (harddisk ran full), and it first recovered nicely. But a few days later, sidekiq jobs have started failing. I looked at the logs, and saw many errors like this:

2021-08-10T06:58:06.439Z pid=7 tid=3ck7 WARN: {"context":"Job raised exception","job":{"retry":8,"queue":"default","backtrace":true,"class":"ActivityPub::ProcessingWorker","args":[106690753903408938,"{\"@context\":\"https://www.w3.org/ns/activitystreams\",\"id\":\"https://mastodon.social/users/Mr91178#accepts/follows/5707882\",\"type\":\"Accept\",\"actor\":\"https://mastodon.social/users/Mr91178\",\"object\":{\"id\":\"https://social.heldom.de/de471bda-345b-40d4-8007-e89dde830bc2\",\"type\":\"Follow\",\"actor\":\"https://social.heldom.de/users/heldom\",\"object\":\"https://mastodon.social/users/Mr91178\"}}",1],"jid":"83435bd2fdd996cc45c14de5","created_at":1628577217.5980966,"enqueued_at":1628578686.2117789,"error_message":"PG::UndefinedFunction: ERROR:  function now() does not exist\nLINE 1: ...wing_count, created_at, updated_at) VALUES (1, 1, now(), now...\n                                                             ^\nHINT:  No function matches the given name and argument types. You might need to add explicit type casts.\n","error_class":"ActiveRecord::StatementInvalid","failed_at":1628577217.6142738,"retry_count":5,"error_backtrace":"eJzN0EFOAzEMBdCrjLqhSBDDgFrUq1QoTRyrRGTsYGdGcPumUCQWrFixs/6X\nrGfvVyC1wRSsSRKGhTiJQpw5FQKd4weMbuvu4EiTQcCWF1JC0XS7cffuwY1Q\ncrwU/qsBFGbqibAPKdRGalDF2lHJ3sp35jTuNttxl3k40Duhr0HDZFerm/9h\nikXwdViPQ6GFil0PPfx0sngM+EJ/k9pcq2j7jXqpIFHti4gxk0Hm7jpTzrbH\npx+0PlXSKTffj8NZlbj5IiH1Hz6fAFEUqmQ=\n","retried_at":1628577934.5520344},"jobstr":"{\"retry\":8,\"queue\":\"default\",\"backtrace\":true,\"class\":\"ActivityPub::ProcessingWorker\",\"args\":[106690753903408938,\"{\\\"@context\\\":\\\"https://www.w3.org/ns/activitystreams\\\",\\\"id\\\":\\\"https://mastodon.social/users/Mr91178#accepts/follows/5707882\\\",\\\"type\\\":\\\"Accept\\\",\\\"actor\\\":\\\"https://mastodon.social/users/Mr91178\\\",\\\"object\\\":{\\\"id\\\":\\\"https://social.heldom.de/de471bda-345b-40d4-8007-e89dde830bc2\\\",\\\"type\\\":\\\"Follow\\\",\\\"actor\\\":\\\"https://social.heldom.de/users/heldom\\\",\\\"object\\\":\\\"https://mastodon.social/users/Mr91178\\\"}}\",1],\"jid\":\"83435bd2fdd996cc45c14de5\",\"created_at\":1628577217.5980966,\"enqueued_at\":1628578686.2117789,\"error_message\":\"PG::UndefinedFunction: ERROR:  function now() does not exist\\nLINE 1: ...wing_count, created_at, updated_at) VALUES (1, 1, now(), now...\\n                                                             ^\\nHINT:  No function matches the given name and argument types. You might need to add explicit type casts.\\n\",\"error_class\":\"ActiveRecord::StatementInvalid\",\"failed_at\":1628577217.6142738,\"retry_count\":5,\"error_backtrace\":\"eJzN0EFOAzEMBdCrjLqhSBDDgFrUq1QoTRyrRGTsYGdGcPumUCQWrFixs/6X\\nrGfvVyC1wRSsSRKGhTiJQpw5FQKd4weMbuvu4EiTQcCWF1JC0XS7cffuwY1Q\\ncrwU/qsBFGbqibAPKdRGalDF2lHJ3sp35jTuNttxl3k40Duhr0HDZFerm/9h\\nikXwdViPQ6GFil0PPfx0sngM+EJ/k9pcq2j7jXqpIFHti4gxk0Hm7jpTzrbH\\npx+0PlXSKTffj8NZlbj5IiH1Hz6fAFEUqmQ=\\n\",\"retried_at\":1628577934.5520344}"}
2021-08-10T06:58:06.439Z pid=7 tid=3ck7 WARN: ActiveRecord::StatementInvalid: PG::UndefinedFunction: ERROR:  function now() does not exist
LINE 1: ...wing_count, created_at, updated_at) VALUES (1, 1, now(), now...
                                                             ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

Does anyone know where that could be coming from?

I am running mastodon 3.4.1 with only the modifications indicated by this diff:

diff --git a/app/javascript/styles/mastodon/components.scss b/app/javascript/styles/mastodon/components.scss
index 7ca027e81..229676977 100644
--- a/app/javascript/styles/mastodon/components.scss
+++ b/app/javascript/styles/mastodon/components.scss
@@ -2105,7 +2105,7 @@ a.account__display-name {
 }
 
 .column {
-  width: 350px;
+  width: 650px;
   position: relative;
   box-sizing: border-box;
   display: flex;
diff --git a/app/models/media_attachment.rb b/app/models/media_attachment.rb
index 3515f6895..fad97f221 100644
--- a/app/models/media_attachment.rb
+++ b/app/models/media_attachment.rb
@@ -151,7 +151,7 @@ class MediaAttachment < ApplicationRecord
   }.freeze
 
   IMAGE_LIMIT = 10.megabytes
-  VIDEO_LIMIT = 40.megabytes
+  VIDEO_LIMIT = 1000.megabytes
 
   MAX_VIDEO_MATRIX_LIMIT = 2_304_000 # 1920x1200px
   MAX_VIDEO_FRAME_RATE   = 60

Edit 1: The same thing happens when I try to upload an image to post:

[fa8d09cd-3755-4791-84ee-f6fa044c6a42] method=POST path=/api/v2/media format=html controller=Api::V2::MediaController action=create status=500 error='ActiveRecord::StatementInvalid: PG::UndefinedFunction: ERROR:  function now() does not exist
LINE 3:       ((date_part('epoch', now()) * 1000))::bigint
                                   ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.
QUERY:  SELECT (
      -- Get the time in milliseconds
      ((date_part('epoch', now()) * 1000))::bigint
      -- And shift it over two bytes
      << 16)
CONTEXT:  PL/pgSQL function timestamp_id(text) line 7 at assignment
' duration=1065.99 view=0.00 db=338.67

Edit 2: I have done a full rebuild/redownload of all docker containers, the issue persists. Does that mean that the database did get corrupted?

I reported this as a bug now, so I guess this can be closed

If anyone stumbles upon this in the future from some search engine:

The solution was to backup the database and recreate it, as this is apparently caused by postgres getting screwed up due to my server running out of memory. See the bug report I made for more details.

2 Likes

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.