API errors after upgrade to v3.3.0: ERROR: materialized view "instances" has not been populated

Server administration Troubleshooting
#1

Hi,

I upgraded my instance (at scicomm.xyz) to 3.3.0 about a week ago, following the instructions on the release page. I didn’t encounter any problems or errors during the upgrade. However, today, a user on my instance reported that the API isn’t working correctly.

Indeed, visiting https://scicomm.xyz/api/v1/ results in the error-phant image.
However, checking https://scicomm.xyz/api/v1/accounts/1/statuses works.

The front-end is working fine.

Restarting nginx and/or the Mastodon processes hasn’t fixed the issue.

Running journalctl -u mastodon-web -f I can see:
path=/api/v1 format=html controller=ApplicationController action=raise_not_found status=404 duration=12.06 view=2.30 db=1.63

And also in the nginx access.log:
xxx.xxx.xxx.xxx - - [08/Jan/2021:11:06:42 +0000] "GET /api/v1 HTTP/2.0" 404 647 "-" "Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0" xxx.xxx.xxx.xxx - - [08/Jan/2021:11:06:42 +0000] "GET /oops.png HTTP/2.0" 304 0 "https://scicomm.xyz/api/v1" "Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0"

The last major change to my system was the upgrade to v3.3.0.

I am not too sure where to go next in troubleshooting this issue so any advice would be appreciated!

Cheers,
Dave

#2

I also seem to have lost some admin functions, e.g, from Settings, if I click Federation I also get the error image, so I cannot perform any domain-block management. It is starting to feel like something went awry during the upgrade process but any comments would be helpful.

#3

https://mastodon.social/api/v1 is not a good endpoint to try (there is no such endpoint).

Any other endpoints that do not work?

How did your database upgrade go?

#4

Hi,

Ah, sorry, I meant https://scicomm.xyz/api/v1/instance no longer works.
Also https://scicomm.xyz/admin/instances?limited=1 similarly returns an error page since the upgrade.

The database migration did not report any errors.

I have backups going back a month so I am prepared to roll the whole instance back to 3.2.2 if it might be too tricky to fix. This would lose a few posts from the past week before the problem was reported.

#5

Can you peek at the logs in response to those two API requests?

#6

Certainly …

The nginx access.log returns a 500 status in both cases above.

PostgresSQL reports, for /api/v1/instance:

2021-01-10 17:30:11 UTC [19920-1] mastodon@mastodon_production ERROR:  materialized view "instances" has not been populated
2021-01-10 17:30:11 UTC [19920-2] mastodon@mastodon_production HINT:  Use the REFRESH MATERIALIZED VIEW command.
2021-01-10 17:30:11 UTC [19920-3] mastodon@mastodon_production STATEMENT:  SELECT COUNT(*) FROM "instances"

and for admin/instances?limited=1:

2021-01-10 17:32:09 UTC [14534-91] mastodon@mastodon_production ERROR:  materialized view "instances" has not been populated
2021-01-10 17:32:09 UTC [14534-92] mastodon@mastodon_production HINT:  Use the REFRESH MATERIALIZED VIEW command.
2021-01-10 17:32:09 UTC [14534-93] mastodon@mastodon_production STATEMENT:  SELECT  1 AS one FROM "instances" INNER JOIN "domain_blocks" ON "domain_blocks"."domain" = "instances"."domain" LEFT OUTER JOIN "domain_allows" ON "domain_allows"."domain" = "instances"."domain" LIMIT $1 OFFSET $2
#7

I have posted this on Github Fix performance on instances list in admin UI by Gargron · Pull Request #15282 · tootsuite/mastodon · GitHub

Maybe you can try to connect to your PostgreSQL mastodon_production database with psql and try

 refresh materialized view instances
#8

Fantastic!

That has solved both problems :grinning:

Thanks for your help!!

1 Like
#9

Great it worked - however I am still curious why this happened in the first place.

#10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.