After 2.7 upgrade, all pages blank

#1

Hey there,

I am completely lost. I was upgrading my instance to 2.7.3 and, since this was an upgrade from < 2.7.0, I followed the upgrade instructions for 2.7.0 found here. Everything appeared to go smoothly, but when I brought it back up…it’s rendering nothing. I mean that very literally. I’m following the mastodon-web logging and there are no errors being printed and it’s pretty faithfully responding 200 to virtually all requests, but my developer pane in Chrome shows an empty HTML head and and empty body and nothing more, not even an error.

With no errors to go on, I’m completely lost and unsure what to do next. If you were in my shoes, what would you be trying next?

#2

can you link the instance in question? are you sure there are no 404 errors or 500 errors in Chrome’s network tab?

#3

My instance is at https://signs.codes and I double-checked the Network tab. No errors at all…just an empty index page, the favicon, and something called “sw.js”.

Logging from around that time shows very pedestrian stuff going on server-side. Certainly no errors:

Feb 26 14:44:46 ip-172-31-40-19 bundle[7579]: [89833757-206f-4015-9796-34ff1816f7c1] method=GET path=/@Vesper/media format=*/* controller=AccountsController action=show status=200 duration=56.55 view=28.95 db=8.71
Feb 26 14:44:48 ip-172-31-40-19 bundle[7579]: [fa0572cb-ca92-40ab-a875-99c80b8182fe] method=GET path=/api/v1/timelines/home format=html controller=Api::V1::Timelines::HomeController action=show status=200 duration=8.16 view=0.40 db=2.34
Feb 26 14:44:48 ip-172-31-40-19 bundle[7579]: [b2ad4fd5-230b-467c-8258-79d80e27b70c] method=GET path=/api/v1/notifications format=html controller=Api::V1::NotificationsController action=index status=200 duration=199.29 view=158.53 db=16.51
Feb 26 14:45:02 ip-172-31-40-19 bundle[7579]: [81a490ed-bca9-4355-8f08-c9e68c764309]   Rendering home/index.html.haml within layouts/application
Feb 26 14:45:02 ip-172-31-40-19 bundle[7579]: [81a490ed-bca9-4355-8f08-c9e68c764309]   Rendered home/index.html.haml within layouts/application (1.1ms)
Feb 26 14:45:02 ip-172-31-40-19 bundle[7579]: [81a490ed-bca9-4355-8f08-c9e68c764309] method=GET path=/ format=html controller=HomeController action=index status=200 duration=21.97 view=4.39 db=3.33
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46]   Rendering stream_entries/show.html.haml within layouts/public
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46]   Rendered stream_entries/_og_description.html.haml (1.1ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46]   Rendered stream_entries/_og_image.html.haml (0.6ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46]   Rendered stream_entries/_simple_status.html.haml (4.3ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46]   Rendered collection of stream_entries/_status.html.haml [1 times] (4.8ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46]   Rendering app/javascript/images/logo.svg
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46]   Rendered app/javascript/images/logo.svg (0.0ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46]   Rendered stream_entries/_detailed_status.html.haml (10.4ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46]   Rendered stream_entries/_status.html.haml (16.3ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46]   Rendered application/_sidebar.html.haml (0.9ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46]   Rendered stream_entries/show.html.haml within layouts/public (25.1ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46]   Rendering app/javascript/images/logo_transparent.svg
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46]   Rendered app/javascript/images/logo_transparent.svg (0.0ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46]   Rendering layouts/application.html.haml
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46]   Rendered layouts/application.html.haml (1.7ms)
Feb 26 14:44:46 ip-172-31-40-19 bundle[7579]: [89833757-206f-4015-9796-34ff1816f7c1] method=GET path=/@Vesper/media format=*/* controller=AccountsController action=show status=200 duration=56.55 view=28.95 db=8.71
Feb 26 14:44:48 ip-172-31-40-19 bundle[7579]: [fa0572cb-ca92-40ab-a875-99c80b8182fe] method=GET path=/api/v1/timelines/home format=html controller=Api::V1::Timelines::HomeController action=show status=200 duration=8.16 view=0.40 db=2.34
Feb 26 14:44:48 ip-172-31-40-19 bundle[7579]: [b2ad4fd5-230b-467c-8258-79d80e27b70c] method=GET path=/api/v1/notifications format=html controller=Api::V1::NotificationsController action=index status=200 duration=199.29 view=158.53 db=16.51
Feb 26 14:45:02 ip-172-31-40-19 bundle[7579]: [81a490ed-bca9-4355-8f08-c9e68c764309] &#160;&#160;Rendering home/index.html.haml within layouts/application
Feb 26 14:45:02 ip-172-31-40-19 bundle[7579]: [81a490ed-bca9-4355-8f08-c9e68c764309] &#160;&#160;Rendered home/index.html.haml within layouts/application (1.1ms)
Feb 26 14:45:02 ip-172-31-40-19 bundle[7579]: [81a490ed-bca9-4355-8f08-c9e68c764309] method=GET path=/ format=html controller=HomeController action=index status=200 duration=21.97 view=4.39 db=3.33
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46] &#160;&#160;Rendering stream_entries/show.html.haml within layouts/public
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46] &#160;&#160;Rendered stream_entries/_og_description.html.haml (1.1ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46] &#160;&#160;Rendered stream_entries/_og_image.html.haml (0.6ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46] &#160;&#160;Rendered stream_entries/_simple_status.html.haml (4.3ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46] &#160;&#160;Rendered collection of stream_entries/_status.html.haml [1 times] (4.8ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46] &#160;&#160;Rendering app/javascript/images/logo.svg
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46] &#160;&#160;Rendered app/javascript/images/logo.svg (0.0ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46] &#160;&#160;Rendered stream_entries/_detailed_status.html.haml (10.4ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46] &#160;&#160;Rendered stream_entries/_status.html.haml (16.3ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46] &#160;&#160;Rendered application/_sidebar.html.haml (0.9ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46] &#160;&#160;Rendered stream_entries/show.html.haml within layouts/public (25.1ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46] &#160;&#160;Rendering app/javascript/images/logo_transparent.svg
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46] &#160;&#160;Rendered app/javascript/images/logo_transparent.svg (0.0ms)
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46] &#160;&#160;Rendering layouts/application.html.haml
Feb 26 14:45:06 ip-172-31-40-19 bundle[7579]: [8b60c849-6de9-4611-8d87-9690b602fc46] &#160;&#160;Rendered layouts/application.html.haml (1.7ms)
#4

can you post your nginx config and logs? it looks like the pages are being rendered, from the log you posted, but it seems like there’s something wrong with your upstream proxy. Or maybe there’s some sort of firewall on outgoing traffic?

#6

How did you upgrade maybe you made some mistake there? Share complete set of cmd you used one by one

I update like this

Stop all services
systemctl stop mastodon-web && systemctl stop mastodon-streaming && systemctl stop mastodon-sidekiq

su - mastodon
cd live
git fetch 
git checkout v2.7.3
gem update --system
bundle install
yarn install --pure-lockfile

then migrate db 
RAILS_ENV=production bundle exec rails db:migrate

Then Precompile
RAILS_ENV=production bundle exec rails assets:precompile

exit

restart all services
sudo systemctl restart mastodon-*
1 Like
#7

So, I think you’re on to something. On a lark, I tried using curl instead of Chrome, and using curl from my command line, I can go to https://signs.codes/about and I get some proper HTML back. I’m not seeing this on Chrome, though, and Lynx says something about “could not open the file for decompression”.

When I force lynx to not accept any special kind of encoding, then I actually can log in and it all works up to the point when Javascript becomes necessary. I notice that there’s a “content-encoding: gzip” header in the response. I’ve tried disabling gzip in my /etc/nginx/sites-available/signs.codes.conf to no avail.

ETA-- I downloaded an extension to Chrome that lets me manually set the “Accept-Encoding” header; when I set this header to an empty string, it’s possible to use my instance from Chrome. So, this seems pretty strongly to be about the gzip feature in nginx, which I can’t seem to actually turn off.

#8

Most of my backlog is gone, but I can confirm that’s the set of commands I used. As I mentioned to nightpool, my site’s actually in there and working as long as I use curl or Lynx with no content encoding accepted. Looks like something’s going weird with qzip compression in nginx?

1 Like
#9

So, I think this might have been related to my choice of using Ruby 2.6.0-rc1; I tried this all over again using Ruby 2.5.3 and didn’t continue having this issue.

1 Like
#10

Thanks a lot, you save me after 9 tuto to make mastodon updated from 2.7.1 and working in a LXC whithout " MASTODON systemd[1]: Failed to reset devices.list: Operation not permitted " … Tx!

#11

I didn’t understand what you mean by that. is your mastodon instance working correctly or is there any issue?

#12

yes, because i seen about 15h of tutorials to update/upgrade,you save me, all is done since i see your post :), so, just thanks for that :slight_smile:

1 Like
#13

Glad it helped you most of update goes Just like this But also check Release page of that version of mastodon as it contains Few information like in mastodon 2.8.0 Ruby version has been bumped to 2.6.1 so keep an eye on that.

Ruby Update will go like this

sudo su - mastodon
cd .rbenv
rbenv install 2.6.1
rbenv global 2.6.1

and sometime restart cmd which i added doesn’t restart all the services so start than one by one

systemctl start mastodon-web && systemctl start mastodon-streaming && systemctl start mastodon-sidekiq

1 Like