502 Bad Gateway: Connection refused while connecting to upstream

[RWinter]

After a while as a Mastodon user thought it time to start my own instance. I setup an Ubuntu VPS via Jelastic and followed the Mastodon install instructions ( Installing from source - Mastodon documentation (joinmastodon.org)).

Everything seemed to work, however, when I try and connect to my site I get a 502 Bad Gateway error. I checked the error logs but can’t work out how to resolve (I can’t post the full log as new users are limited to 3 links per post and Discourse is counting the links in the logs:

2021/02/22 00:48:29 [error] 16606#16606: *31 connect() failed (111: Connection refused) while connecting to upstream, client: 65.154.226.165, server: mastodon.winter.ink, request: “GET / HTTP/2.0”, upstream: “http://127.0.0.1:3000/500.html”, host: “mastodon.winter.ink”

Guidance on how to fix would be most appreciated.

[saper]

Are all mastodon services up and running (behind the webserver)? Seem like web service might be running.

I have promoted you to the next level so that you can post more.

[RWinter]

Many thanks for this.

I believe all services are running as when I go to the domain, I can see the favicon and it redirects to my users profile URL (just the page is blank).

I just tried a reinstall from scratch on a fresh instance (I am using Jelastic) and noticed that when I kept checking the domain name at each stage of the process, I saw the default Apache webpage until I configured NGINX. After which, I get a 502 error.

Here are the full error logs:

2021/02/27 22:22:16 [notice] 701#701: signal process started
2021/02/27 22:22:21 [notice] 730#730: signal process started
2021/02/27 22:22:24 [notice] 758#758: signal process started
2021/02/27 22:22:32 [notice] 786#786: signal process started
2021/02/27 22:22:38 [error] 787#787: *6 connect() failed (111: Connection refused) while connecting to upstream, client: 176.10.99.200, server: social.winter.ink, request: "HEAD / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "social.winter.ink"
2021/02/27 22:22:38 [error] 787#787: *6 connect() failed (111: Connection refused) while connecting to upstream, client: 176.10.99.200, server: social.winter.ink, request: "HEAD / HTTP/1.1", upstream: "http://127.0.0.1:3000/500.html", host: "social.winter.ink"
2021/02/27 22:22:38 [error] 787#787: *8 connect() failed (111: Connection refused) while connecting to upstream, client: 176.10.99.200, server: social.winter.ink, request: "HEAD / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "social.winter.ink"
2021/02/27 22:22:38 [error] 787#787: *8 connect() failed (111: Connection refused) while connecting to upstream, client: 176.10.99.200, server: social.winter.ink, request: "HEAD / HTTP/1.1", upstream: "http://127.0.0.1:3000/500.html", host: "social.winter.ink"
2021/02/27 22:22:46 [error] 787#787: *17 connect() failed (111: Connection refused) while connecting to upstream, client: 137.59.253.56, server: social.winter.ink, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "social.winter.ink"
2021/02/27 22:22:46 [error] 787#787: *17 connect() failed (111: Connection refused) while connecting to upstream, client: 137.59.253.56, server: social.winter.ink, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/500.html", host: "social.winter.ink"
2021/02/27 22:23:34 [error] 787#787: *20 connect() failed (111: Connection refused) while connecting to upstream, client: 176.10.99.200, server: social.winter.ink, request: "HEAD / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "social.winter.ink"
2021/02/27 22:23:34 [error] 787#787: *20 connect() failed (111: Connection refused) while connecting to upstream, client: 176.10.99.200, server: social.winter.ink, request: "HEAD / HTTP/1.1", upstream: "http://127.0.0.1:3000/500.html", host: "social.winter.ink"
2021/02/27 22:23:47 [error] 787#787: *24 connect() failed (111: Connection refused) while connecting to upstream, client: 46.19.141.82, server: social.winter.ink, request: "HEAD / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "social.winter.ink"
2021/02/27 22:23:47 [error] 787#787: *24 connect() failed (111: Connection refused) while connecting to upstream, client: 46.19.141.82, server: social.winter.ink, request: "HEAD / HTTP/1.1", upstream: "http://127.0.0.1:3000/500.html", host: "social.winter.ink"
2021/02/27 22:23:49 [error] 787#787: *29 connect() failed (111: Connection refused) while connecting to upstream, client: 23.129.64.220, server: social.winter.ink, request: "HEAD / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "social.winter.ink"
2021/02/27 22:23:49 [error] 787#787: *29 connect() failed (111: Connection refused) while connecting to upstream, client: 23.129.64.220, server: social.winter.ink, request: "HEAD / HTTP/1.1", upstream: "http://127.0.0.1:3000/500.html", host: "social.winter.ink"

Thank you in advance for your help.

[RWinter]

And for your reference, here is the full NGINX file contents where I had updated each ‘example.com’ with my domain:

map $http_upgrade $connection_upgrade {
default upgrade;
‘’ close;
}

upstream backend {
server 127.0.0.1:3000 fail_timeout=0;
}

upstream streaming {
server 127.0.0.1:4000 fail_timeout=0;
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=CACHE:10m inactive=7d max_size=1g;

server {
if ($host = social.winter.ink) {
return 301 https://$host$request_uri;
} # managed by Certbot

listen 80;
listen [::]:80;
server_name social.winter.ink;
root /home/mastodon/live/public;
location /.well-known/acme-challenge/ { allow all; }
location / { return 301 https://$host$request_uri; }

}

server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name social.winter.ink;

ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;

keepalive_timeout 70;
sendfile on;
client_max_body_size 80m;

root /home/mastodon/live/public;

gzip on;
gzip_disable “msie6”;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

add_header Strict-Transport-Security “max-age=31536000”;

location / {
try_files $uri @proxy;
}

location ~ ^/(emoji|packs|system/accounts/avatars|system/media_attachments/files) {
add_header Cache-Control “public, max-age=31536000, immutable”;
add_header Strict-Transport-Security “max-age=31536000”;
try_files $uri @proxy;
}

location /sw.js {
add_header Cache-Control “public, max-age=0”;
add_header Strict-Transport-Security “max-age=31536000”;
try_files $uri @proxy;
}

location @proxy {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header Proxy “”;
proxy_pass_header Server;

proxy_pass http://backend;
proxy_buffering on;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;

proxy_cache CACHE;
proxy_cache_valid 200 7d;
proxy_cache_valid 410 24h;
proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
add_header X-Cached $upstream_cache_status;
add_header Strict-Transport-Security "max-age=31536000";

tcp_nodelay on;

}

location /api/v1/streaming {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header Proxy “”;

proxy_pass http://streaming;
proxy_buffering off;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;

tcp_nodelay on;

}

error_page 500 501 502 503 504 /500.html;

ssl_certificate /etc/letsencrypt/live/social.winter.ink/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/social.winter.ink/privkey.pem; # managed by Certbot

}

[saper]

@RWinter

believe all services are running

We need to double check that… nginx will serve most of the Mastodon web interface without the web service running. Since you are on Ubuntu, you might what to check systemctl and then journalctl -xe -u mastodon-web to see what went wrong there (if it did).

[RWinter]

Many thanks for the reply. And yes, Houston… we have a problem.

Running the command systemctl showed the following services failed:

run-rpc_pipefs.mount
exim4.service
mastodon-streaming.service
setvtrgb.service
systemd-networkd.service
systemd-networkd.socket

Running journalctl -xe -u mastodon-web shows:

Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: ' duration=183.03 view=0.00 db=27.86
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: [218d0758-de26-44d3-8b04-24c9d4347b47]
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: [218d0758-de26-44d3-8b04-24c9d4347b47] ActionView::Template::Error (Webpacker can't find media/images/preview
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: 1. You want to set webpacker.yml value of compile to true for your environment
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]:    unless you are using the `webpack -w` or the webpack-dev-server.
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: 2. webpack has not yet re-run to reflect updates.
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: 3. You have misconfigured Webpacker's config/webpacker.yml file.
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: 4. Your webpack configuration is not creating a manifest.
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: Your manifest contains:
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: {
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: }
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: ):
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: [218d0758-de26-44d3-8b04-24c9d4347b47]     1: .hero-widget
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: [218d0758-de26-44d3-8b04-24c9d4347b47]     2:   .hero-widget__img
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: [218d0758-de26-44d3-8b04-24c9d4347b47]     3:     = image_tag @instance_presenter.hero&.file&.url || @instanc
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: [218d0758-de26-44d3-8b04-24c9d4347b47]     4:
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: [218d0758-de26-44d3-8b04-24c9d4347b47]     5:   .hero-widget__text
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: [218d0758-de26-44d3-8b04-24c9d4347b47]     6:     %p= @instance_presenter.site_short_description.html_safe.pr
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: [218d0758-de26-44d3-8b04-24c9d4347b47]
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: [218d0758-de26-44d3-8b04-24c9d4347b47] app/views/application/_sidebar.html.haml:3:in `_app_views_application_
Feb 28 11:19:43 node57683-env-7302246.jcloud-ver-jpe.ik-server.com bundle[400]: [218d0758-de26-44d3-8b04-24c9d4347b47] app/views/accounts/show.html.haml:86:in `_app_views_accounts_show_html

[Claire]

It seems you forgot to run the bundle exec rails assets:precompile step or it failed.

[RWinter]

Amazing @Claire! That did the trick! I have successfully deployed my mastodon instance.

And thank you @saper for the help. Great to have a community to which to turn.