[SOLVED] No streaming after migrating to Nginx

So I migrated Todon.nl from Apache to Nginx. First I thought all went well, but now it seems nothing is streaming. In other words: the columns are not automatically updated. This all works fine under Apache. I followed the Production Guide. I can still switch to Apache if I want (stopping Nginx first) and than all works fine.

I’m using Nginx version 1.10.3 from the Debian jessie-backports). For reference: Apache is version 2.4.10. AND I’m using Mastodon 1.3.3 via Docker.

This is my Nginx configuration (note that I redirect todon.nl:80, www.todon.nl:80 and www.todon:443 to https://todon.nl):

map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

server {
  listen 80;
  listen [::]:80;
  server_name todon.nl www.todon.nl;
  # Useful for Let's Encrypt
  location /.well-known/acme-challenge/ { allow all; }
  location / { return 301 https://todon.nl$request_uri; }
}

server {
  listen 443 ssl;
  listen [::]:443 ssl;
  server_name www.todon.nl;

  ssl_protocols TLSv1.2;
  ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;

  ssl_certificate     /etc/letsencrypt/live/todon.nl/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/todon.nl/privkey.pem;
  ssl_dhparam         /etc/ssl/certs/dhparam.pem;

  # Useful for Let's Encrypt
  location /.well-known/acme-challenge/ { allow all; }
  location / { return 301 https://todon.nl$request_uri; }
}

server {
  listen 443 ssl;
  listen [::]:443 ssl;
  server_name todon.nl;

  ssl_protocols TLSv1.2;
  ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;

  ssl_certificate     /etc/letsencrypt/live/todon.nl/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/todon.nl/privkey.pem;
  ssl_dhparam         /etc/ssl/certs/dhparam.pem;

  keepalive_timeout    70;
  sendfile             on;
  client_max_body_size 0;

  root /home/todon/live/public;

  gzip on;
  gzip_disable "msie6";
  gzip_vary on;
  gzip_proxied any;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

  add_header Strict-Transport-Security "max-age=31536000";
  add_header Content-Security-Policy "style-src 'self' 'unsafe-inline'; script-src 'self'; object-src 'self'; img-src data: https:; media-src data: https:; connect-src 'self' wss://example.com; upgrade-insecure-requests";

  location / {
    try_files $uri @proxy;
  }

  location ~ ^/(assets|system/media_attachments/files|system/accounts/avatars) {
    add_header Cache-Control "public, max-age=31536000, immutable";
    try_files $uri @proxy;
  }

  location @proxy {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header Proxy "";
    proxy_pass_header Server;

    proxy_pass http://127.0.0.1:3000;
    proxy_buffering off;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    tcp_nodelay on;
  }

  location /api/v1/streaming {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header Proxy "";

    proxy_pass http://localhost:4000;
    proxy_buffering off;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    tcp_nodelay on;
  }

  error_page 500 501 502 503 504 /500.html;
}

For reference here is my Apache configuration (without the 80/www redirects):

<IfModule mod_ssl.c>
<VirtualHost *:443>
        ServerAdmin admin@localhost
        ServerName todon.nl
        ServerAlias www.todon.nl

        DocumentRoot /home/todon/live/public/

        Header add Strict-Transport-Security "max-age=31536000"
SSLCertificateFile /etc/letsencrypt/live/todon.nl/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/todon.nl/privkey.pem
        Include /etc/letsencrypt/options-ssl-apache.conf

        ErrorLog ${APACHE_LOG_DIR}/todon_error.log
        LogLevel error
        CustomLog ${APACHE_LOG_DIR}/todon_access.log combined

        ProxyPreserveHost On
        RequestHeader set X-Forwarded-Proto "https"
        ProxyPass /500.html !
        ProxyPass /oops.png !
        ProxyPass /api/v1/streaming/ ws://localhost:4000/
        ProxyPassReverse /api/v1/streaming/ ws://localhost:4000/
        ProxyPass / http://localhost:3000/
        ProxyPassReverse / http://localhost:3000/

        ErrorDocument 500 /500.html
        ErrorDocument 501 /500.html
        ErrorDocument 502 /500.html
        ErrorDocument 503 /500.html
        ErrorDocument 504 /500.html
</VirtualHost>
</IfModule>

Please advise?

You’re going to hate me for asking this:
Did you clear your cookies in your browser? This has been the solution for all streaming issues on a user-level.

If the streaming issues are actually server-side you may disregard my suggestion.

It is also in Chromium and when I switch back to Apache and da ctrl+f5 it all works again. But I can try again.

1 Like

Also, I’m not sure if this is important in nginx - I’ve heard suggestions that it is, but I don’t know why it would be - but I see you’re linking to 127.0.0.1:3000 for the standard proxy but localhost:4000 for the streaming API. Try changing that to 127.0.0.1:4000 and restarting nginx?

1 Like

That’s interesting. Let me try that second after I cleared out the cookies (that probably won’t solve it).

1 Like

Both solutions didn’t had any effect. :disappointed:

Changing ports like that is not going to help, those are actually two different services. Anything interesting in the logs? can you connect to localhost 3000 and localhost 4000 both via IPv4 and IPv6 (::1) ?

I added http2 and removed the www.todon.nl redirect (for clarity, can add it later back). But still no effect.

@saper Remember that with Apache it simply works. How to access http://localhost:3000 and http://localhost:4000 from my VPS commandline. If I try w3m http://localhost:4000 it wants to download an index.html file with this:

{“error”:“Error: Missing access token”}

do you have netcat (“nc”) installed?

w3m has also -4 and -6 options (or try http://127.0.0.1:3000 and http://[::1]:3000)

That’s the unlikely reason but do check. Anything in the logs?

It’s the localhost -> 127.0.0.1 that was important there, but it doesn’t seem to have made a difference. The only substantive difference I see between your nginx config and mine is that I’ve disabled ipv6.

Can you confirm through Docker (maybe sidekiq?) that the streaming service is running? I’m not running Docker so I don’t remember how that’s checked.

Missing access token on port 4000 could indicate running streaming service to me…

Yes I have netcat.

If I try to w3m http://127.0.0.1:3000 it is complaining about a ‘bad cert ident’ from another vhost on my VPS.

uh-oh, definitely something to check with netstat -a

Yeah, after looking, the missing access token is what it’s supposed to return if it’s accessed directly.

Well, I’m stumped. Sorry!

I disabled IPv6, but still the problem.

Let me disable all other vhosts first,

@saper I disabled all other vhosts, but still not a result.

I did run a netstat -a and these are the lines with 3000 and 4000:

Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 172.19.0.1:53370        172.19.0.5:4000         ESTABLISHED
tcp        0      0 localhost:42244         localhost:3000          TIME_WAIT  
tcp        0      0 172.19.0.1:53368        172.19.0.5:4000         ESTABLISHED
tcp        0      0 localhost:42259         localhost:3000          TIME_WAIT  
tcp        0      0 172.19.0.1:49494        172.19.0.4:3000         TIME_WAIT  
tcp        0      0 localhost:36318         localhost:4000          ESTABLISHED
tcp        0      0 localhost:36306         localhost:4000          ESTABLISHED
tcp        0      0 172.19.0.1:49506        172.19.0.4:3000         TIME_WAIT  
tcp        0      0 localhost:42251         localhost:3000          TIME_WAIT  
tcp        0      0 172.19.0.1:49492        172.19.0.4:3000         TIME_WAIT  
tcp        0      0 172.19.0.1:49485        172.19.0.4:3000         TIME_WAIT  
tcp        0      0 localhost:36308         localhost:4000          ESTABLISHED
tcp        0      0 172.19.0.1:53380        172.19.0.5:4000         ESTABLISHED
tcp        0      0 localhost:42247         localhost:3000          TIME_WAIT  
tcp        0      0 localhost:42262         localhost:3000          TIME_WAIT

Two connections to localhost:4000 seem to be ESTABLISHED, so it seems fine to me

On my instance I have changed localhost to actual non-loopback IPv4 addresses to avoid loopback stories with FreeBSD jails. But that requires making sure other stuff listens there too.

Yes it does, cause when I do a:

service nginx stop
and a
service apache2 start

streaming works fine.