Upgrade from v1.3.3 to v1.4.1 broke streaming

Heyo, i already asked in the Administration Matrix groupchat and it appears to (should) be fixed in the latest releasetag already - however, this upgrade broke streaming and i am running out of ideas bringing it back to work. :confused:

It often dies with TypeError: req.filteredLanguages.includes is not a function (mainly)
…and sometimes with an invalid auth key error (which i didn’t see lately).


  • The browser dev console doesn’t show anything interesting (like invalid auth key) - just:
Server: nginx
Date: Mon, 29 May 2017 20:55:48 GMT
Connection: upgrade
Upgrade: websocket
Sec-WebSocket-Accept: ddU...5k=
Sec-WebSocket-Extensions: permessage-deflate; client_no_context_takeover
Sec-WebSocket-Protocol: undefined
Sec-WebSocket-Version: 13
WebSocket-Server: uWebSockets

for the response headers.

  • The nginx config for streaming should match the one from the production guide.

  • nodejs should match the deps:

~ % node -v
v7.9.0
  • my startup command for streaming is a tiny bit different:
    ExecStart=/bin/bash -lc "true; source /home/user/.profile; /usr/bin/npm run start

  • i have no streaming url configured


It was suggested to me opening an issue but since it doesn’t appear to be a popular problem, i thought i should consult the forum instead. :cat2:

~greetings

Not having such issues on mastodon.social, with Node 6.10.3… The clause that checks req.filteredLanguages.includes first checks Array.isArray(req.filteredLanguages) so the problem isn’t that it isn’t an array, but a missing prototype function in the Array class. Is there any way your script might still be using the wrong Node version somehow?

Hi,

i did not change anything on my node setup - it was already on v7.9.0 - unfortunally… .

Edit:

Wasn’t sure how to reversecheck but i just did the following:

sudo -u user /bin/bash -lc "true; source /home/user/.profile; node -v"                                                         :(
v7.6.0

Okey, i just had the idea to run node -v in ExecStart - it prints v7.6.0.

Edit:

Now it matches v7.9.0 - but no difference yet.

Edit2:

I also compared the upstream and my index.js copy:

md5sum index.js 
-> 5832c45c8c35a5f637ffadef4d97fd9f  index.js
md5sum index2.js
-> 5832c45c8c35a5f637ffadef4d97fd9f  index2.js
```

That's also fine - and i didn't see any difference to do in the nginx config... so ... yea ... :confused:

Sorry for the doubleposts but i think it is valid in this specific case.

Today i compared the browserconsole with mastodon.social.
The Responseheaders are so far the same - interestingly i didn’t notice that the streaming workers crash (connection refused) on this server.

However - i did closer log checks and even upgraded my nginx from backports for http2 support.
There is nothing to see - in any case - nothing in public/federated (home/notifications appears to work fine, still) and nothing in the streaming or nginx logs (except the frequently happening crash)

I also diffed the suggested an my nginx file for this domain and didn’t find anything - maybe i didn’t see something?

map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

server{
    listen 80;
    listen [::]:80;
    server_name social.tchncs.de;
    return 301 https://social.tchncs.de$request_uri;
}

server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2 ipv6only=on;
  server_name social.tchncs.de;
  keepalive_timeout    70;
  sendfile             on;
  client_max_body_size 0;

  root /var/www/mastodon;
  gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

location / {
    try_files $uri @proxy;       
}

location @proxy {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header Proxy "";
    proxy_pass_header Server;

    proxy_pass http://127.0.0.1:30022;
    proxy_buffering off;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    tcp_nodelay on;
}

  location /api/v1/streaming {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header Proxy "";
 
    proxy_pass http://127.0.0.1:4000;
    proxy_buffering off;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade; 
    proxy_set_header Connection $connection_upgrade;

    tcp_nodelay on;
}

    ssl_certificate /path/to/fullchain.pem;
    ssl_certificate_key /path/to/privkey.pem;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
    ssl_stapling on;
    
    error_page 404 /404.html;
    error_page 500 501 502 503 504 /500.html;
    location = /500.html {
 }
}

Edit:

Oh, and i now downgraded to nodejs v6.10.3 and to ensure the nodeversion, i still have the node -v command in ExecStart.

@milan Can you use the node REPL to test some things? node should open the REPL

Check:

process.version
Array.includes
Array.prototype.includes

At last, try this:

Edit streaming/index.js and add this line to the top:

import includes from 'array-includes';

And below the other imports:

delete Array.prototype.includes;
includes.shim();

And see if that resolves your issue. If it does, we’ll need to do that upstream.

1 Like

I had the same TypeError: req.filteredLanguages.includes is not a function errors until I cleaned a messed up nodejs installation. Now I see many occurrences of WARN: ActiveRecord::RecordNotFound: Couldn’t find Status with ‘id’=some id, which corresponds to replies to toots I already received.

Also, I can find the id in the statuses table…

Any idea what is causing this?

Thanks


> process.version
'v6.10.3'
> Array.includes
undefined
> Array.prototype.includes
[Function: includes]

Editing index.js fixed it! :tada:

Will mark as solved later as my internet connection is currently too
weak for opening the forum. :cat2: