[Solved] Broken Streaming (error related to index.js)


#1

Hello,

I open this tred because streaming doesn’t work on my Mastodon instance, public timeline doesn’t refresh. I already encountered this problem last May after a Mastodon update. It was solved when I upgraded my server to Debian Stretch and reinstalled Mastodon.

But the problem reappeared today, after 2-3 months of good functioning.

The only thing that’s changed is that I have restore directory media from a backup on my instance (full story here).
I’m not sure there is a link between the two, because I only made channge in the /public/system/accounts/ directory. But the probleme have already occure before restart few hours after restoring media.

In web browser, I have this message :

Firefox cannot establish a connection with the server at the following address wss://mastd.racing/api/v1/streaming/?access_token=[...]stream=user.  index.js:4:649

I’ve tried the instructions given here :

  • I have downgraded nodejs from version 7. x to version 6. x

  • I checked without problem process.version; Array.includes; Array.prototype.includes

  • I tried to add those following lines to streaming/index.js unsuccessfully and reversed the changes :

    import includes from ‘array-includes’;
    delete Array.prototype.includes;
    includes.shim();

In npm error log, I have this :

0 info it worked if it ends with ok
1 verbose cli [ '/usr/local/bin/node', '/usr/bin/npm', 'run', 'start' ]
2 info using npm@3.10.10
3 info using node@v6.11.4
4 verbose run-script [ 'prestart', 'start', 'poststart' ]
5 info lifecycle mastodon@~prestart: mastodon@
6 silly lifecycle mastodon@~prestart: no script for prestart, continuing
7 info lifecycle mastodon@~start: mastodon@
8 verbose lifecycle mastodon@~start: unsafe-perm in lifecycle true
9 verbose lifecycle mastodon@~start: PATH: /usr/lib/node_modules/npm/bin/node-gyp-bin:/home/mastodon/live/node_modules/.bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
10 verbose lifecycle mastodon@~start: CWD: /home/mastodon/live
11 silly lifecycle mastodon@~start: Args: [ '-c', 'node ./streaming/index.js' ]
12 silly lifecycle mastodon@~start: Returned: code: 1  signal: null
13 info lifecycle mastodon@~start: Failed to exec start script
14 verbose stack Error: mastodon@ start: `node ./streaming/index.js`
14 verbose stack Exit status 1
14 verbose stack     at EventEmitter.<anonymous> (/usr/lib/node_modules/npm/lib/utils/lifecycle.js:255:16)
14 verbose stack     at emitTwo (events.js:106:13)
14 verbose stack     at EventEmitter.emit (events.js:191:7)
14 verbose stack     at ChildProcess.<anonymous> (/usr/lib/node_modules/npm/lib/utils/spawn.js:40:14)
14 verbose stack     at emitTwo (events.js:106:13)
14 verbose stack     at ChildProcess.emit (events.js:191:7)
14 verbose stack     at maybeClose (internal/child_process.js:920:16)
14 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:230:5)
15 verbose pkgid mastodon@
16 verbose cwd /home/mastodon/live
17 error Linux 4.9.20-std-1
18 error argv "/usr/local/bin/node" "/usr/bin/npm" "run" "start"
19 error node v6.11.4
20 error npm  v3.10.10
21 error code ELIFECYCLE
22 error mastodon@ start: `node ./streaming/index.js`
22 error Exit status 1
23 error Failed at the mastodon@ start script 'node ./streaming/index.js'.
23 error Make sure you have the latest version of node.js and npm installed.
23 error If you do, this is most likely a problem with the mastodon package,
23 error not with npm itself.
23 error Tell the author that this fails on your system:
23 error     node ./streaming/index.js
23 error You can get information on how to open an issue for this project with:
23 error     npm bugs mastodon
23 error Or if that isn't available, you can get their info via:
23 error     npm owner ls mastodon
23 error There is likely additional logging output above.
24 verbose exit [ 1, true ]

In nginx error log :
I have those lines corresponding to a python script running on the same server forwarding of tweets from Twitter accounts to a Mastodon one :

2017/10/05 06:42:19 [error] 3452#3452: *361684 client intended to send too large body: 1313962 bytes, client: 212.47.246.104, server: mastd.racing, request: "POST /api/v1/media HTTP/1.1", host: "mastd.racing"
2017/10/05 06:42:19 [error] 3453#3453: *361685 client intended to send too large body: 1168014 bytes, client: 212.47.246.104, server: mastd.racing, request: "POST /api/v1/media HTTP/1.1", host: "mastd.racing"

and those lines correspondig to my web browser :

2017/10/05 09:48:09 [error] 3453#3453: *365750 connect() failed (111: Connection refused) while connecting to upstream, client: XX.X.XX.XXX, server: mastd.racing, request: "GET /api/v1/streaming/?access_token=a733ac9912be6499$
2017/10/05 09:48:09 [error] 3452#3452: *365754 connect() failed (111: Connection refused) while connecting to upstream, client: XX.X.XX.XXX, server: mastd.racing, request: "GET /api/v1/streaming/?access_token=a733ac9912be6499$

They are repeated every minute.

I don’t think there are any problems with my nginx configuration, because it worked very well before.
But here it is:

map $http_upgrade $connection_upgrade {
        default upgrade;
        ''        close;
}

server {
        listen 80;
        listen [::]:80;
        server_name mastd.racing;
        return 301 https://mastd.racing$request_uri;

        access_log /dev/null;
        error_log /dev/null;
}

server {
        listen 443 ssl;
        #listen [::]:443 ssl;
        server_name mastd.racing;

        ssl_certificate /etc/letsencrypt/live/www.mastd.racing/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/www.mastd.racing/privkey.pem;
        ssl_trusted_certificate /etc/letsencrypt/live/www.mastd.racing/chain.pem;
        ssl_protocols TLSv1.2;
        ssl_ecdh_curve prime256v1;
        ssl_ciphers EECDH+AESGCM:EECDH+AES;
        ssl_prefer_server_ciphers on;
        resolver 80.67.169.12 80.67.169.40 valid=300s;
        resolver_timeout 5s;
        ssl_session_cache shared:SSL:10m;
        add_header Strict-Transport-Security "max-age=15768000";

        add_header Referrer-Policy "strict-origin-when-cross-origin";
        add_header Content-Security-Policy "default-src 'none'; font-src 'self'; media-src 'self'; style-src 'self' 'unsafe-inline'; script-src 'self'; img-src 'self' data:; connect-src 'self' wss://mastd.racing; frame-ancestors 'none';";


        root /home/mastodon/live/public;

        location / {
                try_files $uri @proxy;
        }

        location @proxy {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto https;
                proxy_set_header Proxy "";
                proxy_pass_header Server;

                proxy_pass http://127.0.0.1:3000;
                proxy_buffering off;
                proxy_redirect off;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection $connection_upgrade;

                tcp_nodelay on;
}

        location /api/v1/streaming {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto https;
                proxy_set_header Proxy "";

                proxy_pass http://127.0.0.1:4000;
                proxy_buffering off;
                proxy_redirect off;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_pass_header Server;

                proxy_pass http://127.0.0.1:3000;
                proxy_buffering off;
                proxy_redirect off;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection $connection_upgrade;

                tcp_nodelay on;
        }

        location /api/v1/streaming {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto https;
                proxy_set_header Proxy "";

                proxy_pass http://127.0.0.1:4000;
                proxy_buffering off;
                proxy_redirect off;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection $connection_upgrade;

                tcp_nodelay on;
        }

        error_page 500 501 502 503 504 /500.html;
}

Do any of you have any ideas?

Thank you for your help.


#2

I think I’ve found the solution (from here).

In production directory :

NODE_ENV=production /usr/bin/npm run start

/home/mastodon/live/node_modules/uws/uws.js:38
            throw new Error('Compilation of µWebSockets has failed and there isno pre-compiled binary ' +
            ^

Error: Compilation of µWebSockets has failed and there is no pre-compiled binary available for your system. Please install a supported C++11 compiler and reinstall the module 'uws'.
    at e (/home/mastodon/live/node_modules/uws/uws.js:38:19)
    at Object.<anonymous> (/home/mastodon/live/node_modules/uws/uws.js:42:3)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/home/mastodon/live/streaming/index.js:10:19)

then :

npm rebuild uws

at last, as root :

systemctl restart mastodon-streaming.service

I don’t understand how the binary could have disappeared or why I didn’t have a compilation error (I reinstalled from the master to see if the problem came from there), but the problem seems solved.

I’ll test for a few hours and if all goes well I’ll close the discussion.