Streaming fails to start: uws_linux_59.node: undefined symbol: SSL_library_init

Nothing happend, just a reboot and mastodon-streaming fails to start. Here the systemd log. Unlike described here I’m not able to recover using

npm rebuild uws

I’ve even tried to remove node_modules/uws and installed it back using yarn add uws, rebuilt binaries and nothing help. Still get the error.

[mastodon@server live]$ NODE_ENV=production /usr/bin/npm run start

> mastodon@ start /home/mastodon/live
> node ./streaming/index.js

/home/mastodon/live/node_modules/uws/uws.js:38
            throw new Error('Compilation of µWebSockets has failed and there is no pre-compiled binary ' +
            ^

Error: Compilation of µWebSockets has failed and there is no pre-compiled binary available for your system. Please install a supported C++11 compiler and reinstall the module 'uws'.
    at native (/home/mastodon/live/node_modules/uws/uws.js:38:19)
    at Object.<anonymous> (/home/mastodon/live/node_modules/uws/uws.js:42:3)
    at Module._compile (module.js:649:30)
    at Object.Module._extensions..js (module.js:660:10)
    at Module.load (module.js:561:32)
    at tryModuleLoad (module.js:501:12)
    at Function.Module._load (module.js:493:3)
    at Module.require (module.js:593:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/home/mastodon/live/streaming/index.js:10:19)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! mastodon@ start: `node ./streaming/index.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the mastodon@ start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/mastodon/.npm/_logs/2018-03-17T14_27_22_722Z-debug.log

uws build log
And I run it as standalone installation.

It’s compiling just fine with node-gyp rebuild

Weird. I’ve added a console.print function to see what binary it’s looking fore and its’ uws_linux_59 and it’s there! But it doesn’t see it!

[mastodon@server live]$ ls node_modules/uws/                                                                                                    
binding.gyp    LICENSE       uws_darwin_46.node  uws_darwin_51.node  uws.js             uws_linux_48.node      uws_win32_51.node 
build          package.json  uws_darwin_47.node  uws_darwin_57.node  uws_linux_46.node  uws_linux_51.node  uws_linux_59.node  uws_win32_57.node
build_log.txt  src           uws_darwin_48.node  uws_darwin_59.node  uws_linux_47.node  uws_linux_57.node  uws_win32_48.node  uws_win32_59.node
[mastodon@server live]$ NODE_ENV=production /usr/bin/npm run start                                                                              
                                                                                                                                                
> mastodon@ start /home/mastodon/live
> node ./streaming/index.js

MODULE TO LOOKUP: ./uws_linux_59                                                                                                                                  
/home/mastodon/live/node_modules/uws/uws.js:39                                                                                                  
            throw new Error('Compilation of µWebSockets has failed and there is no pre-compiled binary ' +                                      

what does your node -v say?

also, can you post full output of your yarn install stage?

[mastodon@server live]$ node -v
v9.8.0

yarn install --pure-lock --verbose log

What happens if you cd live/node_modules/uws and then try

node -p "require('./uws_linux_59')"
module.js:678                                                                                                                    
  return process.dlopen(module, path.toNamespacedPath(filename));
                 ^

Error: /home/mastodon/live/node_modules/uws/uws_linux_59.node: undefined symbol: SSL_library_init

Where is your node engine coming from? nodejs.org? operating system package?

Can you post the output of

ldd `which node`

Also

node -p process.config

It’s nodejs 9.8 from archlinux package repo. Here is the requested output

The node engine uses node_shared_openssl: true, which means it uses OpenSSL from archlinux and it does not have OpenSSL compiled into node. This maybe a source of the problem. There is a problem filed because node on Arch Linux uses OpenSSL 1.1, but I do not think that uws is incompatible.

As a workaround, try node downloaded from Index of /dist/v9.8.0/ - it might work better.

One thing I don’t understand is that your locally compiled copy (npm rebuild uws) should work.

Can you try

 node -p "require('./uws_linux_59')"

on your rebuilt copy?

If you get the same issue, you might want to try adding LIBS := -lcrypto -lssl somewhere in the build/uws.target.mk (there is certainly some way to set when calling npm rebuild but I don’t have a time to figure it out now). Also please set V=1 in your environment when building to see full compiler command line.

(added: did you copy uws.node as uws_linux_59.node to your node_modules/uws?)

Using downloaded node:

[mastodon@server node_modules]$ /home/speranza/.build/node-v9.8.0-linux-x64/bin/npm build uws                                    
                                                                                                                                 
> uws@9.14.0 install /home/mastodon/live/node_modules/uws
> node-gyp rebuild > build_log.txt 2>&1 || exit 0

[mastodon@server node_modules]$ cd uws/                                                                                          
[mastodon@server uws]$ node -p "require('./uws_linux_59')"                                                                       
module.js:678                                                                                                                    
  return process.dlopen(module, path.toNamespacedPath(filename));
                 ^

Error: /home/mastodon/live/node_modules/uws/uws_linux_59.node: undefined symbol: SSL_library_init
    at Object.Module._extensions..node (module.js:678:18)
    at Module.load (module.js:561:32)
    at tryModuleLoad (module.js:501:12)
    at Function.Module._load (module.js:493:3)
    at Module.require (module.js:593:17)
    at require (internal/module.js:11:18)
    at [eval]:1:1
    at Script.runInThisContext (vm.js:65:33)
    at Object.runInThisContext (vm.js:197:38)
    at Object.<anonymous> ([eval]-wrapper:6:22)

Can you try

node -p “require(’./uws_linux_59’)”

on your rebuilt copy?

If you get the same issue, you might want to try adding LIBS := -lcrypto -lssl somewhere in the build/uws.target.mk (there is certainly some way to set when calling npm rebuild but I don’t have a time to figure it out now).

Nothing helps. Should I set V=1 as export V=1? If so- doesn’t change anything.

Actually, I think I’ll give up if you don’t mind. It’s my local issue obviously, so I’ll try to reinstall using Docker container then.

I’d try with node downloaded from nodejs.org first.

Docker has its own issues :slight_smile:

IDK how to make the downloaded binaries systemwide, but seems like requre does work with the nodejs.org bin:

[mastodon@server uws]$ /home/speranza/.build/node-v9.8.0-linux-x64/bin/node -p "require('./uws_linux_59')"                              [16/1834]
{ server:                                                                                                                               [15/1834]
   { send: [Function: send],                                                                                                            [14/1834]
     close: [Function: close],                                                                                                          [13/1834]
     terminate: [Function: terminate],                                                                                                  [12/1834]
     prepareMessage: [Function: prepareMessage],                                                                                        [11/1834]
     sendPrepared: [Function: sendPrepared],                                                                                            [10/1834]
     finalizeMessage: [Function: finalizeMessage],                                                                                       [9/1834]
     group:                                                                                                                              [8/1834]
      { onConnection: [Function: onConnection],                                                                                          [7/1834]
        onMessage: [Function: onMessage],                                                                                                [6/1834]
        onDisconnection: [Function: onDisconnection],                                                                                    [5/1834]
        forEach: [Function: forEach],                                                                                                    [4/1834]
        getSize: [Function: getSize],                                                                                                    [3/1834]
        startAutoPing: [Function: startAutoPing],                                                                                        [2/1834]
        listen: [Function: listen],                                                                                                      [1/1834]
        onPing: [Function: onPing],
        onPong: [Function: onPong],
        create: [Function: create],
        delete: [Function: delete],
        close: [Function: close],
        terminate: [Function: terminate],
        broadcast: [Function: broadcast] } },
  client:
   { send: [Function: send],
     close: [Function: close],
     terminate: [Function: terminate],
     prepareMessage: [Function: prepareMessage],
     sendPrepared: [Function: sendPrepared],
     finalizeMessage: [Function: finalizeMessage],
     group:
      { onConnection: [Function: onConnection],
        onMessage: [Function: onMessage],
        onDisconnection: [Function: onDisconnection],
        onError: [Function: onError],
        onPing: [Function: onPing],
        onPong: [Function: onPong],
        create: [Function: create],
        delete: [Function: delete],
        close: [Function: close],
        terminate: [Function: terminate],
        broadcast: [Function: broadcast] } },
  httpServer:
   { [Function]
     createServer: [Function: createServer],
     getExpressApp: [Function: getExpressApp],
     getResponsePrototype: [Function: getResponsePrototype],
     getRequestPrototype: [Function: getRequestPrototype] },
  setUserData: [Function: setUserData],
  getUserData: [Function: getUserData],
  clearUserData: [Function: clearUserData],
  getAddress: [Function: getAddress],
  transfer: [Function: transfer],
  upgrade: [Function: upgrade],
  connect: [Function: connect],
  setNoop: [Function: setNoop] }

I’ve created a symbolic link for /usr/bin/node pointing to the downloaded bin and mastodon-streaming is working now. Thank you!

Yes, or modify PATH and point it first to the path where node is unpacked.

I would report the bug to the Arch Linux tracker, although it probably needs to be fixed somewhere else (uws or node-gyp). uws closed they issue tracker, they don’t need one :frowning:

Glad you got it working!