Unable to start sidekiq, cannot open charlock_holmes

When I first setup my instance everything worked correctly, now after a reboot I get this error when trying to start sidekiq:

Nov 13 21:00:30 ArchKaruta audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=mastodon-sidekiq comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 13 21:00:30 ArchKaruta bundle[9417]: [9417] Puma starting in cluster mode...
Nov 13 21:00:30 ArchKaruta bundle[9417]: [9417] * Version 3.12.0 (ruby 2.5.1-p57), codename: Llamas in Pajamas
Nov 13 21:00:30 ArchKaruta bundle[9417]: [9417] * Min threads: 5, max threads: 5
Nov 13 21:00:30 ArchKaruta bundle[9417]: [9417] * Environment: production
Nov 13 21:00:30 ArchKaruta bundle[9417]: [9417] * Process workers: 2
Nov 13 21:00:30 ArchKaruta bundle[9417]: [9417] * Preloading application
Nov 13 21:00:32 ArchKaruta bundle[9417]: [9417] ! Unable to load application: LoadError: libicudata.so.62: cannot open shared object file: No such file or directory - /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/charlock_holmes-0.7.6/li>
Nov 13 21:00:32 ArchKaruta bundle[9417]: bundler: failed to load command: puma (/home/mastodon/live/vendor/bundle/ruby/2.5.0/bin/puma)
Nov 13 21:00:32 ArchKaruta bundle[9417]: LoadError: libicudata.so.62: cannot open shared object file: No such file or directory - /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/charlock_holmes-0.7.6/lib/charlock_holmes/charlock_holmes.so
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/charlock_holmes-0.7.6/lib/charlock_holmes.rb:1:in `<top (required)>'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:71:in `require'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:71:in `block in require_with_bootsnap_lfi'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:70:in `require_with_bootsnap_lfi'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:79:in `require'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:81:in `block (2 levels) in require'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:76:in `each'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:76:in `block in require'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:65:in `each'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:65:in `require'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler.rb:114:in `require'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/config/application.rb:7:in `<top (required)>'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/config/environment.rb:2:in `require_relative'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/config/environment.rb:2:in `<top (required)>'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   config.ru:4:in `require'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   config.ru:4:in `block in <main>'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/rack-2.0.5/lib/rack/builder.rb:55:in `instance_eval'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/rack-2.0.5/lib/rack/builder.rb:55:in `initialize'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   config.ru:in `new'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   config.ru:in `<main>'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/rack-2.0.5/lib/rack/builder.rb:49:in `eval'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/rack-2.0.5/lib/rack/builder.rb:49:in `new_from_string'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/rack-2.0.5/lib/rack/builder.rb:40:in `parse_file'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/puma-3.12.0/lib/puma/configuration.rb:318:in `load_rackup'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/puma-3.12.0/lib/puma/configuration.rb:243:in `app'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/puma-3.12.0/lib/puma/runner.rb:145:in `load_and_bind'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/puma-3.12.0/lib/puma/cluster.rb:410:in `run'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/puma-3.12.0/lib/puma/launcher.rb:184:in `run'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/puma-3.12.0/lib/puma/cli.rb:78:in `run'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/puma-3.12.0/bin/puma:10:in `<top (required)>'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/bin/puma:23:in `load'
Nov 13 21:00:32 ArchKaruta bundle[9417]:   /home/mastodon/live/vendor/bundle/ruby/2.5.0/bin/puma:23:in `<top (required)>'
Nov 13 21:00:32 ArchKaruta systemd[1]: mastodon-web.service: Main process exited, code=exited, status=1/FAILURE
Nov 13 21:00:32 ArchKaruta systemd[1]: mastodon-web.service: Failed with result 'exit-code'.

I decided to try executing the command from the systemd unit manually and checking that the files exist but I still receive the same errors. File ownership etc looks to all be in order.

Here is the output from manual execution:

[mastodon@ArchKaruta live]$ /home/mastodon/.rbenv/shims/bundle exec sidekiq -c 5 -q default -q push -q mailers -q pull
bundler: failed to load command: sidekiq (/home/mastodon/live/vendor/bundle/ruby/2.5.0/bin/sidekiq)
LoadError: libicudata.so.62: cannot open shared object file: No such file or directory - /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/charlock_holmes-0.7.6/lib/charlock_holmes/charlock_holmes.so
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/charlock_holmes-0.7.6/lib/charlock_holmes.rb:1:in `<top (required)>'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:71:in `require'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:71:in `block in require_with_bootsnap_lfi'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:70:in `require_with_bootsnap_lfi'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:79:in `require'
  /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:81:in `block (2 levels) in require'
  /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:76:in `each'
  /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:76:in `block in require'
  /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:65:in `each'
  /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:65:in `require'
  /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler.rb:114:in `require'
  /home/mastodon/live/config/application.rb:7:in `<top (required)>'
  /home/mastodon/live/config/environment.rb:2:in `require_relative'
  /home/mastodon/live/config/environment.rb:2:in `<top (required)>'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/cli.rb:272:in `require'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/cli.rb:272:in `boot_system'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/cli.rb:53:in `run'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/bin/sidekiq:12:in `<top (required)>'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/bin/sidekiq:23:in `load'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/bin/sidekiq:23:in `<top (required)>'

Any ideas on what I can do from here?

has anything changed since the last time you booted up your instance?

Nothing has changed other than my IP address as far as I’m aware of.

please check if you have ICU libraries installed and which version. You need to get libicudata.so.62 on your system again.

I have 63.1-2 installed, should I revert?

Edit: I tried reverting to 62.1-1 and got a different but similar error

[mastodon@ArchKaruta live]$ /home/mastodon/.rbenv/shims/bundle exec sidekiq -c 5 -q default -q push -q mailers -q pull
bundler: failed to load command: sidekiq (/home/mastodon/live/vendor/bundle/ruby/2.5.0/bin/sidekiq)
LoadError: libicuuc.so.63: cannot open shared object file: No such file or directory - /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/nokogumbo-1.5.0/lib/nokogumboc.so
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/nokogumbo-1.5.0/lib/nokogumbo.rb:2:in `<top (required)>'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/sanitize-4.6.4/lib/sanitize.rb:3:in `<top (required)>'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:71:in `require'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:71:in `block in require_with_bootsnap_lfi'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:70:in `require_with_bootsnap_lfi'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:79:in `require'
  /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:81:in `block (2 levels) in require'
  /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:76:in `each'
  /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:76:in `block in require'
  /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:65:in `each'
  /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:65:in `require'
  /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler.rb:114:in `require'
  /home/mastodon/live/config/application.rb:7:in `<top (required)>'
  /home/mastodon/live/config/environment.rb:2:in `require_relative'
  /home/mastodon/live/config/environment.rb:2:in `<top (required)>'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/cli.rb:272:in `require'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/cli.rb:272:in `boot_system'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/cli.rb:53:in `run'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/bin/sidekiq:12:in `<top (required)>'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/bin/sidekiq:23:in `load'
  /home/mastodon/live/vendor/bundle/ruby/2.5.0/bin/sidekiq:23:in `<top (required)>'

Looks like a total mess. You need both. You can also try to rebuild those rubygems from source, it should fix things.

If your instance server was updated, the best thing is to run a bundle pristine to rebuild the gems from source.

If your bundler version is not up-to-date, it might not rebuild the two gems that are pulled in from their sources (posix-spawn and http_parser.rb). I described a bit the solution for arch linux/rvm here, it might be similar for your configuration: Running a Mastodon instance using Arch Linux · Lond's Stuff

Ok after doing that and recompiling it works, but now anytime I restart it I need to recompile again else I get this error:

Nov 19 23:23:49 ArchKaruta audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=mastodon-streaming comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 19 23:23:50 ArchKaruta kernel: audit: type=1130 audit(1542687829.999:401697): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=mastodon-streaming comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 19 23:23:50 ArchKaruta kernel: audit: type=1131 audit(1542687829.999:401698): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=mastodon-streaming comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 19 23:23:50 ArchKaruta nginx[402]: 2018/11/19 23:23:50 [error] 8112#8112: *19936 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.86.1, server: koito.kaisei.app, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:3000/", host: "koito.kaisei.app", referrer: "https://koito.kaisei.app/sw.js"
Nov 19 23:23:50 ArchKaruta npm[22005]: > mastodon@ start /home/mastodon/live
Nov 19 23:23:50 ArchKaruta npm[22005]: > node ./streaming/index.js
Nov 19 23:23:50 ArchKaruta npm[22005]: /home/mastodon/live/node_modules/uws/uws.js:40
Nov 19 23:23:50 ArchKaruta npm[22005]:             throw new Error('Compilation of µWebSockets has failed and there is no pre-compiled binary ' +
Nov 19 23:23:50 ArchKaruta npm[22005]:             ^
Nov 19 23:23:50 ArchKaruta npm[22005]: Error: Compilation of µWebSockets has failed and there is no pre-compiled binary available for your system. Please install a supported C++11 compiler and reinstall the module 'uws'.
Nov 19 23:23:50 ArchKaruta npm[22005]:     at /home/mastodon/live/node_modules/uws/uws.js:40:19
Nov 19 23:23:50 ArchKaruta npm[22005]:     at Object.<anonymous> (/home/mastodon/live/node_modules/uws/uws.js:44:3)
Nov 19 23:23:50 ArchKaruta npm[22005]:     at Module._compile (internal/modules/cjs/loader.js:722:30)
Nov 19 23:23:50 ArchKaruta npm[22005]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:733:10)
Nov 19 23:23:50 ArchKaruta npm[22005]:     at Module.load (internal/modules/cjs/loader.js:620:32)
Nov 19 23:23:50 ArchKaruta npm[22005]:     at tryModuleLoad (internal/modules/cjs/loader.js:560:12)
Nov 19 23:23:50 ArchKaruta npm[22005]:     at Function.Module._load (internal/modules/cjs/loader.js:552:3)
Nov 19 23:23:50 ArchKaruta npm[22005]:     at Module.require (internal/modules/cjs/loader.js:658:17)
Nov 19 23:23:50 ArchKaruta npm[22005]:     at require (internal/modules/cjs/helpers.js:22:18)
Nov 19 23:23:50 ArchKaruta npm[22005]:     at Object.<anonymous> (/home/mastodon/live/streaming/index.js:10:19)
Nov 19 23:23:50 ArchKaruta npm[22005]: npm ERR! code ELIFECYCLE
Nov 19 23:23:50 ArchKaruta npm[22005]: npm ERR! errno 1
Nov 19 23:23:50 ArchKaruta npm[22005]: npm ERR! mastodon@ start: `node ./streaming/index.js`
Nov 19 23:23:50 ArchKaruta npm[22005]: npm ERR! Exit status 1
Nov 19 23:23:50 ArchKaruta npm[22005]: npm ERR!
Nov 19 23:23:50 ArchKaruta npm[22005]: npm ERR! Failed at the mastodon@ start script.
Nov 19 23:23:50 ArchKaruta npm[22005]: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
Nov 19 23:23:51 ArchKaruta npm[22005]: npm ERR! A complete log of this run can be found in:
Nov 19 23:23:51 ArchKaruta npm[22005]: npm ERR!     /home/mastodon/.npm/_logs/2018-11-20T04_23_50_963Z-debug.log
Nov 19 23:23:51 ArchKaruta systemd[1]: mastodon-streaming.service: Main process exited, code=exited, status=1/FAILURE
Nov 19 23:23:51 ArchKaruta systemd[1]: mastodon-streaming.service: Failed with result 'exit-code'.
Nov 19 23:23:51 ArchKaruta audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=mastodon-streaming comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Nov 19 23:23:51 ArchKaruta systemd[1]: mastodon-streaming.service: Service RestartSec=100ms expired, scheduling restart.
Nov 19 23:23:51 ArchKaruta systemd[1]: mastodon-streaming.service: Scheduled restart job, restart counter is at 5.

Maybe I missed a step?

Edit seems like the websocket isn’t working.

The step I said is related to the ruby dependencies of Mastodon, the error you’re seeing is related to the node dependencies, can you try doing a yarn install --pure-lockfile to see if it works or if it prints any errors?

The command gives no errors, and there is no change to the error produced when starting mastodon.

what does npm rebuild uws do ?

Closed due to inactivity