When I first setup my instance everything worked correctly, now after a reboot I get this error when trying to start sidekiq:
Nov 13 21:00:30 ArchKaruta audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=mastodon-sidekiq comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 13 21:00:30 ArchKaruta bundle[9417]: [9417] Puma starting in cluster mode...
Nov 13 21:00:30 ArchKaruta bundle[9417]: [9417] * Version 3.12.0 (ruby 2.5.1-p57), codename: Llamas in Pajamas
Nov 13 21:00:30 ArchKaruta bundle[9417]: [9417] * Min threads: 5, max threads: 5
Nov 13 21:00:30 ArchKaruta bundle[9417]: [9417] * Environment: production
Nov 13 21:00:30 ArchKaruta bundle[9417]: [9417] * Process workers: 2
Nov 13 21:00:30 ArchKaruta bundle[9417]: [9417] * Preloading application
Nov 13 21:00:32 ArchKaruta bundle[9417]: [9417] ! Unable to load application: LoadError: libicudata.so.62: cannot open shared object file: No such file or directory - /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/charlock_holmes-0.7.6/li>
Nov 13 21:00:32 ArchKaruta bundle[9417]: bundler: failed to load command: puma (/home/mastodon/live/vendor/bundle/ruby/2.5.0/bin/puma)
Nov 13 21:00:32 ArchKaruta bundle[9417]: LoadError: libicudata.so.62: cannot open shared object file: No such file or directory - /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/charlock_holmes-0.7.6/lib/charlock_holmes/charlock_holmes.so
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/charlock_holmes-0.7.6/lib/charlock_holmes.rb:1:in `<top (required)>'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:71:in `require'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:71:in `block in require_with_bootsnap_lfi'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:70:in `require_with_bootsnap_lfi'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:79:in `require'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:81:in `block (2 levels) in require'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:76:in `each'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:76:in `block in require'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:65:in `each'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:65:in `require'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler.rb:114:in `require'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/config/application.rb:7:in `<top (required)>'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/config/environment.rb:2:in `require_relative'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/config/environment.rb:2:in `<top (required)>'
Nov 13 21:00:32 ArchKaruta bundle[9417]: config.ru:4:in `require'
Nov 13 21:00:32 ArchKaruta bundle[9417]: config.ru:4:in `block in <main>'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/rack-2.0.5/lib/rack/builder.rb:55:in `instance_eval'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/rack-2.0.5/lib/rack/builder.rb:55:in `initialize'
Nov 13 21:00:32 ArchKaruta bundle[9417]: config.ru:in `new'
Nov 13 21:00:32 ArchKaruta bundle[9417]: config.ru:in `<main>'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/rack-2.0.5/lib/rack/builder.rb:49:in `eval'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/rack-2.0.5/lib/rack/builder.rb:49:in `new_from_string'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/rack-2.0.5/lib/rack/builder.rb:40:in `parse_file'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/puma-3.12.0/lib/puma/configuration.rb:318:in `load_rackup'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/puma-3.12.0/lib/puma/configuration.rb:243:in `app'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/puma-3.12.0/lib/puma/runner.rb:145:in `load_and_bind'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/puma-3.12.0/lib/puma/cluster.rb:410:in `run'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/puma-3.12.0/lib/puma/launcher.rb:184:in `run'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/puma-3.12.0/lib/puma/cli.rb:78:in `run'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/puma-3.12.0/bin/puma:10:in `<top (required)>'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/bin/puma:23:in `load'
Nov 13 21:00:32 ArchKaruta bundle[9417]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/bin/puma:23:in `<top (required)>'
Nov 13 21:00:32 ArchKaruta systemd[1]: mastodon-web.service: Main process exited, code=exited, status=1/FAILURE
Nov 13 21:00:32 ArchKaruta systemd[1]: mastodon-web.service: Failed with result 'exit-code'.
I decided to try executing the command from the systemd unit manually and checking that the files exist but I still receive the same errors. File ownership etc looks to all be in order.
Here is the output from manual execution:
[mastodon@ArchKaruta live]$ /home/mastodon/.rbenv/shims/bundle exec sidekiq -c 5 -q default -q push -q mailers -q pull
bundler: failed to load command: sidekiq (/home/mastodon/live/vendor/bundle/ruby/2.5.0/bin/sidekiq)
LoadError: libicudata.so.62: cannot open shared object file: No such file or directory - /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/charlock_holmes-0.7.6/lib/charlock_holmes/charlock_holmes.so
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/charlock_holmes-0.7.6/lib/charlock_holmes.rb:1:in `<top (required)>'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:71:in `require'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:71:in `block in require_with_bootsnap_lfi'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:70:in `require_with_bootsnap_lfi'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:79:in `require'
/home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:81:in `block (2 levels) in require'
/home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:76:in `each'
/home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:76:in `block in require'
/home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:65:in `each'
/home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler/runtime.rb:65:in `require'
/home/mastodon/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.17.1/lib/bundler.rb:114:in `require'
/home/mastodon/live/config/application.rb:7:in `<top (required)>'
/home/mastodon/live/config/environment.rb:2:in `require_relative'
/home/mastodon/live/config/environment.rb:2:in `<top (required)>'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/cli.rb:272:in `require'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/cli.rb:272:in `boot_system'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/cli.rb:53:in `run'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/bin/sidekiq:12:in `<top (required)>'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/bin/sidekiq:23:in `load'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/bin/sidekiq:23:in `<top (required)>'
Any ideas on what I can do from here?