Migration troubles [SOLVED]


#1

Hello people,

A few days ago I tried to migrate my whole instance to a brand new server. I strictly followed the Mastodon documentation related to migrations.
However, during the timeline rebuild phase, it just fails and give me this message:

RAILS_ENV=production bundle exec rails mastodon:feeds:build
rails aborted!
Don't know how to build task 'mastodon:feeds:build' (see --tasks)
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/railties-5.2.1/lib/rails/commands/rake/rake_command.rb:23:in `block in perform'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/railties-5.2.1/lib/rails/commands/rake/rake_command.rb:20:in `perform'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/railties-5.2.1/lib/rails/command.rb:48:in `invoke'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/railties-5.2.1/lib/rails/commands.rb:18:in `<top (required)>'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `block in require'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
/home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:287:in `require'
bin/rails:4:in `<main>'
(See full trace by running task with --trace)

I also checked the logs from mastodon-web and it shows something kinda suspicious(sorry had to include it via pastebin cause Discourse flags it as suspicious :confused: )

/var/log/nginx/instance.example.com-error.log is completely empty.
/var/log/nginx/instance.example.com-access.log shows this :

private_ip - - [15/Nov/2018:21:06:57 +0000] “POST /inbox HTTP/1.1” 401 100 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://mstdn.io/)”
private_ip - - [15/Nov/2018:21:07:07 +0000] “POST /inbox HTTP/1.1” 401 97 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://onster.farm/)”
private_ip - - [15/Nov/2018:21:07:10 +0000] “POST /inbox HTTP/1.1” 401 102 “-” “hackney/1.13.0”
private_ip - - [15/Nov/2018:21:07:21 +0000] “POST /inbox HTTP/1.1” 401 97 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://onster.farm/)”
private_ip - - [15/Nov/2018:21:07:24 +0000] “POST /inbox HTTP/1.1” 401 99 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://niu.moe/)”
private_ip - - [15/Nov/2018:21:07:30 +0000] “POST /inbox HTTP/1.1” 401 96 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://mamot.fr/)”
private_ip - - [15/Nov/2018:21:07:33 +0000] “POST /inbox HTTP/1.1” 401 97 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://onster.farm/)”
private_ip - - [15/Nov/2018:21:07:47 +0000] “POST /inbox HTTP/1.1” 401 97 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://onster.farm/)”
private_ip - - [15/Nov/2018:21:07:49 +0000] “POST /inbox HTTP/1.1” 401 94 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://niu.moe/)”
private_ip - - [15/Nov/2018:21:07:55 +0000] “POST /api/subscriptions/2309 HTTP/1.1” 200 5 “-” “GNU social/1.2.1-beta1 (Not decided yet)”
private_ip - - [15/Nov/2018:21:07:57 +0000] “POST /inbox HTTP/1.1” 401 96 “-” “hackney/1.13.0”
private_ip - - [15/Nov/2018:21:07:58 +0000] “POST /inbox HTTP/1.1” 401 97 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://onster.farm/)”
private_ip - - [15/Nov/2018:21:07:59 +0000] “POST /inbox HTTP/1.1” 401 101 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://framapiaf.org/)”
private_ip - - [15/Nov/2018:21:08:04 +0000] “POST /inbox HTTP/1.1” 401 103 “-” “http.rb/3.3.0 (Mastodon/2.6.1+glitch; +https://miaou.drycat.fr/)”
private_ip - - [15/Nov/2018:21:08:04 +0000] “POST /api/subscriptions/2309 HTTP/1.1” 200 5 “-” “GNU social/1.2.1-beta1 (Not decided yet)”
private_ip - - [15/Nov/2018:21:08:07 +0000] “POST /inbox HTTP/1.1” 401 96 “-” “hackney/1.13.0”
private_ip - - [15/Nov/2018:21:08:10 +0000] “POST /inbox HTTP/1.1” 401 97 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://onster.farm/)”
private_ip - - [15/Nov/2018:21:08:15 +0000] “POST /api/subscriptions/2309 HTTP/1.1” 200 5 “-” “GNU social/1.2.1-beta1 (Not decided yet)”
private_ip - - [15/Nov/2018:21:08:19 +0000] “POST /inbox HTTP/1.1” 401 96 “-” “hackney/1.13.0”
private_ip - - [15/Nov/2018:21:08:29 +0000] “POST /inbox HTTP/1.1” 401 106 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://mastodon.social/)”
private_ip - - [15/Nov/2018:21:08:42 +0000] “POST /inbox HTTP/1.1” 401 96 “-” “hackney/1.13.0”
private_ip - - [15/Nov/2018:21:08:45 +0000] “POST /inbox HTTP/1.1” 401 97 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://onster.farm/)”

As you can see(or guess), my instance runs behind a proxy and on a standard ubuntu lxc container :

                           lxd host                              lxc container
+-------------+           +----------------------------+        +------------------------------+
|             |           |                            |        |                              |
|  Internet   |           |  NGINX Proxy               |  NAT   | Mastodon Instance / No Docker|
|             +----------->                            +--------> Second NGINX Server          |
|             |           |                            |        |                              |
+-------------+           +----------------------------+        +------------------------------+
                           public ip                             private ip

Maybe I just fucked up my nginx config files so I’ve also included an overview of both of them :

Of course both my domain and my ip addresses have been redacted.
If you need anything else to troubleshoot this kind of stuff, just ask

Thanks for taking the time to help me :kissing_closed_eyes:


#2

If you are on version 2.6.0 and above please use RAILS_ENV=production bin/tootctl feeds build


#3

Okay ! I just did that and uncommented some stuff. I also restart my container and now I receive a blank page (but according to chromium it returns a 500 error code).

However journalctl mastodon-web still gives me this :

-5e2a-452e-839e-ce7a4e5189b7] ActionView::Template::Error (Webpacker can’t find about.js in /home/mastodon/live/public/packs/manifest.json. Possible causes:
nt to set webpacker.yml value of compile to true for your environment
you are using the webpack -w or the webpack-dev-server.
k has not yet re-run to reflect updates.
ve misconfigured Webpacker’s config/webpacker.yml file.
ebpack configuration is not creating a manifest.
fest contains:

-5e2a-452e-839e-ce7a4e5189b7] 4: - content_for :header_tags do
-5e2a-452e-839e-ce7a4e5189b7] 5: %link{ rel: ‘canonical’, href: about_url }/
-5e2a-452e-839e-ce7a4e5189b7] 6: %script#initial-state{ type: ‘application/json’ }!= json_escape(@initial_state_json)
-5e2a-452e-839e-ce7a4e5189b7] 7: = javascript_pack_tag ‘about’, integrity: true, crossorigin: ‘anonymous’
-5e2a-452e-839e-ce7a4e5189b7] 8: = render partial: ‘shared/og’
-5e2a-452e-839e-ce7a4e5189b7] 9:
-5e2a-452e-839e-ce7a4e5189b7] 10: .landing-page.alternative
-5e2a-452e-839e-ce7a4e5189b7]
-5e2a-452e-839e-ce7a4e5189b7] app/views/about/show.html.haml:7:in block in _app_views_about_show_html_haml___884544350327874517_70358011843760' -5e2a-452e-839e-ce7a4e5189b7] app/views/about/show.html.haml:4:in_app_views_about_show_html_haml___884544350327874517_70358011843760’
-68b9-4850-932e-748c3a445ea7] method=POST path=/inbox format=html controller=ActivityPub::InboxesController action=create status=202 duration=6.54 view=0.00 db=0.93

the access-log nginx file is full of these, which looks better :

10.79.175.1 - - [16/Nov/2018:21:00:42 +0000] “POST /inbox HTTP/1.1” 202 5 “-” “hackney/1.13.0”
10.79.175.1 - - [16/Nov/2018:21:00:44 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://freedom.horse/)”
10.79.175.1 - - [16/Nov/2018:21:00:57 +0000] “POST /inbox HTTP/1.1” 202 5 “-” “http.rb/3.3.0 (Mastodon/2.3.3; +https://cronk.stenoweb.net/)”
10.79.175.1 - - [16/Nov/2018:21:01:06 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://niu.moe/)”
10.79.175.1 - - [16/Nov/2018:21:01:10 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://beepboop.one/)”
10.79.175.1 - - [16/Nov/2018:21:01:12 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://beepboop.one/)”
10.79.175.1 - - [16/Nov/2018:21:01:16 +0000] “GET /api/v1/instance HTTP/1.1” 500 0 “-” “python-requests/2.19.1”
10.79.175.1 - - [16/Nov/2018:21:01:17 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://freedom.horse/)”
10.79.175.1 - - [16/Nov/2018:21:01:24 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.1+glitch; +https://toot-lab.reclaim.technology/)”
10.79.175.1 - - [16/Nov/2018:21:01:24 +0000] “GET /.well-known/webfinger?resource=acct:adidal@chat.cdstm.ch HTTP/1.1” 200 1079 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://icosahedron.website/)”
10.79.175.1 - - [16/Nov/2018:21:01:24 +0000] “GET /users/adidal HTTP/1.1” 200 1351 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://icosahedron.website/)”
10.79.175.1 - - [16/Nov/2018:21:01:24 +0000] “GET /users/adidal HTTP/1.1” 200 1351 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://icosahedron.website/)”
10.79.175.1 - - [16/Nov/2018:21:01:24 +0000] “GET /users/adidal/outbox HTTP/1.1” 200 439 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://icosahedron.website/)”
10.79.175.1 - - [16/Nov/2018:21:01:25 +0000] “GET /users/adidal/following HTTP/1.1” 200 416 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://icosahedron.website/)”
10.79.175.1 - - [16/Nov/2018:21:01:25 +0000] “GET /users/adidal/followers HTTP/1.1” 200 413 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://icosahedron.website/)”
10.79.175.1 - - [16/Nov/2018:21:01:25 +0000] “GET /users/adidal/collections/featured HTTP/1.1” 200 400 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://icosahedron.website/)”
10.79.175.1 - - [16/Nov/2018:21:01:54 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://freedom.horse/)”
10.79.175.1 - - [16/Nov/2018:21:02:03 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://mastodon.social/)”
10.79.175.1 - - [16/Nov/2018:21:02:12 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.1+glitch; +https://toot-lab.reclaim.technology/)”
10.79.175.1 - - [16/Nov/2018:21:02:12 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://freedom.horse/)”
10.79.175.1 - - [16/Nov/2018:21:02:43 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://freedom.horse/)”
10.79.175.1 - - [16/Nov/2018:21:02:46 +0000] “POST /inbox HTTP/1.1” 202 5 “-” “hackney/1.13.0”
10.79.175.1 - - [16/Nov/2018:21:02:58 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://mastodon.social/)”
10.79.175.1 - - [16/Nov/2018:21:03:04 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.2.0 (Mastodon/2.5.2-cybre; +https://cybre.space/)”
10.79.175.1 - - [16/Nov/2018:21:03:05 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://niu.moe/)”
10.79.175.1 - - [16/Nov/2018:21:03:14 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://mastodon.xyz/)”
10.79.175.1 - - [16/Nov/2018:21:03:20 +0000] “GET /api/v1/instance HTTP/1.1” 500 46 “-” “MastodonInstances (https://instances.social)”
10.79.175.1 - - [16/Nov/2018:21:03:40 +0000] “POST /inbox HTTP/1.1” 202 5 “-” “hackney/1.13.0”
10.79.175.1 - - [16/Nov/2018:21:03:42 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://mastodon.xyz/)”
10.79.175.1 - - [16/Nov/2018:21:03:51 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.2rc1; +https://friend.camp/)”
10.79.175.1 - - [16/Nov/2018:21:03:52 +0000] “GET /about HTTP/1.1” 500 0 “-” “Mozilla/5.0 (X11; Fedora; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36”
10.79.175.1 - - [16/Nov/2018:21:03:58 +0000] “GET / HTTP/1.1” 500 0 “-” “Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0”
10.79.175.1 - - [16/Nov/2018:21:03:59 +0000] “GET /sw.js HTTP/1.1” 404 36 “-” “Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0”
10.79.175.1 - - [16/Nov/2018:21:04:00 +0000] “GET /sw.js HTTP/1.1” 404 36 “-” “Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0”
10.79.175.1 - - [16/Nov/2018:21:04:01 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://social.oupsman.fr/)”
10.79.175.1 - - [16/Nov/2018:21:04:03 +0000] “POST /inbox HTTP/1.1” 202 5 “-” “hackney/1.13.0”
10.79.175.1 - - [16/Nov/2018:21:04:09 +0000] “GET / HTTP/1.1” 500 0 “-” “Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0”
10.79.175.1 - - [16/Nov/2018:21:04:09 +0000] “GET /favicon.ico HTTP/1.1” 200 9086 “-” “Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0”
10.79.175.1 - - [16/Nov/2018:21:04:10 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://niu.moe/)”
10.79.175.1 - - [16/Nov/2018:21:04:21 +0000] “POST /inbox HTTP/1.1” 202 5 “-” “hackney/1.13.0”
10.79.175.1 - - [16/Nov/2018:21:04:42 +0000] “POST /inbox HTTP/1.1” 202 5 “-” “hackney/1.13.0”
10.79.175.1 - - [16/Nov/2018:21:04:52 +0000] “POST /inbox HTTP/1.1” 202 36 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://mastodon.social/)”
10.79.175.1 - - [16/Nov/2018:21:04:54 +0000] “HEAD /users/jeannoelsd HTTP/1.1” 301 0 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://niu.moe/)”
10.79.175.1 - - [16/Nov/2018:21:04:55 +0000] “HEAD /@jeannoelsd HTTP/1.1” 500 0 “-” “http.rb/3.3.0 (Mastodon/2.6.1; +https://niu.moe/)”

Maybe I just fucked up my nginx configuration :confused:

EDIT :


Yeah it seems that there has been troubles during the installation. But no idea what went wrong. During the normal installation procedure I didn’t encounter any specific error :confused:


#4

Oookay this issue is solved ! I’ve found a github issue telling me to do a good old

RAILS_ENV=production bundle exec rails assets:precompile

I restarted and now it works like a charm ! Thanks @Gargron :smile:


#5

Quick followup question for @gargron then:
should that now be run using tootctl instead?


#6

If by “that” you mean compiling assets, then no. tootctl is just for mastodon-specific maintenance tasks, standard rails deployment remains unchanged.