Non-docker post-upgrade web service issue

Atlas · December 29, 2020, 10:02am

Environment: Ubuntu 18.04
Upgrading from v3.2.1 to v3.3.0, all release notes were followed, bundle install, yarn install and precompile went smoothly. Visiting site produces the error page. Checked log for mastodon-sidekiq, mastodon-streaming and mastodon-web while the only error seems to be with mastodon-web.

Error log as following:

-- Logs begin at Sat 2020-10-10 05:13:45 UTC. --
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]: TypeError: Cannot read property 'compress' of undefined
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Object.<anonymous> (/home/mastodon/live/config/webpack/development.js:31:35)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Module._compile (/home/mastodon/live/node_modules/v8-compile-cache/v8-compile-cache.js:192:30)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Module.load (internal/modules/cjs/loader.js:863:32)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Function.Module._load (internal/modules/cjs/loader.js:708:14)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Module.require (internal/modules/cjs/loader.js:887:19)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at require (/home/mastodon/live/node_modules/v8-compile-cache/v8-compile-cache.js:159:20)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at WEBPACK_OPTIONS (/home/mastodon/live/node_modules/webpack-cli/bin/utils/convert-argv.js:114:13)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at requireConfig (/home/mastodon/live/node_modules/webpack-cli/bin/utils/convert-argv.js:116:6)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at /home/mastodon/live/node_modules/webpack-cli/bin/utils/convert-argv.js:123:17
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Array.forEach (<anonymous>)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at module.exports (/home/mastodon/live/node_modules/webpack-cli/bin/utils/convert-argv.js:121:15)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at /home/mastodon/live/node_modules/webpack-cli/bin/cli.js:71:45
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Object.parse (/home/mastodon/live/node_modules/webpack-cli/node_modules/yargs/yargs.js:576:18)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at /home/mastodon/live/node_modules/webpack-cli/bin/cli.js:49:8
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Object.<anonymous> (/home/mastodon/live/node_modules/webpack-cli/bin/cli.js:366:3)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Module._compile (internal/modules/cjs/loader.js:999:30)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Module.load (internal/modules/cjs/loader.js:863:32)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Function.Module._load (internal/modules/cjs/loader.js:708:14)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Module.require (internal/modules/cjs/loader.js:887:19)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at require (internal/modules/cjs/helpers.js:74:18)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Object.<anonymous> (/home/mastodon/live/node_modules/webpack/bin/webpack.js:156:2)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Module._compile (internal/modules/cjs/loader.js:999:30)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Module.load (internal/modules/cjs/loader.js:863:32)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Function.Module._load (internal/modules/cjs/loader.js:708:14)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at internal/main/run_main_module.js:17:47
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]: [06f4d08b-d16c-4d0c-9431-9df608a71a02] [Webpacker] Compiling...
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: [06f4d08b-d16c-4d0c-9431-9df608a71a02] [Webpacker] Compilation failed:
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: /home/mastodon/live/node_modules/webpack-cli/bin/cli.js:93
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:                                 throw err;
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:                                 ^
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: TypeError: Cannot read property 'compress' of undefined
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Object.<anonymous> (/home/mastodon/live/config/webpack/development.js:31:35)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Module._compile (/home/mastodon/live/node_modules/v8-compile-cache/v8-compile-cache.js:192:30)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Module.load (internal/modules/cjs/loader.js:863:32)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Function.Module._load (internal/modules/cjs/loader.js:708:14)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Module.require (internal/modules/cjs/loader.js:887:19)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at require (/home/mastodon/live/node_modules/v8-compile-cache/v8-compile-cache.js:159:20)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at WEBPACK_OPTIONS (/home/mastodon/live/node_modules/webpack-cli/bin/utils/convert-argv.js:114:13)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at requireConfig (/home/mastodon/live/node_modules/webpack-cli/bin/utils/convert-argv.js:116:6)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at /home/mastodon/live/node_modules/webpack-cli/bin/utils/convert-argv.js:123:17
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Array.forEach (<anonymous>)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at module.exports (/home/mastodon/live/node_modules/webpack-cli/bin/utils/convert-argv.js:121:15)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at /home/mastodon/live/node_modules/webpack-cli/bin/cli.js:71:45
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Object.parse (/home/mastodon/live/node_modules/webpack-cli/node_modules/yargs/yargs.js:576:18)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at /home/mastodon/live/node_modules/webpack-cli/bin/cli.js:49:8
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Object.<anonymous> (/home/mastodon/live/node_modules/webpack-cli/bin/cli.js:366:3)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Module._compile (internal/modules/cjs/loader.js:999:30)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Module.load (internal/modules/cjs/loader.js:863:32)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Function.Module._load (internal/modules/cjs/loader.js:708:14)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Module.require (internal/modules/cjs/loader.js:887:19)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at require (internal/modules/cjs/helpers.js:74:18)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Object.<anonymous> (/home/mastodon/live/node_modules/webpack/bin/webpack.js:156:2)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Module._compile (internal/modules/cjs/loader.js:999:30)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Module.load (internal/modules/cjs/loader.js:863:32)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Function.Module._load (internal/modules/cjs/loader.js:708:14)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at internal/main/run_main_module.js:17:47
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: [06f4d08b-d16c-4d0c-9431-9df608a71a02] method=GET path=/poco format=*/* controller=ApplicationController action=raise_not_found status=500 error='ActionView::Template::Error: Webpacker can't find cat-light.css in /home/mastodon/live/public/packs/manifest.json. Possible causes:
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: 1. You want to set webpacker.yml value of compile to true for your environment
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:    unless you are using the `webpack -w` or the webpack-dev-server.
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: 2. webpack has not yet re-run to reflect updates.
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: 3. You have misconfigured Webpacker's config/webpacker.yml file.
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: 4. Your webpack configuration is not creating a manifest.
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: Your manifest contains:
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: {
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:   "about.js": {
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     "src": "/packs/js/about-3d75891be6f7315f8e7c.chunk.js",
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     "integrity": "sha256-4Y/iU+r0IE6j9uxurgQs6+/3Ub4/QheAOX1br68gFWc="
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:   },
... thousands of lines about manifest error log...

Tried git reset --hard tootsuite-mastodon/master to pull the official repo and also rm -rf tmp public/packs then re-precompiled for multiple times, the issue still remains.
Ruby and rails version checking:

mastodon@mastodon-ubuntu-euforia : ~/live $ ruby -v
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]

mastodon@mastodon-ubuntu-euforia : ~/live $ RAILS_ENV=production rails -v
Rails 5.2.4.4

Seems to be a webpacker issue. Referred to a few threads online for other projects, it however seems to be beyond my ability. Any help will be appreciated.

saper · December 29, 2020, 1:54pm

Can you check if you have this line in config/webpacker.yml:

github.com

tootsuite/mastodon/blob/v3.3.0/config/webpacker.yml#L65


  # Reference: https://webpack.js.org/configuration/dev-server/
  dev_server:
    https: false
    host: localhost
    port: 3035
    public: localhost:3035
    hmr: false
    # Inline should be set to true if using HMR
    inline: true
    overlay: true
    compress: true
    disable_host_check: true
    use_local_ip: false
    quiet: false
    headers:
      'Access-Control-Allow-Origin': '*'
    watch_options:
      ignored: '**/node_modules/**'
test:
  <<: *default

saper · December 29, 2020, 2:01pm

Also, are you running in production or development mode?

Atlas · December 29, 2020, 2:29pm

It’s running in production mode since I’m copying & pasting from release note RAILS_ENV=production bundle exec rails assets:precompile.
Tricky thing is that I’ve checked about config/webpacker.yml which I do have and this file goes:

# Note: You must restart bin/webpack-dev-server for changes to take effect

default: &default
  source_path: app/javascript
  source_entry_path: packs
  public_root_path: public
  public_output_path: packs
  cache_path: tmp/cache/webpacker
  check_yarn_integrity: false
  webpack_compile_output: false

  # Additional paths webpack should lookup modules
  # ['app/assets', 'engine/foo/app/assets']
  resolved_paths: []

  # Reload manifest.json on all requests so we reload latest compiled packs
  cache_manifest: false

  # Extract and emit a css file
  extract_css: true

  static_assets_extensions:
    - .jpg
    - .jpeg
    - .png
    - .tiff
    - .ico
    - .svg
    - .eot
    - .otf
    - .ttf
    - .woff
    - .woff2

  extensions:
    - .mjs
    - .js
    - .sass
    - .scss
    - .css
    - .module.sass
    - .module.scss
    - .module.css
    - .png
    - .svg
    - .gif
    - .jpeg
    - .jpg

development:
  <<: *default

  compile: true

  # Reference: https://webpack.js.org/configuration/dev-server/
  dev_server:
    https: false
    host: localhost
    port: 3035
    public: localhost:3035
    hmr: false
    # Inline should be set to true if using HMR
    inline: true
    overlay: true
    compress: true
    disable_host_check: true
    use_local_ip: false
    quiet: false
    headers:
      'Access-Control-Allow-Origin': '*'
    watch_options:
      ignored: '**/node_modules/**'

test:
  <<: *default

  # CircleCI precompiles packs prior to running the tests.
  # Also avoids race conditions in parallel_tests.
  compile: false

  # Compile test packs to a separate directory
  public_output_path: packs-test

production:
  <<: *default

  # Production depends on precompilation of packs prior to booting for performance.
  compile: false

  # Cache manifest.json for performance
  cache_manifest: true

saper · December 29, 2020, 2:32pm

and when you run it as a service? you have .env* files, is it all production there as well?

Atlas · December 29, 2020, 2:39pm

Can you be more explicit about this? Which .env* files should I be checking?

saper · December 29, 2020, 2:40pm

You configuration files where you have your passwords etc. set (obviously don’t publish them here…)

Atlas · December 29, 2020, 2:53pm

I’m still unsure which files to look at but I tried cat .env* at /live and searched for keyword “production”, which did output few lines stating out that it is under production: DB_NAME=mastodon_production

saper · December 29, 2020, 4:05pm

I have a question here - you say asset precompile completed ok, can you check if it generates the file in question?

msh · December 29, 2020, 5:08pm

I’m not getting status=500 (I am getting status 404 in the log though the web client is sent a 200 plus the not-found page lol), so it doesn’t produce a useful stack trace, it it could be a similar problem since I had to make a couple of attempts to install prereqs and get the asset precompile stage to work. I don’t recall seeing errors but there were warnings.

saper · December 29, 2020, 5:11pm

I don’t have any 3.2.0 to upgrade to 3.3.0 or something like that but I’d try finding leftover files with git status and/or git clean -dxn. Otherwise, out of ideas (and knowledge about webpack really).

msh · December 29, 2020, 7:08pm

I cleaned out the files (preserving .env.production and public/system of course), reinstalled all dependencies, did the asset precompile again and cleared cache. Unfortunately I got the same result (a mostly functional instance that is usable through a mobile app but still with the missing /poco). This time the build was uneventful except for one warning message:

warning " > react-redux-loading-bar@4.0.8" has incorrect peer dependency "react-redux@^3.0.0 || ^4.0.0 || ^5.0.0".

This occured twice: once during “yarn install” of the prereqs and the other during asset precompilie. In both occurrences this warning came up during the “linking dependencies” stage. Since there were no other warnings or errors during the process I will pursue the cause of this one and hope I come up with something.

saper · December 29, 2020, 7:31pm

This one looks harmless to me and got fixed in

I have a feeling it’s something something webpack

msh · December 29, 2020, 9:07pm

It does indeed look unrelated. I am also very inexperienced with webpack. The frustrating thing us that poco files do indeed seem to be present after the yarn install and asset precompile, so I don’t think that is the problem, just that the app cannot find it.

If ANYONE knows how to troubleshoot this it would be great!

From: joinmastodon@discoursemail.com
Sent: December 29, 2020 12:41 PM
To: mark@haydensplace.com
Reply-to: joinmastodon+b3f9608a3f790ba9ec627825383845be@discoursemail.com
Subject: [Mastodon Meta Discussion Board] [Serveradministration/Troubleshooting] Non-docker post-upgrade web service issue

saper Instance Admin
December 29

This one looks harmless to me and got fixed in

github.com/mironov/react-redux-loading-bar

Update react-redux peer dependency requirement`mironov:master` ← `hugomn:master`

opened 01:48PM - 16 Jul 19 UTC
hugomn
+1 -1

I have a feeling it’s something something webpack

saper · December 29, 2020, 9:08pm

Can you check if the development webpack server is running in parallel by chance or something?

Atlas · December 30, 2020, 1:25am

I think it does

mastodon@mastodon-ubuntu-euforia:~/live$ cd public/packs
mastodon@mastodon-ubuntu-euforia:~/live/public/packs$ ls
css  js  manifest.json  manifest.json.gz  media  ocr  report.html

Atlas · December 30, 2020, 2:15am

Though I think I might know what this problem is about. I did add two custom themes with name cat-light and cat and when I switched to master/test branch and git reset --hard to official repo, the scss files added under app/javascript/styles were gone. Yet I checked the precompiled public/packs/css and public/pack/js, there are files with name cat-*.chunk.* and cat-light-*.chunk.*. Not knowing the cause yet.

~~Edit: Also checked other files modified, including config/themes.yml and config/locales/*.yml. All modifications were erased in current branch.~~

Edit: Re-precompiled, said cat-*.chunk.* and cat-light-*.chunk.* are gone. Restarted services, site still not working. And the journal log still contains following lines:

-- Logs begin at Sat 2020-10-10 05:13:45 UTC. --
Dec 30 02:09:15 mastodon-ubuntu-euforia bundle[28440]: [007a7cf7-6bbe-4b19-9ddd-a706071d8b59] method=GET path=/ format=*/\* controller=HomeController action=index status=500 error='ActionView::Template::Error: Webpacker can't find cat-light.css in /home/mastodon/live/public/packs/manifest.json. Possible causes:
Dec 30 02:09:15 mastodon-ubuntu-euforia bundle[28440]: 1. You want to set webpacker.yml value of compile to true for your environment
Dec 30 02:09:15 mastodon-ubuntu-euforia bundle[28440]:    unless you are using the `webpack -w` or the webpack-dev-server.
Dec 30 02:09:15 mastodon-ubuntu-euforia bundle[28440]: 2. webpack has not yet re-run to reflect updates.
Dec 30 02:09:15 mastodon-ubuntu-euforia bundle[28440]: 3. You have misconfigured Webpacker's config/webpacker.yml file.
Dec 30 02:09:15 mastodon-ubuntu-euforia bundle[28440]: 4. Your webpack configuration is not creating a manifest.

saper · December 30, 2020, 3:16pm

Ok, so your customizations are somewhere in the way. Tried git clean -dxn and then - after carefully examining the result git clean -dxf ? Warning: It will remove important files if not careful. Backup first!

Atlas · December 31, 2020, 8:54am

Tried git clean -dxn but didn’t see anything fishy.

Would remove .bundle/
Would remove .env.production
Would remove build/
Would remove node_modules/
Would remove public/assets/
Would remove public/system/
Would skip repository vendor/bundle/ruby/2.7.0/bundler/gems/nilsimsa-fd184883048b
Would skip repository vendor/bundle/ruby/2.7.0/bundler/gems/health_check-0b799ead604f
Would remove vendor/bundle/ruby/2.7.0/bundler/gems/extensions
Would remove vendor/bundle/ruby/2.7.0/bin
Would remove vendor/bundle/ruby/2.7.0/doc
Would remove vendor/bundle/ruby/2.7.0/gems
Would remove vendor/bundle/ruby/2.7.0/specifications
Would remove vendor/bundle/ruby/2.7.0/cache
Would remove vendor/bundle/ruby/2.7.0/extensions
Would remove vendor/bundle/ruby/2.7.0/build_info
Would skip repository vendor/bundle/ruby/2.6.0/bundler/gems/http_parser.rb-54b17ba8c7d8
Would skip repository vendor/bundle/ruby/2.6.0/bundler/gems/nilsimsa-fd184883048b
Would skip repository vendor/bundle/ruby/2.6.0/bundler/gems/posix-spawn-58465d2e2139
Would skip repository vendor/bundle/ruby/2.6.0/bundler/gems/health_check-0b799ead604f
Would remove vendor/bundle/ruby/2.6.0/bundler/gems/extensions
Would remove vendor/bundle/ruby/2.6.0/bin
Would remove vendor/bundle/ruby/2.6.0/doc
Would remove vendor/bundle/ruby/2.6.0/gems
Would remove vendor/bundle/ruby/2.6.0/specifications
Would remove vendor/bundle/ruby/2.6.0/cache
Would remove vendor/bundle/ruby/2.6.0/extensions
Would remove vendor/bundle/ruby/2.6.0/build_info

I also tried stopping all services, deleting nginx cache and starting all services again as they did in “File /poco does not exist.” on web client after 3.3.0 upgrade which didn’t work for me unfortunately.

Edit: I figure it could be a similar cache issue as they had, yet since I couldn’t locate whereas, I reversely added the custom cat.scss and cat-light.scss files and configs back, now the site is working properly.
For anyone who might be facing similar situation in the future: The cause of this issue is still unclear but should be able to be solved by doing so.

system · January 14, 2021, 8:09am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Non-docker post-upgrade web service issue

Update react-redux peer dependency requirementmironov:master ← hugomn:master

Update react-redux peer dependency requirement`mironov:master` ← `hugomn:master`