Non-docker post-upgrade web service issue

Environment: Ubuntu 18.04
Upgrading from v3.2.1 to v3.3.0, all release notes were followed, bundle install, yarn install and precompile went smoothly. Visiting site produces the error page. Checked log for mastodon-sidekiq, mastodon-streaming and mastodon-web while the only error seems to be with mastodon-web.

Error log as following:

-- Logs begin at Sat 2020-10-10 05:13:45 UTC. --
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]: TypeError: Cannot read property 'compress' of undefined
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Object.<anonymous> (/home/mastodon/live/config/webpack/development.js:31:35)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Module._compile (/home/mastodon/live/node_modules/v8-compile-cache/v8-compile-cache.js:192:30)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Module.load (internal/modules/cjs/loader.js:863:32)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Function.Module._load (internal/modules/cjs/loader.js:708:14)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Module.require (internal/modules/cjs/loader.js:887:19)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at require (/home/mastodon/live/node_modules/v8-compile-cache/v8-compile-cache.js:159:20)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at WEBPACK_OPTIONS (/home/mastodon/live/node_modules/webpack-cli/bin/utils/convert-argv.js:114:13)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at requireConfig (/home/mastodon/live/node_modules/webpack-cli/bin/utils/convert-argv.js:116:6)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at /home/mastodon/live/node_modules/webpack-cli/bin/utils/convert-argv.js:123:17
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Array.forEach (<anonymous>)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at module.exports (/home/mastodon/live/node_modules/webpack-cli/bin/utils/convert-argv.js:121:15)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at /home/mastodon/live/node_modules/webpack-cli/bin/cli.js:71:45
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Object.parse (/home/mastodon/live/node_modules/webpack-cli/node_modules/yargs/yargs.js:576:18)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at /home/mastodon/live/node_modules/webpack-cli/bin/cli.js:49:8
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Object.<anonymous> (/home/mastodon/live/node_modules/webpack-cli/bin/cli.js:366:3)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Module._compile (internal/modules/cjs/loader.js:999:30)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Module.load (internal/modules/cjs/loader.js:863:32)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Function.Module._load (internal/modules/cjs/loader.js:708:14)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Module.require (internal/modules/cjs/loader.js:887:19)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at require (internal/modules/cjs/helpers.js:74:18)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Object.<anonymous> (/home/mastodon/live/node_modules/webpack/bin/webpack.js:156:2)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Module._compile (internal/modules/cjs/loader.js:999:30)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Module.load (internal/modules/cjs/loader.js:863:32)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Function.Module._load (internal/modules/cjs/loader.js:708:14)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12)
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]:     at internal/main/run_main_module.js:17:47
Dec 29 08:06:51 mastodon-ubuntu-euforia bundle[14538]: [06f4d08b-d16c-4d0c-9431-9df608a71a02] [Webpacker] Compiling...
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: [06f4d08b-d16c-4d0c-9431-9df608a71a02] [Webpacker] Compilation failed:
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: /home/mastodon/live/node_modules/webpack-cli/bin/cli.js:93
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:                                 throw err;
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:                                 ^
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: TypeError: Cannot read property 'compress' of undefined
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Object.<anonymous> (/home/mastodon/live/config/webpack/development.js:31:35)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Module._compile (/home/mastodon/live/node_modules/v8-compile-cache/v8-compile-cache.js:192:30)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Module.load (internal/modules/cjs/loader.js:863:32)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Function.Module._load (internal/modules/cjs/loader.js:708:14)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Module.require (internal/modules/cjs/loader.js:887:19)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at require (/home/mastodon/live/node_modules/v8-compile-cache/v8-compile-cache.js:159:20)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at WEBPACK_OPTIONS (/home/mastodon/live/node_modules/webpack-cli/bin/utils/convert-argv.js:114:13)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at requireConfig (/home/mastodon/live/node_modules/webpack-cli/bin/utils/convert-argv.js:116:6)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at /home/mastodon/live/node_modules/webpack-cli/bin/utils/convert-argv.js:123:17
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Array.forEach (<anonymous>)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at module.exports (/home/mastodon/live/node_modules/webpack-cli/bin/utils/convert-argv.js:121:15)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at /home/mastodon/live/node_modules/webpack-cli/bin/cli.js:71:45
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Object.parse (/home/mastodon/live/node_modules/webpack-cli/node_modules/yargs/yargs.js:576:18)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at /home/mastodon/live/node_modules/webpack-cli/bin/cli.js:49:8
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Object.<anonymous> (/home/mastodon/live/node_modules/webpack-cli/bin/cli.js:366:3)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Module._compile (internal/modules/cjs/loader.js:999:30)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Module.load (internal/modules/cjs/loader.js:863:32)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Function.Module._load (internal/modules/cjs/loader.js:708:14)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Module.require (internal/modules/cjs/loader.js:887:19)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at require (internal/modules/cjs/helpers.js:74:18)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Object.<anonymous> (/home/mastodon/live/node_modules/webpack/bin/webpack.js:156:2)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Module._compile (internal/modules/cjs/loader.js:999:30)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Module.load (internal/modules/cjs/loader.js:863:32)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Function.Module._load (internal/modules/cjs/loader.js:708:14)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12)
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     at internal/main/run_main_module.js:17:47
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: [06f4d08b-d16c-4d0c-9431-9df608a71a02] method=GET path=/poco format=*/* controller=ApplicationController action=raise_not_found status=500 error='ActionView::Template::Error: Webpacker can't find cat-light.css in /home/mastodon/live/public/packs/manifest.json. Possible causes:
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: 1. You want to set webpacker.yml value of compile to true for your environment
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:    unless you are using the `webpack -w` or the webpack-dev-server.
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: 2. webpack has not yet re-run to reflect updates.
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: 3. You have misconfigured Webpacker's config/webpacker.yml file.
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: 4. Your webpack configuration is not creating a manifest.
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: Your manifest contains:
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]: {
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:   "about.js": {
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     "src": "/packs/js/about-3d75891be6f7315f8e7c.chunk.js",
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:     "integrity": "sha256-4Y/iU+r0IE6j9uxurgQs6+/3Ub4/QheAOX1br68gFWc="
Dec 29 08:06:54 mastodon-ubuntu-euforia bundle[14538]:   },
... thousands of lines about manifest error log...

Tried git reset --hard tootsuite-mastodon/master to pull the official repo and also rm -rf tmp public/packs then re-precompiled for multiple times, the issue still remains.
Ruby and rails version checking:

mastodon@mastodon-ubuntu-euforia : ~/live $ ruby -v
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]

mastodon@mastodon-ubuntu-euforia : ~/live $ RAILS_ENV=production rails -v
Rails 5.2.4.4

Seems to be a webpacker issue. Referred to a few threads online for other projects, it however seems to be beyond my ability. Any help will be appreciated.

1 Like

Can you check if you have this line in config/webpacker.yml:

Also, are you running in production or development mode?

It’s running in production mode since I’m copying & pasting from release note RAILS_ENV=production bundle exec rails assets:precompile.
Tricky thing is that I’ve checked about config/webpacker.yml which I do have and this file goes:

# Note: You must restart bin/webpack-dev-server for changes to take effect

default: &default
  source_path: app/javascript
  source_entry_path: packs
  public_root_path: public
  public_output_path: packs
  cache_path: tmp/cache/webpacker
  check_yarn_integrity: false
  webpack_compile_output: false

  # Additional paths webpack should lookup modules
  # ['app/assets', 'engine/foo/app/assets']
  resolved_paths: []

  # Reload manifest.json on all requests so we reload latest compiled packs
  cache_manifest: false

  # Extract and emit a css file
  extract_css: true

  static_assets_extensions:
    - .jpg
    - .jpeg
    - .png
    - .tiff
    - .ico
    - .svg
    - .eot
    - .otf
    - .ttf
    - .woff
    - .woff2

  extensions:
    - .mjs
    - .js
    - .sass
    - .scss
    - .css
    - .module.sass
    - .module.scss
    - .module.css
    - .png
    - .svg
    - .gif
    - .jpeg
    - .jpg

development:
  <<: *default

  compile: true

  # Reference: https://webpack.js.org/configuration/dev-server/
  dev_server:
    https: false
    host: localhost
    port: 3035
    public: localhost:3035
    hmr: false
    # Inline should be set to true if using HMR
    inline: true
    overlay: true
    compress: true
    disable_host_check: true
    use_local_ip: false
    quiet: false
    headers:
      'Access-Control-Allow-Origin': '*'
    watch_options:
      ignored: '**/node_modules/**'

test:
  <<: *default

  # CircleCI precompiles packs prior to running the tests.
  # Also avoids race conditions in parallel_tests.
  compile: false

  # Compile test packs to a separate directory
  public_output_path: packs-test

production:
  <<: *default

  # Production depends on precompilation of packs prior to booting for performance.
  compile: false

  # Cache manifest.json for performance
  cache_manifest: true

and when you run it as a service? you have .env* files, is it all production there as well?

Can you be more explicit about this? Which .env* files should I be checking?

You configuration files where you have your passwords etc. set (obviously don’t publish them here…)

I’m still unsure which files to look at but I tried cat .env* at /live and searched for keyword ā€œproductionā€, which did output few lines stating out that it is under production: DB_NAME=mastodon_production

I have a question here - you say asset precompile completed ok, can you check if it generates the file in question?

I’m not getting status=500 (I am getting status 404 in the log though the web client is sent a 200 plus the not-found page lol), so it doesn’t produce a useful stack trace, it it could be a similar problem since I had to make a couple of attempts to install prereqs and get the asset precompile stage to work. I don’t recall seeing errors but there were warnings.

1 Like

I don’t have any 3.2.0 to upgrade to 3.3.0 or something like that but I’d try finding leftover files with git status and/or git clean -dxn. Otherwise, out of ideas (and knowledge about webpack really).

I cleaned out the files (preserving .env.production and public/system of course), reinstalled all dependencies, did the asset precompile again and cleared cache. Unfortunately I got the same result (a mostly functional instance that is usable through a mobile app but still with the missing /poco). This time the build was uneventful except for one warning message:

warning " > react-redux-loading-bar@4.0.8" has incorrect peer dependency "react-redux@^3.0.0 || ^4.0.0 || ^5.0.0".

This occured twice: once during ā€œyarn installā€ of the prereqs and the other during asset precompilie. In both occurrences this warning came up during the ā€œlinking dependenciesā€ stage. Since there were no other warnings or errors during the process I will pursue the cause of this one and hope I come up with something.

This one looks harmless to me and got fixed in

I have a feeling it’s something something webpack

It does indeed look unrelated. I am also very inexperienced with webpack. The frustrating thing us that poco files do indeed seem to be present after the yarn install and asset precompile, so I don’t think that is the problem, just that the app cannot find it.

If ANYONE knows how to troubleshoot this it would be great!



From: joinmastodon@discoursemail.com
Sent: December 29, 2020 12:41 PM
To: mark@haydensplace.com
Reply-to: joinmastodon+b3f9608a3f790ba9ec627825383845be@discoursemail.com
Subject: [Mastodon Meta Discussion Board] [Serveradministration/Troubleshooting] Non-docker post-upgrade web service issue

saper Instance Admin
December 29

This one looks harmless to me and got fixed in

github.com/mironov/react-redux-loading-bar

Update react-redux peer dependency requirementmironov:master ← hugomn:master

opened 01:48PM - 16 Jul 19 UTC
hugomnhugomn
+1 -1

I have a feeling it’s something something webpack

Can you check if the development webpack server is running in parallel by chance or something?

I think it does

mastodon@mastodon-ubuntu-euforia:~/live$ cd public/packs
mastodon@mastodon-ubuntu-euforia:~/live/public/packs$ ls
css  js  manifest.json  manifest.json.gz  media  ocr  report.html

Though I think I might know what this problem is about. I did add two custom themes with name cat-light and cat and when I switched to master/test branch and git reset --hard to official repo, the scss files added under app/javascript/styles were gone. Yet I checked the precompiled public/packs/css and public/pack/js, there are files with name cat-*.chunk.* and cat-light-*.chunk.*. Not knowing the cause yet.

Edit: Also checked other files modified, including config/themes.yml and config/locales/*.yml. All modifications were erased in current branch.

Edit: Re-precompiled, said cat-*.chunk.* and cat-light-*.chunk.* are gone. Restarted services, site still not working. And the journal log still contains following lines:

-- Logs begin at Sat 2020-10-10 05:13:45 UTC. --
Dec 30 02:09:15 mastodon-ubuntu-euforia bundle[28440]: [007a7cf7-6bbe-4b19-9ddd-a706071d8b59] method=GET path=/ format=*/\* controller=HomeController action=index status=500 error='ActionView::Template::Error: Webpacker can't find cat-light.css in /home/mastodon/live/public/packs/manifest.json. Possible causes:
Dec 30 02:09:15 mastodon-ubuntu-euforia bundle[28440]: 1. You want to set webpacker.yml value of compile to true for your environment
Dec 30 02:09:15 mastodon-ubuntu-euforia bundle[28440]:    unless you are using the `webpack -w` or the webpack-dev-server.
Dec 30 02:09:15 mastodon-ubuntu-euforia bundle[28440]: 2. webpack has not yet re-run to reflect updates.
Dec 30 02:09:15 mastodon-ubuntu-euforia bundle[28440]: 3. You have misconfigured Webpacker's config/webpacker.yml file.
Dec 30 02:09:15 mastodon-ubuntu-euforia bundle[28440]: 4. Your webpack configuration is not creating a manifest.

Ok, so your customizations are somewhere in the way. Tried git clean -dxn and then - after carefully examining the result git clean -dxf ? Warning: It will remove important files if not careful. Backup first!

Tried git clean -dxn but didn’t see anything fishy.

Would remove .bundle/
Would remove .env.production
Would remove build/
Would remove node_modules/
Would remove public/assets/
Would remove public/system/
Would skip repository vendor/bundle/ruby/2.7.0/bundler/gems/nilsimsa-fd184883048b
Would skip repository vendor/bundle/ruby/2.7.0/bundler/gems/health_check-0b799ead604f
Would remove vendor/bundle/ruby/2.7.0/bundler/gems/extensions
Would remove vendor/bundle/ruby/2.7.0/bin
Would remove vendor/bundle/ruby/2.7.0/doc
Would remove vendor/bundle/ruby/2.7.0/gems
Would remove vendor/bundle/ruby/2.7.0/specifications
Would remove vendor/bundle/ruby/2.7.0/cache
Would remove vendor/bundle/ruby/2.7.0/extensions
Would remove vendor/bundle/ruby/2.7.0/build_info
Would skip repository vendor/bundle/ruby/2.6.0/bundler/gems/http_parser.rb-54b17ba8c7d8
Would skip repository vendor/bundle/ruby/2.6.0/bundler/gems/nilsimsa-fd184883048b
Would skip repository vendor/bundle/ruby/2.6.0/bundler/gems/posix-spawn-58465d2e2139
Would skip repository vendor/bundle/ruby/2.6.0/bundler/gems/health_check-0b799ead604f
Would remove vendor/bundle/ruby/2.6.0/bundler/gems/extensions
Would remove vendor/bundle/ruby/2.6.0/bin
Would remove vendor/bundle/ruby/2.6.0/doc
Would remove vendor/bundle/ruby/2.6.0/gems
Would remove vendor/bundle/ruby/2.6.0/specifications
Would remove vendor/bundle/ruby/2.6.0/cache
Would remove vendor/bundle/ruby/2.6.0/extensions
Would remove vendor/bundle/ruby/2.6.0/build_info

I also tried stopping all services, deleting nginx cache and starting all services again as they did in ā€œFile /poco does not exist.ā€ on web client after 3.3.0 upgrade which didn’t work for me unfortunately.

Edit: I figure it could be a similar cache issue as they had, yet since I couldn’t locate whereas, I reversely added the custom cat.scss and cat-light.scss files and configs back, now the site is working properly.
For anyone who might be facing similar situation in the future: The cause of this issue is still unclear but should be able to be solved by doing so.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.