"File /poco does not exist." on web client after 3.3.0 upgrade

Hi everyone,

After upgrading my instance to v3.3.0 I cannot use the web front end. Everything at https:///web/* comes up with an error “File /poco does not exist” followed by a directory listing of /home/mastodon/live on my server (thankfully the hyperlinks all return 404 instead of the actual file contents!). The page is entitled “404” but it actually is returning a 200 code and the HTML source for the message and directory.

It appears the api, sidekiq, and the ActiityPub endpoints are all working fine (in the logs I see statuses being POSTed and other instances GETing them etc), plus I can go into https:///admin as well. I don’t understand what happened here except to suspect that it is something related to node.js rather than Ruby?

Does anyone know what may be going on here? how would I fix this?

can you show/check your nginx configuration?

Certainly. It is identical to the example config file at /home/mastodon/live/dist/nginx.conf except for the domain being replaced with my instance I should reiterate that the API, streaming, admin page, public profiles etc. all work. I even still get notifications from my web.browser of faves and boosts, and interacting through Tusky is working fine. It is only the web front end client (urls that are /web/…) that don’t work.

The file has not been altered during the upgrade; it was working prior to 3.3:

map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

upstream backend {
    server 127.0.0.1:3000 fail_timeout=0;
}

upstream streaming {
    server 127.0.0.1:4000 fail_timeout=0;
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=CACHE:10m inactive=7d     max_size=1g;

server {

  listen 80;
  listen [::]:80;
  server_name coales.co;
  root /home/mastodon/live/public;
  location /.well-known/acme-challenge/ { allow all; }
  location / { return 301 https://$host$request_uri; }


}

server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name coales.co;

  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;

  ssl_certificate     /etc/letsencrypt/live/coales.co/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/coales.co/privkey.pem;

  keepalive_timeout    70;
  sendfile             on;
  client_max_body_size 80m;

  root /home/mastodon/live/public;

  gzip on;
  gzip_disable "msie6";
  gzip_vary on;
  gzip_proxied any;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json application/javascript text/xml application/xml     application/xml+rss text/javascript;

  add_header Strict-Transport-Security "max-age=31536000";

  location / {
    try_files $uri @proxy;
  }

  location ~ ^/(emoji|packs|system/accounts/avatars|system/media_attachments/files) {
    add_header Cache-Control "public, max-age=31536000, immutable";
    add_header Strict-Transport-Security "max-age=31536000";
    try_files $uri @proxy;
  }

  location /sw.js {
    add_header Cache-Control "public, max-age=0";
    add_header Strict-Transport-Security "max-age=31536000";
    try_files $uri @proxy;
  }

  location @proxy {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header Proxy "";
    proxy_pass_header Server;

    proxy_pass http://backend;
    proxy_buffering on;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    proxy_cache CACHE;
    proxy_cache_valid 200 7d;
    proxy_cache_valid 410 24h;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    add_header X-Cached $upstream_cache_status;
    add_header Strict-Transport-Security "max-age=31536000";

    tcp_nodelay on;
  }

  location /api/v1/streaming {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header Proxy "";

    proxy_pass http://streaming;
    proxy_buffering off;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    tcp_nodelay on;
  }

  error_page 500 501 502 503 504 /500.html;

}

Very much stock config here. Perhaps it is permissions or something? What else can I check.

Yes, looks pretty much fine to me. Did you rebuild the frontend during the upgrade?

As in assets:precompile? yes, I did go through that step as well. However it may be where the issue lies as I sort of fumbled my way through upgrading Ruby and installing prerequisites (seems it did something weird with yarn?). I actually did that step twice, but since the second time went way faster I am wondering if maybe there is some bundle/yarn incantations I can do to approximate a “make clean” function properly (vs backing up config and going in like a bull in a china shop with rm -Rf)?

You can check what git clean -dxn lists for you… those should be the files generated by the installations process. Remove the frontend ones and try again.

@msh Do you something similar to Non-docker post-upgrade web service issue - #9 by saper in your logs? maybe not all assets were created (although webpack seems to complete successfully).

I have no idea what curse has befallen my system. I preserved my .env.production and /home/mastodon/live/public/system then completely blew away the contents of /home/mastodon. I upgraded nodejs, and re-built Ruby, Mastodon and pre-reqs from scratch before restoring my .envproduction and public/system directory.

THE ERROR IS STILL HAPPENING!

this strongly suggests there is an issue with the environment in which I am trying to install Mastodon perhaps? I have NO idea where to start looking for that

In any case, I cannot find any file or directory containing “poco” in /home/mastodon/live after this clean rebuild, so for whatever reason it did not get pulled in and built. On a functional 3.3.0 instance where is this supposed to be put? Is there a way I can pull it in so my web front end will work again?

Did you reboot the machine completely? Maybe some webpack dev server is running there on port 3000

I did reboot in fact. I actually ended up getting in touch with Eugen on mastodon and in the process we figured out the issue!

Turns out there was something corrupt in the nginx cache (which is kept outside the mastodon Directory and also survives reboots)! I stopped all services including nginx, deleted the nginx cache and restarted and things started working properly again.

There is nothing called poco used by Mastodon so it is still a mystery why this was cached, but it does seem like a temporary corruption issue which is likely due to a hangup I had on my original attempt to upgrade.



From: joinmastodon@discoursemail.com
Sent: December 30, 2020 8:29 AM
To: mark@haydensplace.com
Reply-to: joinmastodon+d2c492f7855b2bb73e3217909ebfa254@discoursemail.com
Subject: [Mastodon Meta Discussion Board] [Serveradministration/Troubleshooting] “File /poco does not exist.” on web clientafter 3.3.0 upgrade

saper Instance Admin
December 30

Did you reboot the machine completely? Maybe some webpack dev server is running there on port 3000

2 Likes

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.