Media federation issue from other instance

Images including profile image and headers are not federated from other instances. They have no issue federating to other instances.

When I read journalctl -u mastodon-sidekiq.service there was this part

Dec 12 22:55:14 canors_ubuntu bundle[7872]: 2019-12-12T13:55:14.814Z 7872 TID-gou7ficj8 ActivityPub::ProcessingWorker JID-252edf44d589635787607368 INFO: start
Dec 12 22:55:14 canors_ubuntu bundle[7872]: Command :: file -b --mime '/tmp/8d777f385d3dfec8815d20f7496026dc20191212-7872-z1zc7k'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: [paperclip] Trying to link /tmp/8d777f385d3dfec8815d20f7496026dc20191212-7872-z1zc7k to /tmp/8d777f385d3dfec8815d20f7496026dc20191212-7872-dwdkej.jpeg
Dec 12 22:55:14 canors_ubuntu bundle[7872]: Command :: file -b --mime '/tmp/8d777f385d3dfec8815d20f7496026dc20191212-7872-dwdkej.jpeg'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: [paperclip] Trying to link /tmp/8d777f385d3dfec8815d20f7496026dc20191212-7872-z1zc7k to /tmp/0bc5cbda56600d5005db26885c08459f20191212-7872-9909cs
Dec 12 22:55:14 canors_ubuntu bundle[7872]: [paperclip] Link failed with No such file or directory @ rb_file_s_link - (/tmp/8d777f385d3dfec8815d20f7496026dc20191212-7872-z1zc7k, /tmp/0bc5cbda56600d5
Dec 12 22:55:14 canors_ubuntu bundle[7872]: 2019-12-12T13:55:14.837Z 7872 TID-gou70566k ActivityPub::ProcessingWorker JID-926cdff35aefc56454254672 INFO: fail: 2.378 sec
Dec 12 22:55:14 canors_ubuntu bundle[7872]: 2019-12-12T13:55:14.837Z 7872 TID-gou70566k WARN: {"context":"Job raised exception","job":{"class":"ActivityPub::ProcessingWorker","args":[3910,"{\"@conte
Dec 12 22:55:14 canors_ubuntu bundle[7872]: 2019-12-12T13:55:14.837Z 7872 TID-gou70566k WARN: Errno::ENOENT: No such file or directory @ rb_sysopen - /tmp/8d777f385d3dfec8815d20f7496026dc20191212-78
Dec 12 22:55:14 canors_ubuntu bundle[7872]: 2019-12-12T13:55:14.837Z 7872 TID-gou70566k WARN: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/2.6.0/fileutils.rb:1385:in `initialize'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/2.6.0/fileutils.rb:1385:in `open'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/2.6.0/fileutils.rb:1385:in `copy_file'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/2.6.0/fileutils.rb:492:in `copy_file'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/2.6.0/fileutils.rb:419:in `block in cp'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/2.6.0/fileutils.rb:1557:in `block in fu_each_src_dest'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/2.6.0/fileutils.rb:1573:in `fu_each_src_dest0'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/2.6.0/fileutils.rb:1555:in `fu_each_src_dest'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/2.6.0/fileutils.rb:418:in `cp'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/paperclip-6.1.0/lib/paperclip/io_adapters/abstract_adapter.rb:67:in `rescue in link_or_copy_
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/paperclip-6.1.0/lib/paperclip/io_adapters/abstract_adapter.rb:60:in `link_or_copy_file'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/paperclip-6.1.0/lib/paperclip/io_adapters/abstract_adapter.rb:55:in `copy_to_tempfile'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/paperclip-6.1.0/lib/paperclip/io_adapters/file_adapter.rb:21:in `cache_current_values'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/paperclip-6.1.0/lib/paperclip/io_adapters/file_adapter.rb:11:in `initialize'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/paperclip-6.1.0/lib/paperclip/io_adapters/registry.rb:33:in `new'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/paperclip-6.1.0/lib/paperclip/io_adapters/registry.rb:33:in `for'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/paperclip-6.1.0/lib/paperclip/attachment.rb:549:in `post_process_style'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/paperclip-6.1.0/lib/paperclip/attachment.rb:524:in `post_process_styles'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/paperclip-6.1.0/lib/paperclip/attachment.rb:517:in `block (2 levels) in post_process'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:132:in `run_callbacks'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/paperclip-6.1.0/lib/paperclip/callbacks.rb:38:in `run_paperclip_callbacks'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/paperclip-6.1.0/lib/paperclip/attachment.rb:515:in `block in post_process'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:132:in `run_callbacks'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/paperclip-6.1.0/lib/paperclip/callbacks.rb:38:in `run_paperclip_callbacks'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/paperclip-6.1.0/lib/paperclip/attachment.rb:514:in `post_process'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/paperclip-6.1.0/lib/paperclip/attachment.rb:470:in `post_process_file'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/paperclip-6.1.0/lib/paperclip/attachment.rb:112:in `assign'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/paperclip-6.1.0/lib/paperclip/has_attached_file.rb:66:in `block in define_setter'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/models/concerns/remotable.rb:39:in `block (2 levels) in remotable_attachment'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/lib/request.rb:75:in `perform'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/models/concerns/remotable.rb:24:in `block in remotable_attachment'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/lib/activitypub/activity/create.rb:201:in `block in process_attachments'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/lib/activitypub/activity/create.rb:191:in `each'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/lib/activitypub/activity/create.rb:191:in `process_attachments'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/lib/activitypub/activity/create.rb:71:in `process_status_params'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/lib/activitypub/activity/create.rb:33:in `process_status'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/lib/activitypub/activity/create.rb:14:in `block in perform'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/mario-redis-lock-1.2.1/lib/redis_lock.rb:44:in `acquire'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/lib/activitypub/activity/create.rb:7:in `perform'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/services/activitypub/fetch_remote_status_service.rb:23:in `call'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/lib/activitypub/activity.rb:159:in `fetch_remote_original_status'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/lib/activitypub/activity.rb:153:in `status_from_object'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/lib/activitypub/activity/announce.rb:7:in `perform'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/services/activitypub/process_collection_service.rb:41:in `process_item'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/services/activitypub/process_collection_service.rb:32:in `block in process_items'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/services/activitypub/process_collection_service.rb:32:in `reverse_each'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/services/activitypub/process_collection_service.rb:32:in `each'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/services/activitypub/process_collection_service.rb:32:in `map'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/services/activitypub/process_collection_service.rb:32:in `process_items'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/services/activitypub/process_collection_service.rb:19:in `call'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/workers/activitypub/processing_worker.rb:9:in `perform'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
Dec 12 22:55:14 canors_ubuntu bundle[7872]: /home/mastodon/live/app/lib/sidekiq_error_handler.rb:5:in `call'

I went over this thread but I had no idea where I should be looking when not running in docker. Any suggestion would be appreciated . Thanks.

This error message seems to be saying that it can’t find your image files in /tmp/... - I don’t think they are supposed to be there and your webserver shouldn’t have permission to save things in /tmp anyway.

Try looking in the file at /etc/nginx/sites-available/mastodon

And then look for a line like this:

location ~ ^/(emoji|packs|system/accounts/avatars|system/media_attachments/files) {

This is line 52 in this file. If it says something like:

location ~ ^/(emoji|packs|system/accounts/avatars|/tmp) {

…then that’s your problem. Change to the first example above and it will hopefully fix your issue.

1 Like

Hi thanks for the reply.

At first I thought it’s strange to have webserver saving file to /tmp too, so I gave 777 permission to /tmp folder, and that didn’t solve the issue

Also, I checked my mastodon nginx file and it did have location ~ ^/(emoji|packs|system/accounts/avatars|system/media_attachments/files) { line

Maybe I should restart nginx service?

paperclip uses /tmp for it’s intermediate image processing (resizing, converting formats, etc). in this case, I’ve occasionally seen this error when users ran out of disk space or available file handles. Maybe check those stats?

1 Like

thanks for the reply nightpool

here’s what I got from df -H

Filesystem      Size  Used Avail Use% Mounted on
udev            8.4G     0  8.4G   0% /dev
tmpfs           1.7G  1.5M  1.7G   1% /run
/dev/sda2       126G   96G   25G  80% /
tmpfs           8.4G  279M  8.1G   4% /dev/shm
tmpfs           5.3M     0  5.3M   0% /run/lock
tmpfs           8.4G     0  8.4G   0% /sys/fs/cgroup
/dev/loop0      229M  229M     0 100% /snap/nextcloud/16402
/dev/loop1       72M   72M     0 100% /snap/google-cloud-sdk/109
/dev/loop2       71M   71M     0 100% /snap/google-cloud-sdk/108
/dev/loop3       60M   60M     0 100% /snap/powershell/77
/dev/loop4      129M  129M     0 100% /snap/docker/418
/dev/loop5       94M   94M     0 100% /snap/core/8213
/dev/loop6       60M   60M     0 100% /snap/powershell/39
/dev/loop8       94M   94M     0 100% /snap/core/8268
/dev/loop9      223M  223M     0 100% /snap/nextcloud/16739
/dev/loop7      127M  127M     0 100% /snap/docker/423
/dev/sdb        984G  650G  284G  70% /mnt/sdb
tmpfs           1.7G     0  1.7G   0% /run/user/1000

Won’t 25G be enough? when I removed media with tootctl media remove only 18M was deleted.

25G should be more then enough, yeah

Looks like the logfile you have posted here is truncated a bit - can you try to post it again? (Copy paste from the window is not good…). Long lines got cut short, also would be good to have more lines about TID-gou7ficj8 and TID-gou70566k before the problem happens

1 Like

Hi saper Thanks for replying

As you mentioned I tried to export full log to txt file and then paste it here not to miss out any part. Unfortunately I wasn’t able to get the exactly same part but I found similar part so I append last 500 lines below:

Oh I got caught by maximum character limit. Pasted to gist, link is here

1 Like

Thanks, looks better… but I still don’t understand why this happens. Looks like a race condition. Which version are you running?

1 Like

Fix attachment not being re-downloaded even if file is not stored by Gargron · Pull Request #12125 · tootsuite/mastodon · GitHub could be related, it’s there since 3.0.1

1 Like

I still don’t understand why it happened but I just pulled from Mastodon repository and overwrote it onto my instance. Now it seems working so… Thank you saper, nightpool and hugh for helping me out. Really appreciated it.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.