DigitalOcean Droplet & Media Cache

I’m using the DigitalOcean one click install & I noticed, unless I missed it, that there were no directions for setting up a cron for clearing media cache. My instance after a month crashed from having no more disk space (at 60gbs) & it took me a long time to figure out what had happened & how to fix it, especially being not so competent. Basically I had people hold my hand in the irc channel, it was very confusing. Just thought I’d mention it because I imagine if I didn’t miss the instructions then I won’t be the only person whose instance crashes due to no free disc space.

3 Likes

Maybe you could post the steps you needed to do, to enable cron to do just that?

1 Like

If left unchecked, storage can easily balloon on a Mastodon instance. I regularly use the tootctl commands (via bash scripts and cron jobs) to keep media in check, as this is one of the biggest consumers of storage on my instance.

As an example, in case anyone finds it useful, here is what I use. This is for an Ubuntu server where Mastodon is installed to the live folder within the home folder of the user ‘mastodon’ and should work on Mastodon v3.1.0 and later. All storage is local to the server; there are no S3 buckets or storage providers involved.

=-=-=-=-=-=-=

When logged into the server as a regular user, first switch to the ‘mastodon’ user:

sudo su - mastodon

Next, create the bash script clean_media.sh and edit it with nano (or your favourite editor), e.g.:

nano clean_media.sh

Then add the following lines:

#!/bin/bash
cd /home/mastodon/live
PATH=/home/mastodon/bin:/home/mastodon/.local/bin:/home/mastodon/.rbenv/plugins/ruby-build/bin:/home/mastodon/.rbenv/shims:/home/mastodon/.rbenv/bin:/usr/bin:/bin
RAILS_ENV=production bin/tootctl media remove > log/media_remove.log 2>&1

Save it (ctrl-o) and exit nano (ctrl-x), back to the command prompt.

Repeat for the other two scripts:

clean_previewcards.sh

nano clean_previewcards.sh

then add:

#!/bin/bash
cd /home/mastodon/live
PATH=/home/mastodon/bin:/home/mastodon/.local/bin:/home/mastodon/.rbenv/plugins/ruby-build/bin:/home/mastodon/.rbenv/shims:/home/mastodon/.rbenv/bin:/usr/bin:/bin
RAILS_ENV=production bin/tootctl preview-cards remove > log/preview-cards_remove.log 2>&1

then save and exit.

clean_orphans.sh

nano clean_orphans.sh

then add:

#!/bin/bash
cd /home/mastodon/live
PATH=/home/mastodon/bin:/home/mastodon/.local/bin:/home/mastodon/.rbenv/plugins/ruby-build/bin:/home/mastodon/.rbenv/shims:/home/mastodon/.rbenv/bin:/usr/bin:/bin
RAILS_ENV=production bin/tootctl media remove-orphans > log/media_remove-orphans.log 2>&1

then save and exit.

Next, make each script executable:

chmod +x clean_media.sh
chmod +x clean_previewcards.sh
chmod +x clean_orphans.sh

Running these scripts will clean up a different facet of media from your instance:

  • clean_media.sh will remove media files which are more than 7 days old,
  • clean_orphans.sh will remove files that do not belong to existing media attachments,
  • clean_previewcards.sh will remove preview cards from users which are more than 180 days old.

You can find out more about these (and other) tootctl commands at the official documentation site, e.g. should you wish to change your scripts to use parameters other than the defaults.

Run each script like this:

./clean_media.sh
./clean_previewcards.sh
./clean_orphans.sh

The first time you run each of these, it will take a while to complete, especially if your instance is well-federated, has had many users, or has been operating for a while. Removing orphaned media is especially time-consuming.

As each command is run, the output will be saved to a text file. You can see the output of each file using cat, e.g. cat /home/mastodon/live/log/preview-cards_remove.log, to get an idea of how much space was recovered after running that command.

Now set up cron jobs (you shoudl still be logged in as your ‘mastodon’ user) by running:

crontab -e

You might have to select an editor if it’s the first time you’ve run crontab. I choose to run these three scripts at different intervals because some will be more useful when run regularly than others, e.g. clean_media is run daily and often frees up the most amount of space, whereas clean_orphans I run less frequently because it takes a long time to complete and it does not recover a great amount of space (compared to the first time it was run).

Next add the following to the end:

00 3 * * * /home/mastodon/clean_media.sh
15 3 1 * * /home/mastodon/clean_previewcards.sh
30 3 * * 2 /home/mastodon/clean_orphans.sh

Save the crontab and exit the editor (if using nano then it’s ctrl-o to save and ctrl-x to exit, as before).

Cron will then run the scripts according to the following schedule:

  • Remove old media files at 0300h every day (it is reasonably fast and always finds material to remove)
  • Remove old preview cards at 0315h on the first day of each month (it is fast but, once running routinely, it rarely finds much to remove)
  • Remove orphaned media files at 0330h every Tuesday (this one can take quite a long time, so it’s set to run last in the set and to run only once a week)

Enough time is given between each task, in case they should all be required to run consecutively on the same day, e.g. all three will run when the first day of the month is also a Tuesday. All tasks are run (and should finish) close to when I make my site backups, to keep those smaller too.

Using these scripts has kept my storage growth relatively flat and manageable and it’s been a while since I had to hop to the next higher droplet configuration. If you are using DigitalOcean as your provider you can also enable Droplet Monitoring Alerts; these will email you when certain conditions are met, e.g. when storage is running low, processor runnning high, etc so you can do something about it before it becomes a bigger problem.

Anyway, I hope someone finds this useful!

Cheers,
Dave

1 Like