Clean instance (unused accounts, older media,...)

Hello,

I’m running my instance for quite a long time now, and I notice that it uses more and more disk space. I understand that new toots, users, media,… use disk space, there is nothing wrong with that.

However, I was wondering if it was possible to “clean” the instance : delete old/unused account, old toots, old media,…

Well, before actually doing it, what does the community think about that?
Nowadays, we are all used to never delete anything. I can still find the post I made on Facebook years ago, and my first photo I sent on Twitter is still available.
However, how sustainable is it for Mastodon and all these instances? Some of them are run by experienced administrator, on big servers,… But some other are just small instances, ran by “hobbyist admin” on small servers.
On these smaller instances, one day of another, the database and media will use too much disk space, or request too much CPU power to run.

What are you thoughts about this?

Thanks!

Hello JF002,

I think it is a good thing to clean data in order to save space and to avoid to manage too much data.
You may be interested by this in Mastodon’s documentation: List-of-Rake-tasks.md#media-storage

There are Rake commands which allow you to clean old data and save on disk space. You just need to setup them with a cron every week for example.

Killian

Hi @KillianKemps, and thanks for the tip.

I’ve tried these Rake tasks on Mastodon v2.6.5 (running inside dockers), but it won’t work:

$ docker-compose run --rm web rake mastodon:media:remove_remote
Starting mastodon_db_1    ... done
Starting mastodon_redis_1 ... done
rake aborted!  
Don't know how to build task 'mastodon:media:remove_remote' (see --tasks)
(See full trace by running task with --trace)

Have these tasks been replaced by other tasks/tools?

EDIT : they have been replaced by the tool ‘tootctl’ since version 2.6. Any idea how to use this tools with docker?
EDIT 2 : trying docker-compose run --rm web bin/tootctl …

Thanks!

JF

Hello @JF002,

Yes these Rake tasks have now changed and have been replaced by tootctl.

In order to clean your media files, you can now use this command:

docker-compose run --rm web tootctl media remove

You can use the help command to get details about how this command works:

docker-compose run --rm web tootctl media help remove
Usage:
  tootctl media remove

Options:
  [--days=N]                         
                                     # Default: 7
  [--background], [--no-background]  
  [--verbose], [--no-verbose]        
  [--dry-run], [--no-dry-run]        

Description:
  Removes locally cached copies of media attachments from other servers.

  The --days option specifies how old media attachments have to be before they are removed. It defaults to 7 days.

  With the --background option, instead of deleting the files sequentially, they will be queued into Sidekiq and the command will exit as soon as possible. In Sidekiq they will be processed 
  with higher concurrency, but it may impact other operations of the Mastodon server, and it may overload the underlying file storage.

  With the --dry-run option, no work will be done.

  With the --verbose option, when media attachments are processed sequentially in the foreground, the IDs of the media attachments will be printed.

Killian

1 Like

Thank you very much, @KillianKemps. It’s just ran for a very long time and freed more than 50Go!! I should run this more often !

You’re welcome!

I suggest you to create a daily cron which will run the command every night.

To do this, edit your crontab (it will open the file with Vim as default editor). I assume here you run the command as root user.

crontab -e

And then copy this command (you may adapt it to how you installed Mastodon):

30 2 * * * sudo -iu mastodon && cd /home/mastodon/live && /usr/local/bin/docker-compose run --rm web tootctl media remove >> /var/log/cron_mastodon_cleaning_remote_images.log 2>&1

This cron will

  • run everyday at 2:30AM (30 2 * * *)
  • change to user mastodon (sudo -iu mastodon)
  • move to the directory where you have docker-compose (cd /home/mastodon/live)
  • and then run the cleaning command (/usr/local/bin/docker-compose run --rm web tootctl media remove).
  • Finally, the end (>> /var/log/cron_mastodon_cleaning_remote_images.log 2>&1) is for keeping the messages of the command in the /var/log/cron_mastodon_cleaning_remote_images.log file in case a problem happened and you want to check what it was.

Maybe you have no dedicated mastodon user for your Mastodon installation and you may directly write the command as root user. For example, something like this:

30 2 * * * cd /root/mastodon/live && /usr/local/bin/docker-compose run --rm web tootctl media remove >> /var/log/cron_mastodon_cleaning_remote_images.log 2>&1

In any case to avoid possible issues use /usr/local/bin/docker-compose instead of docker-compose only, as in cron the PATH is not always defined and it may not find the command.

Hope it helps :slight_smile:

Killian

Hello !
I’m reviving this thread as I’m still in the same issue than in the OP : the instance uses more and more disk space, and it’s becoming harder and harder to maintain and backup.

I’ve just run docker-compose run --rm web tootctl media remove and docker-compose run --rm web tootctl cache clear, and the instance still use more than 57GB on the HDD.

Is there any way to recover more disk space (less than 50GB would be ideal) ? Is it possible to delete older content, media, users,…
Note that this is a personal instance, I’m the only user, and I do not need to keep all the history of the instance. It would not be an issue for me to delete the content from more than 1 or 2 years, for example.

Thanks!

Ok, I did some more analysis and noticed that Mastodon only use 14GB, not 40GB!

All this disk space was used by docker itself : it was using the storage driver “vfs” which waste a lot of space by duplicating all the files for each and every layer of the images.

The solution was to switch docker to the aufs storage driver.

Here’s what I did:

  • Stop & delete all containers
  • Delete all images
  • Notice that 15GB have been freed on the disk
  • Edit the file /etc/docker/daemon.json and add the following field:
{
    ...
    "storage-driver": "aufs"
}
  • Restart docker and check with docker info that the storage driver is aufs
  • Recreate and restart all containers with docker-compose up -d
  • It’s working, and the total disk space used on the disk is now 25GB, which is a lot better than before!

EDIT:
I’ve just found a new step, but do it at your own risk:

  • Delete the folder /var/lib/docker/vfs as there is still a lot data (5GB) in this folder, and I do not have any other container/volume/…

Now, the whole system is using 20GB, and it’s fine to me!