Importing content from other social services

Some social services let users export their data as an archive file. Instagram’s export, for example, provides an archive with JSON and media files that describe all the posts and stories posted by the user on the service. It would be nice if this data could be imported into mastodon. Such an import would help users migrate away from non-federated alternatives such as FB and IG to mastodon. Or at least, it would help users to consider mastodon as a viable alternative.

I’m creating this topic as a way to gather (technical) feedback on this idea. I’d be working on this myself and would like to know what problems I can anticipate.

Some potential pitfalls I have thought about:

  • Exporting the data archive would be a manual task for the user, typically the service emails you a download link and then you download the archive to your local storage. This would be a lot of effort to automate.
  • As the exports contain media files (images, video), the resulting archives tend to be rather large. Think 200Mb to 1-2GB.
  • Exports contain privacy-sensitive data, so privacy-conscious users are unlikely to upload it to some shady third party service that offers the import.
  • To me this sounds like such an import application should be run by a trusted party that has the resources necessary to process potentially big archives. For example, either the mastodon instance itself or a computing device of your choosing (desktop, headless server, laptop, mobile device(?)).

When posting the exported contents to mastodon, I see some issues as well:

  • If we want to keep the dates of the imported statuses, we need to be able to post statuses in the past. I believe this is disallowed by the mastodon REST api, according to the documentation. I don’t yet know a way around this, without altering the REST API.
  • Rate limiting: most mastodon instances don’t allow you to submit hundreds of REST API requests per second.
  • Some posts might violate the instance’s rules (there would then need to be moderation for posts in the past?)
  • Some posts might violate the instance’s restriction: e.g. more characters than allowed or more media messages than allowed.

Are there any other issues that I should expect? I want to have a working solution for myself (to import all the contents of a travel instagram account to my mastodon instance), but it would be nice if the result ends up being usable by others as well.

Cheers!

I’ve stumbled upon mastodon.py, which would do nicely to abstract the interfacing between mastodon and a client app.

‘Posting toots in the past’ appears to be the biggest issue. Might have to alter the API here, to accept scheduled_at to be in past.

Rather than working around the restrictions of the REST API (which exist for good reason), it might be better to add the import functionality to mastodon directly. This would give much greater control over altering the instance’s state. I think I’ll start by adding a CLI script that does all of the work, similar to the ones in this folder mastodon/lib/mastodon at main · tootsuite/mastodon · GitHub

If there would be interest, it could be added as a proper service with its own REST resource later on.

The real question here is, do we really want people dump their IG/FB/Twitter junk into our instances and into the fediverse?

If a lack of an alternative for hosting ‘past’ social media posts is keeping someone from exercising their right to be forgotten on the service hosting said posts, then I would argue that an import service does hold value. Of course, not everyone considers their past posts to hold value. I imagine there is little value in importing ramblings from twitter. But there’s plenty of examples that are (e.g. travel diaries, family photos).

Whether mastodon should be such an alternative is subject to debate. Fortunately instances can decide that for themselves.

1 Like

I’ve completed a first attempt at importing the posts.json (and media) exported from instagram. It ended up as an admin CLI script, as this seemed most sensible for a ‘one time import’ function. It’s available on this branch under lib/mastodon/ig_import_cli.rb (use the script at your own risk, though it doesn’t contain anything destructive). The result of an import can be found here: Floris cycles (@floriscycles@social.vdna.be) - social.vdna.be mastodon

I’m not too happy with the way posts longer than 500 characters are handled, as the 111 posts turned into 178 toots (which are all displayed on the profile page). It would be better if replies wouldn’t be shown on the profile, but I think this cannot be changed in mastodon.

1 Like

An integrated ‘live’ client website a meta-mastodon and meta-facebook (yes, pun intended) that actually uses browser automation (maybe chrome puppeteer, maybe autohotkey) to simulate user events so that it literally is just doing exactly the same as a user would with mouse and keyboard.

I have thought about this idea for quite a while, and i think that for everybody to be able to migrate, it should be like an extension that ‘hooks’ into facebook and just maybe inserts mastodon posts and links facebook users to mastodon users. …