Robots.txt will by honored by my instance picker, heads up!


#1

I will be honoring all robots.txt files for my instance picker by Friday, July 14.

This means that your instance info may disappear if you use the easy:

User-Agent: *
Disallow: /

to kick bots in the pants.

If you still want comprehensive info listed at my instance picker, you can grant access to user-agent “UpsideBot” in robots.txt, by adding this:

User-Agent: UpsideBot
Allow: /

Or, if you’re picky:

User-Agent: UpsideBot
Allow: /about
Allow: /api/v1/timelines/public
Allow: /api/v1/instance.json
Disallow: /

To be honest, /about satisfies UpsideBot, and with that alone you’ll be listed properly. Disallowing /api/timelines/public will elide all toots from the “fun” version, which might be a good idea depending on your instance needs. (I will be removing toot previews soon anyways, but until then…)

More Info

UpsideBot sets its user-agent HTTP header thus:

User-Agent: UpsideBot/0.9 (+https://github.com/upsided/DescribedInstanceList/blob/master/tools/AboutThisInstance.py)

I run UpsideBot manually about once every 2 days and put a delay between GETs (currently 1 second). Trying to be kind.

Obviously, you can set whatever rules you like, and UpsideBot will honor them. Thanks for your time!