I will be honoring all
robots.txt files for my instance picker by Friday, July 14.
This means that your instance info may disappear if you use the easy:
User-Agent: * Disallow: /
to kick bots in the pants.
If you still want comprehensive info listed at my instance picker, you can grant access to user-agent “UpsideBot” in
robots.txt, by adding this:
User-Agent: UpsideBot Allow: /
Or, if you’re picky:
User-Agent: UpsideBot Allow: /about Allow: /api/v1/timelines/public Allow: /api/v1/instance.json Disallow: /
To be honest,
/about satisfies UpsideBot, and with that alone you’ll be listed properly. Disallowing
/api/timelines/public will elide all toots from the “fun” version, which might be a good idea depending on your instance needs. (I will be removing toot previews soon anyways, but until then…)
UpsideBot sets its user-agent HTTP header thus:
User-Agent: UpsideBot/0.9 (+https://github.com/upsided/DescribedInstanceList/blob/master/tools/AboutThisInstance.py)
I run UpsideBot manually about once every 2 days and put a delay between GETs (currently 1 second). Trying to be kind.
Obviously, you can set whatever rules you like, and UpsideBot will honor them. Thanks for your time!