Problem using Exoscale as an external host for media storage

cwahlers · February 26, 2020, 7:47pm

Hi all, i think i’m going nuts.

I’m trying to use Exoscale for my Mastodon instance’s media storage. I signed up, created bucket, transferred all media files to that bucket, and enabled S3 in .env.production:

S3_ENABLED=true
S3_BUCKET=my-bucket-name
AWS_ACCESS_KEY_ID=$EXOKEYHERE
AWS_SECRET_ACCESS_KEY=$EXOSECRETHERE
S3_REGION=ch-dk-2
S3_HOSTNAME=sos-ch-dk-2.exo.io
S3_PROTOCOL=https

Oh yeah i followed this guide

Everything works fine, the site pulls all media files from that bucket.
BUT. It fails uploading new media files to that bucket.
For example when i try and just create a new toot and upload an image, i get this in the logs:

method=POST path=/api/v1/media format=html controller=Api::V1::MediaController action=create status=500 error='Seahorse::Client::NetworkingError: Failed to open TCP connection to my-bucket-name.s3.ch-dk-2.amazonaws.com:443 (getaddrinfo: Name or service not known)' duration=722.12 view=0.00 db=23.13

Why does it use s3.ch-dk-2.amazonaws.com and not the host i provided in S3_HOSTNAME?

I tried commenting out S3_REGION, and then got:

method=POST path=/api/v1/media format=html controller=Api::V1::MediaController action=create status=500 error='Aws::S3::Errors::InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.' duration=1410.25 view=0.00 db=50.97

That key definitely exists.

Maybe any of you have an idea what is going wrong here?
Thanks so much in advance!

Cheers,
Claus.

saper · February 26, 2020, 12:47am

Mastodon is using a library called “paperclip” to talk to the S3 storage.

I don’t know how Exoscale is supposed to work, but here is an explanation about how the URLs for the storage calls are being constructed:

github.com

thoughtbot/paperclip/blob/15f85a88394d2d90295527349d9b1460c7d327b0/lib/paperclip/storage/s3.rb#L71-L91


# * +url+: There are four options for the S3 url. You can choose to have the bucket's name
#   placed domain-style (bucket.s3.amazonaws.com) or path-style (s3.amazonaws.com/bucket).
#   You can also specify a CNAME (which requires the CNAME to be specified as
#   :s3_alias_url. You can read more about CNAMEs and S3 at
#   http://docs.amazonwebservices.com/AmazonS3/latest/index.html?VirtualHosting.html
#   Normally, this won't matter in the slightest and you can leave the default (which is
#   path-style, or :s3_path_url). But in some cases paths don't work and you need to use
#   the domain-style (:s3_domain_url). Anything else here will be treated like path-style.
#
#   Notes:
#   * The value of this option is a string, not a symbol.
#     <b>right:</b> <tt>":s3_domain_url"</tt>
#     <b>wrong:</b> <tt>:s3_domain_url</tt>
#   * If you use a CNAME for use with CloudFront, you can NOT specify https as your
#     :s3_protocol;
#     This is *not supported* by S3/CloudFront. Finally, when using the host
#     alias, the :bucket parameter is ignored, as the hostname is used as the bucket name
#     by S3. The fourth option for the S3 url is :asset_host, which uses Rails' built-in
#     asset_host settings.
#   * To get the full url from a paperclip'd object, use the
#     image_path helper; this is what image_tag uses to generate the url for an img tag.

If S3_ENDPOINT is given there is an additional logic using S3_OVERRIDE_PATH_STYLE variable:

github.com

tootsuite/mastodon/blob/7face973fa1c7d6c18b06d427ea0b7a741d11466/config/initializers/paperclip.rb#L54-L61


if ENV.has_key?('S3_ENDPOINT')
  Paperclip::Attachment.default_options[:s3_options].merge!(
    endpoint: ENV['S3_ENDPOINT'],
    force_path_style: ENV['S3_OVERRIDE_PATH_STYLE'] != 'true',
  )


  Paperclip::Attachment.default_options[:url] = ':s3_path_url'
end

Maybe this is something you can start with

cwahlers · February 26, 2020, 2:50am

Thanks for the hint! I’m still unsure what i did wrong (if anything) though.

In the meantime i just created a good ole S3 bucket on AWS and things worked on first try. I’ll just go with that for the time being.

saper · February 26, 2020, 7:47pm

AWS is widely used. I think there are some configuration tweaks that may be needed to get it work, maybe there are other that did configure this successfully?

cwahlers · February 28, 2020, 6:55am

Apparently in addition to S3_HOSTNAME, S3_ENDPOINT must also be set (i did not do that)

S3_ENDPOINT is what Mastodon will use to upload files. S3_HOSTNAME is what will be used in the user’s browser. Both are required.

This is a quote from here: https://angristan.xyz/2018/05/moving-mastodon-media-files-to-wasabi-object-storage/#isso-400 (it’s Wasabi specific, but should apply to Exoscale too).

I’m gonna test if that solves it later tonight.

system · March 11, 2020, 11:25pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.