AWS presigned posts

Josh McArthur August 26, 2018

When designing a JSON API that needs to deal with uploaded files, there’s a few options, but all of them have a degree of compromise. At the end of the day, a file uploaded via HTTP must be provided as a form-encoded name-value set, and this conflicts with how most JSON APIs work.

At Rabid, we had a need to support just such an API. With the help of the AWS S3 SDK, we found a great pattern for supporting file uploads without compromising on our APIs.

The trick is to use the AWS API to create a presigned post request, and treat this as an API resource. API clients can then pass several parameters, which can be validated before the request is created and returned as the response. In our application, we call this an “Upload Request”:

class UploadRequestsController < ApplicationController def create render json: aws_presigned_post.fields, status: :created, location: aws_presigned_post.url end


  private
  def aws_presigned_post
    @presigned_post ||= aws_bucket.presigned_post(
      key: upload_request_params[:filename],
      expires: 1.hour.from_now,
      success_action_status: "201"
    )
  end
  def aws_bucket
    bucket_name = Rails.application.secrets["aws"]["uploads_bucket"]
    Aws::S3::Bucket.new(bucket_name)
  end

def upload_request_params params.require(:upload_request).permit(:filename) end end

This allows the API client to perform a simple POST request with the file name, and get back a JSON object that can be passed directly to many HTTP libraries, or even jQuery, if your API client is a web browser:

POST /upload_requests HTTP/1.1 Host: localhost:3000 Content-Type: application/json


{
    "upload_request": {
        "filename": "test.jpg"
    }
}

Status: 201 Created Location →https://myapp-uploads.s3-ap-southeast-2.amazonaws.com { "key": "test.jpg", "Expires": "Thu, 28 Jul 2016 21:27:29 GMT", "success_action_status": "201", "policy": "eyJleHBpcmF0aW9uIj...snip", "x-amz-credential": "ACCESS_KEY/20160728/ap-southeast-2/s3/aws4_request", "x-amz-algorithm": "AWS4-HMAC-SHA256", "x-amz-date": "20160728T202729Z", "x-amz-signature": "SIGNATURE" }

This is pretty capable already, but could have some undesirable side-effects, as we’re not really restricting what can uploaded. Fortunately, the presigned_post method supports two important options to restrict the file that is permitted:

content_type: The mime type of the file can be uploaded
content_length: The permitted range of file size that can be uploaded

Adding these restrictions to the controller is pretty simple:

class UploadRequestController < ApplicationController include ActionView::Helpers::NumberHelper


  PERMITTED_CONTENT_TYPES = %w( application/pdf image/jpeg )
  PERMITTED_CONTENT_RANGE = 0..50.megabytes
  before_action :validate_content_type!,
                :validate_content_range!,
                only: :create
  # ..snip
  private
  def validate_content_type!
    content_type = upload_request_params[:content_type]
    return if PERMITTED_CONTENT_TYPES.include?(content_type)
    render json: { error: "#{content_type} is not permitted" }, status: :unsupported_media_type
  end
  def validate_content_range!
    file_size = upload_request_params[:file_size]
    return if PERMITTED_CONTENT_RANGE.include?(file_size.to_i)
    error = "File must be larger than #{number_to_human_size(PERMITTED_CONTENT_RANGE.first)}"
    error << " and smaller than #{number_to_human_size(PERMITTED_CONTENT_RANGE.last)}"

    render json: { error: error }, status: :payload_too_large
  end

# ..snip end

Now that the content type and file size are “validated”, we receive responses that the client can react to easily, with large files receiving an error:

Status: 413 Payload Too Large { "error": "File must be larger than 0 Bytes and smaller than 50 MB"

And files with an incorrect content type receiving the error:

Status: 415 Unsupported Media Type { "error": "application/ms-word is not permitted" }

We’ve found that this pattern has worked well for us, especially for API-dependent client front end libraries such as Ember. There are some aspects to this pattern that have not been covered, as they’re a bit subjective depending on your own app:

IE Compatibility: Internet Explorer before version 10 does not send through the file size and content type for the file. Depending on the level of compatibility you need to offer users, this may not be an issue for you. If it is, you may wish to explore fallbacks as we did – we use the mime-types gem to guess the content-type from the file extension to generate the presigned post request, and just have a maximum cap for the file size restriction.
Storing permitted file size/type collections: for this example, we’ve chosen to store the collections of permitted file sizes and types in constants in the controller. For a production app though, you probably want to move these to your application’s config, or a Ruby object.

Ackama says...