Vikrum Nijjar
Vikrum Nijjar Head of Operations July 30, 2012

Making the Switch from Amazon Cloudfront to Fastly

tl;dr: We made the mistake of using Amazon Cloudfront. We’ve switched to using Fastly instead. Fastly is spectacular

The ops footprint of Firebase has two major components: the realtime servers and the serving of static files via HTTP. Both concerns are split between staging and production environments. We set out with the goal of keeping the ops of serving static files to a minimum so we could put our efforts in the realtime infrastructure when and if needed.

Edit: The primary static file we are serving is our JS library that is <script> included by our end users. Being able to quickly and seamlessly upgrade our users without having them change their websites is crucial.

We chose Amazon’s Cloudfront backed by S3 with the thinking that not having to manage actual servers would make things operationally easier for us. It turns out that that wasn’t the case. To be fair, S3 makes no claims that the service can or should be used as an origin for a CDN. After duct-taping a few hacks together we had something that mostly worked.

The Amazon Setup

Those with some experience with Amazon’s services will know that there is a cottage industry of software and services built around Amazon’s APIs providing tools you would expect to find as a dashboard to any other self-serve web service. The Cloudfront website doesn’t tell you anything meaningful about utilization. What’s the cache hit rate? What files are being served? Everything has to happen on a per object level through the API—which makes it great to interact with programmatically but not so great if you haven’t built out your own tooling.

So, instead of managing servers we had to write a bunch of code that would do the following when we wanted to ship new static files:

  1. Delete the backing S3 bucket.
  2. Gzip content most people get via browser; don’t gzip content developers are usually grabbing via command line. Since S3 doesn’t vary by accept-headers, you can’t have both. Hack the planet!
  3. Upload the new bucket and set the appropriate encoding and cache headers per object as per number 2 above.
  4. Invalidate each and every object in Cloudfront one by one. Since you can’t enumerate Cloudfront, you need to reference the S3 bucket backing it to get a list of all of the objects needing invalidation. Don’t forget to invalidate directories with and without the trailing slash! Oh, also, make sure you appropriately handle special characters, like spaces, coming from S3 buckets. 

There might be other corner cases which we haven’t encountered. It’s a mystery.

Our bill for this setup was pretty amusing. Since we ship so often, and each deployment to production might have included several to staging, this meant there would be a huge number of invalidation calls to the Cloudfront API. Amazon begins charging for invalidations per file after the first 1000 files. Ouch. The invalidation portion of our bill was routinely larger than the amount that was coming from actually storing and serving the data. Sure, we could have intelligently invalidated and optimized for doing deltas but that additional complexity in the deployment code never seemed worth the overhead.

With regard to SSL, neither Cloudfront or S3 supports subject alternative names or using our own certificates. For Firebase users wanting to include files over SSL to avoid browsers warnings, they’ll have to use an Amazon domain name.

The Fastly Setup

The prerequisite for getting started with Fastly was that we stand up our own origin server (Fastly does support using S3 as an origin, but given the above, we decided against that). We went with Nginx. It’s a vanilla install with compression and cache control headers set in the configuration. Our new deployment of static files looks like this:

  1. rsync to the origin.
  2. Call Fastly’s purge_all API command.

No questions and no worries about corner cases. What a relief!

Not only does Fastly support SSL, they have an amazing dashboard. Insight into our CDN utilization is provided in realtime. Everything can be done using their web frontend.

Wonderful!

We sat down with the Fastly team prior to making the switch and found them to be a joy to work with. They are a hacker friendly group that are extremely receptive and accessible. If you’re looking for a CDN solution we highly recommend checking them out.

Help us spread the word

More Firebase Articles

Feb 13, 2015
Fireside Chat with Nau App
Feb 11, 2015
The 2^120 Ways to Ensure Unique Identifiers
Jan 23, 2015
A Video Walkthrough of Swift Fundamentals
Jan 21, 2015
Fireside Chat with TalkToUsers
Dec 30, 2014
2014 in Review: Google, Queries, and more!
Dec 17, 2014
Moment: Anonymous Chat With Firebase
Dec 12, 2014
Building Reviewable With Firebase
Dec 08, 2014
Fireside Chat with SurveyLegend
Dec 05, 2014
[Cross Platform Demo] Inetech Office Mover 5000
Nov 17, 2014
Tips & Tricks for Firebase Hosting