Vikrum Nijjar
Vikrum Nijjar Head of Operations July 30, 2012

Making the Switch from Amazon Cloudfront to Fastly

tl;dr: We made the mistake of using Amazon Cloudfront. We’ve switched to using Fastly instead. Fastly is spectacular

The ops footprint of Firebase has two major components: the realtime servers and the serving of static files via HTTP. Both concerns are split between staging and production environments. We set out with the goal of keeping the ops of serving static files to a minimum so we could put our efforts in the realtime infrastructure when and if needed.

Edit: The primary static file we are serving is our JS library that is <script> included by our end users. Being able to quickly and seamlessly upgrade our users without having them change their websites is crucial.

We chose Amazon’s Cloudfront backed by S3 with the thinking that not having to manage actual servers would make things operationally easier for us. It turns out that that wasn’t the case. To be fair, S3 makes no claims that the service can or should be used as an origin for a CDN. After duct-taping a few hacks together we had something that mostly worked.

The Amazon Setup

Those with some experience with Amazon’s services will know that there is a cottage industry of software and services built around Amazon’s APIs providing tools you would expect to find as a dashboard to any other self-serve web service. The Cloudfront website doesn’t tell you anything meaningful about utilization. What’s the cache hit rate? What files are being served? Everything has to happen on a per object level through the API—which makes it great to interact with programmatically but not so great if you haven’t built out your own tooling.

So, instead of managing servers we had to write a bunch of code that would do the following when we wanted to ship new static files:

  1. Delete the backing S3 bucket.
  2. Gzip content most people get via browser; don’t gzip content developers are usually grabbing via command line. Since S3 doesn’t vary by accept-headers, you can’t have both. Hack the planet!
  3. Upload the new bucket and set the appropriate encoding and cache headers per object as per number 2 above.
  4. Invalidate each and every object in Cloudfront one by one. Since you can’t enumerate Cloudfront, you need to reference the S3 bucket backing it to get a list of all of the objects needing invalidation. Don’t forget to invalidate directories with and without the trailing slash! Oh, also, make sure you appropriately handle special characters, like spaces, coming from S3 buckets. 

There might be other corner cases which we haven’t encountered. It’s a mystery.

Our bill for this setup was pretty amusing. Since we ship so often, and each deployment to production might have included several to staging, this meant there would be a huge number of invalidation calls to the Cloudfront API. Amazon begins charging for invalidations per file after the first 1000 files. Ouch. The invalidation portion of our bill was routinely larger than the amount that was coming from actually storing and serving the data. Sure, we could have intelligently invalidated and optimized for doing deltas but that additional complexity in the deployment code never seemed worth the overhead.

With regard to SSL, neither Cloudfront or S3 supports subject alternative names or using our own certificates. For Firebase users wanting to include files over SSL to avoid browsers warnings, they’ll have to use an Amazon domain name.

The Fastly Setup

The prerequisite for getting started with Fastly was that we stand up our own origin server (Fastly does support using S3 as an origin, but given the above, we decided against that). We went with Nginx. It’s a vanilla install with compression and cache control headers set in the configuration. Our new deployment of static files looks like this:

  1. rsync to the origin.
  2. Call Fastly’s purge_all API command.

No questions and no worries about corner cases. What a relief!

Not only does Fastly support SSL, they have an amazing dashboard. Insight into our CDN utilization is provided in realtime. Everything can be done using their web frontend.

Wonderful!

We sat down with the Fastly team prior to making the switch and found them to be a joy to work with. They are a hacker friendly group that are extremely receptive and accessible. If you’re looking for a CDN solution we highly recommend checking them out.

More Firebase Articles

Apr 22, 2014
Our New Home
Apr 16, 2014
Winners Use Firebase at HackIllinois
Apr 12, 2014
Firebase Turns Two Today!
Apr 08, 2014
OpenSSL Security Update
Apr 04, 2014
Fireside Chat with Ben Drucker from Valet.io
Apr 02, 2014
Fireside Chat with Kabriel Robichaux from Flawk
Mar 28, 2014
Firebase at EmberConf 2014
Mar 24, 2014
Announcing Streaming for the Firebase REST API
Mar 21, 2014
Announcing New Bindings for EmberJS
Mar 13, 2014
Fireside Chat: Union Test Prep