Why Firefly Requires an SLA to Support Outside Hosts
Why Firefly Requires an SLA to Support Outside Hosts
by Jeremy Usher, Co-Founder, President
Occasionally a client will ask us why Firefly charges more to work with third-party web hosts than we do to physically host applications internally. Because many factors are at play, we have put together a brief post describing a few of the more important ideas.
Firefly specializes in high quality, enterprise level hosting. Our hosting clients include state government, school districts, colleges, municipalities and multinational corporations. We utilize only the best Tier III or better datacenters in the US and Western Europe - guaranteeing geographic redundancy, biometric access, 24 hour armed security, redundant access to major carriers, backup generators and more. Firefly's applications are monitored around the clock for performance and threat mitigation. Meanwhile, complete backups of sites are synced offsite to Amazon S3 each week so that all materials can be retrieved and restored to any location in the event natural disaster or other calamity. Since our incorporation 10 years ago, Firefly has never lost a single client because they were unhappy with our hosting. Furthermore, because clients always have complete access to their site and materials, they are always free to migrate materials elsewhere if they ever desired to do so.
Virtually all Firefly clients host on Firefly's affordable, powerful and secure infrastructure, but for those requiring an outside presence for any reason, Firefly requires a minimum $1,200 / year Service Level Agreement (SLA) to cover the time needed to handle third-party environments.
There are many reasons that working with outside hosts requires extra time and effort and so necessitates a separate contract governing systems administration and support. A few examples are below.
Outside Hosts are Unreliable and Create Billing Confusion
In the days before Firefly established a multinational network of high performance servers, we hosted sites with third parties more often. A particular scenario became common. It went like this: An obscure problem would disable a website or cause some kind of performance issue. Not knowing the cause, a client would contact our team. Firefly would launch a process of inspecting logs, debugging code and searching for the cause of the problem. 19 times out of 20 we would discover the problem was the host or another third party component (like a web service going down) - or perhaps something the client themselves broke (like regenerating their Authorize.net transaction key without telling anyone!). So what was the result? More often than not we had spent 5+ hours of developer time ($500+) tracking down a problem that wasn't ours and we had no ability to fix. There are at least two problems with that result. First: we don't like wasting half a day tracking down problems that have nothing to do with us. Second: clients don't like being surprised, and have trouble understanding, a $500 bill for a problem that isn't necessarily fixed. The only solution is to avoid the confusion entirely by controlling the full stack. Today, if something breaks - the server, the application, the datacenter, California - we'll know what happened and how to fix it (we can't fix CA, but we can move your site to New Jersey). You'll know to contact us if there is a problem and we'll be able to help. Hosting on a third-party machine revives the possibility of these difficulties, which is one reason an SLA is required to cover these sorts of costs without surprises.
There is no "Standard" Server Environment
Firefly's web software (Apollo CMF) runs on top of the most prevalent server software stack in the world: the Linux-Apache-MySQL-PHP (LAMP) stack. For readers not well versed in these technologies, Linux is the operating system, Apache is the web server, MySQL is the database and PHP is the programming language. All are open source and all are the most common technology in their respective domains worldwide.
The prevalence of these technologies suggests that it is very easy to find a host that supports LAMP software at a rudimentary level. However, there are literally billions of permutations of modules, settings and other configuration variables making each hosting environment very much unique. Are you running PHP 5.6? How about MySQL 5.6? Does MySQL run with the UTF-8 character set? Are PHP uploads limited to 50MB or 2MB? What user does PHP run under? Are .htaccess overrides permitted? What is the sendmail limit per hour? Is multibyte string support enabled? How about cURL or ImageMagick? Are you using a single file for all InnoDB tables? Do stream wrappers work as expected? Has shell exec() been forcibly disabled? Is SSH access allowed? These questions will all affect whether your application runs smoothly, or at all, and virtually every host will have a different answer. Provided at least the correct versions of PHP and MySQL and the necessary modules, it is usually possible to work around configuration idiosyncrasies, but it will necessarily take time to research the foreign environment and investigate workarounds.
Big Hosts are like Big Box Stores - Low Price, Low Quality
Critically, it is important to understand that the quality and security of large hosts is limited by their very structure. Because large dedicated hosting companies do not create their own software, but rather run existing packages for often hundreds of customers on a single machine, upgrades to major components like the programming language or database will often break the sites of many customers residing on that machine. Do you think the hosting companies want to receive 100 angry phone calls from customers wondering why their site was broken? No - and so they let the software on their machines become out of date and insecure. Moreover, because hosting so many unknown and ultimately untrusted applications is in fact a great risk, a large number of features are typically disabled or removed from the hosting platform to protect users on a machine from malicious (or compromised) clients. At Firefly, we host only our own software and we know exactly what must be done during an upgrade, allowing us to keep machines up to date with the latest in features and security.
Big Box Host Changes are Slow and Arduous
Almost anyone who has ever contacted support at a big box host will need little convincing that it is best to avoid repeating the ordeal. Take a common scenario - you need a PHP module enabled and you don't have root access to your machine. After retrieving the login credentials and support website location for the host in question, you contact support. Like all big box hosting companies, you are fed through a ticket system and they assign you, by default, to "Level I" support, hoping their most inexperienced technicians can help. An hour after you hear from Level I support, who asks you to clarify the question (being Level I, they are unlikely to understand complicated issues). Half an hour later, they respond saying they have escalated you to Level II support. Another half hour later you hear from Level II support who decides they can't install a module for your site because the host is shared with many other customers and there is some kind of conflict - they offer to migrate you to a new machine with newer software (probably involving Level III support). In all probability, this will take a day or two and they will break something in the process (like email delivery) or perhaps introduce new configuration problems.
When you don't have control over your own server and you have even one question or sophisticated task you can instantly expect to lose between 2 hours and several days getting it resolved with hierarchical teams that have no personal knowledge of your application. And let me tell you, programmers do not like repetition and unnecessary inefficiency - we're the people who streamline and automate processes all day. Alternatively, if you had control over the server and the application, this process would be complete within 5 minutes.
We have seen this and similar scenarios play out dozens, if not hundreds of times. We have seen big box hosts destroy websites irreparably. We have seen big box hosts go out of business - locking all of their customers out of their account! We have seen big box hosts erroneously hold domain names hostage. In the end the crux of the matter is simple - outside hosts will always demand extensive (and unnecessary) time and effort that has to be accounted for in some fashion or another. Firefly's policy is to charge fixed fees for almost all projects because we don't like surprises. Offering an SLA to cover the costs of working with a third party host is a result of this philosophy.
