Of sailors and webhosts ...

– Brett Barron 22 September 2009

I'm a perfectionist. I surround myself with perfectionists. My business partners, my wife, even my five-year-old daughter strive for perfection. No one has feared more a potential blight on his "permanent record" than I. But, alas, even with conscientious, deliberate carefulness the vessel guided by my own hand does, from time to time, find a way to run aground.

Kaleidoscope provides hosting services to our Dialogs customers. We do so for one primary reason: we truly don't trust anyone else to do a better job. We're meticulous, security-minded folk that have been around long enough to know how important it is to take the appropriate precautions and - importantly - how to take the appropriate precautions. We employ seasoned administrators with decades of technical know-how, own high quality, top-manufacturer equipment in a state-of-the-art, high capacity, high security data center. We keep spare critical devices for rapid replacement. We backup data with obsession. We insist on redundancy: redundant power, redundant data paths, redundant HVAC, redundant backups. We nightly off-site our backups. We schedule time to brainstorm how things can go wrong and what to do about it. The steps we take to ensure a quality hosting experience for our customers is quite extensive. And it's a good thing we do this, because technology in a word, breaks.

One of the servers in the Kaleidoscope data center suffered downtime this week. The server began to experience periodic episodes of non-responsiveness. The specifics were odd, really, but in the end, we restored several of our hosting customers to a hot-spare server waiting for the opportunity to get off the bench and actually do something. We had a protocol; we had current data; we had a stand-by server; we had all hands on deck to resolve the issue. Still, several of our customers experienced multiple episodes of downtime spanning from one to several minutes each before the entire issue was fully diagnosed and resolved. As a perfectionist, knowing that my hosting customers were experiencing even a few minutes of downtime was excruciating.

It is the way with complex systems that they are complex. And complex things are prone to failure. We've all come to experience high levels of reliability in the technical implements of our world. What begins as appreciation becomes expectation. I can easily find myself irritated when my cell phone connection with someone half-way across the globe begins to falter for a few seconds, expecting absolute perfection out of a string of technologies I couldn't hope to count - let alone fully understand. The hosting industry requires high levels of redundancy and backup for the same reason that ships carry life-boats: things do go wrong.

A long-time friend (and excellent sailor) once told me, while at the wheel of his beautiful 36-foot Alden ketch, "There are three kinds of sailors, Brett: those who have run aground, those about to run aground ... and liars."

I thought about those words yesterday as I worked alongside my team to restore stability to our affected web hosting customers. Website up-time is important. Critical. Business depend on it. Our hosting environment, equipment, software and protocols are designed to maximize up-time. We exposed a bit of imperfection this week I'll concede; but we did not expose failure: we did resolve the issue with no customer data loss. As a Google apps customer I know that even the largest organizations with the most inexhaustible budgets and resources also are susceptible to occasional downtime (Google apps were down for a few hours earlier this month).

Like Google I assume, my team learns from every technical issue we encounter. We identify previously unconsidered weak points and address them. My team is hard at work devising adjustments to our setup that will better protect from this specific type of issue should it reoccur. We've refined our protocols so that we will be even more efficient in restoring service. It's what experienced professionals do. In my mind, the most important differentiator of web hosting options is not the amount of storage or bandwidth for a particular price or any of the other statistics you typically see, but the level of experience and professionalism of the actual people at the wheel. If you pick the right people, you can depend on them to make the right decisions about storage space, bandwidth, and redundancies of technologies you really need not fully understand. And, critically, you can depend on them to safely pull you off the rocks when you do go aground.

Kaleidoscope has hosted websites for our customers for over 14 years and the occasional, non-scheduled downtime - no more than 0.0001% down time in total - still strikes me to the core like a bad mark on my permanent record. But bear in mind, any organization that touts a history of no down-time has either not been around long enough or will eventually be humbled. Or, I suppose, they could be liars ....