Managing change and risk with Drupal Hosting

If there's one thing that's constant in the web world, it's change. We've seen huge change over the years: personal pages, forums, RSS, blogs, YouTube, MySpace, Twitter, Facebook, Pinterest, spam, ubiquitous performance attacks, Internet Explorer 10, Firefox going from 0.9 to 22, smart phones, tablets, and more.

Within business, we've seen web sites evolve from static brochures to active content, e-commerce, client management portals, project management, reporting dashboards, customer service management, vendor management, and more.

What's next? Who knows. What we do know is that Drupal will most likely be one of the first places you can quickly implement whatever "next" is -- and this is one of the main reasons for using Drupal in the first place.

Accommodating change is one area Drupal excels, but if you're not careful with planning, you can easily be headed for a train wreck. Having your hosting set up to accommodate regular updates is critical to keeping your site up and running smoothly.

Deployment questions to answer up front

We've developed a list of questions to help identify appropriate hosting for a given project. Some of these have a huge impact on budget -- a reasonable budget for Drupal hosting can be anywhere from ten dollars per month to several thousand dollars a month. The vast majority of our clients are spending between $150 and $600 a month on hosting and maintenance. Spend less than that and it becomes very difficult to keep up with change. If uptime and prevention of all data loss are critical to your business, and you'll need to spend a lot more.

So here are some questions we will ask and data we will collect, if you come to us wanting us to take over or manage your site:

1. Where is the site hosted?
2. Is it a dedicated server (physical or virtual), a shared server, or multiple servers?
3. If it's a new site, who is responsible for provisioning, and what date will it be ready?
4. What operating system/release is the host server?
5. Do we have SSH access?
6. Do we have root access?
7. Do we have access to git on production?
8. Do we have access to drush on production?
9. Is a continuous integration (CI) service such as Jenkins running?
10. Is Freelock to provide server monitoring and maintenance?
11. Is Freelock to provide Drupal site maintenance?
12. The database is running [on the web server] [on RDS/another server]
13. The site [is] [is not] using a Content Delivery Network (CDN): __________
14. Apache Solr [is] [is not] in use. Solr is configured [on web server] [on another server: _______]
15. Varnish [is] [is not] in use. Varnish is configured [on web server] [on another server: _______]
16. Memcached [is] [is not] in use. Memcached is configured [on web server] [using Amazon ElastiCache] [on another server: _______]
17. Primary backup, nightly/weekly snapshots of entire server, is available using this account: ___________
18. Secondary backup, historical/offsite backups of all data, is available here: ____________
19. Database slaves [are] [are not] running, and can be checked here: _________
20. DNS is managed [by Freelock] [by customer] [by another vendor]
21. Worst Case time to recover from secondary backup with catastrophic failure of host and primary backup: [weeks] [days] [hours] [minutes] [seconds]
22. Maximum acceptable lost data: [weeks] [days] [hours] [minutes]
23. The site [needs] [does not need] policies and processes in place for Payment Card Industry Data Security Standards (PCI-DSS) at level [SAQ-A] [SAQ-C] [SAQ-D] and/or other data policies: [HIPAA] [other privacy regulation]

    There's a lot packed into these questions. Together, they address several key aspects you'll need to consider when planning for hosting and deployment:

• Managing change, updating code
• Handling traffic spikes, accommodating higher loads
• Mitigating data loss and service outage risks associated with making a system your organization relies upon
• Properly securing your data and complying with necessary regulations

  If you're not concerned with any of those, congratulations! You'll be just fine in a $10/month shared hosting plan. If you are, read on.

Managing change, updating code

Our base Drupal repository that we begin all projects with contains nearly 1/2 million lines of code. The simple fact is, there are thousands of bugs in that code. And we hit them all the time.

Sometimes we fix them, or add a work-around. Sometimes other people in the Drupal community fixes them.

But a bigger part of this discussion is adding functionality to your site -- keeping your site agile, anticipating and helping you leverage your changing business environment. So there are a couple crucial questions here:

• How do you get code up to the production site quickly and reliably, with minimal downtime?
• What do you do if new changes break something unexpected on your production site?

  For the first, we start with code management tools including git, drush, Drupal features, upgrade scripts, and a manual deployment list of instructions that we test on a staging copy of the site before trying a large deployment on production. If there's a lot of changes to be deployed, you need to plan out that deployment and make sure you know ahead of time where the risks are and exactly what needs to be done. On active sites with major upgrades, we run a test upgrade first to see how it goes. And we're very close to rolling out "Continuous Integration," automating the deployment process with equally automated rollback processes.

  For the second, it's really going to depend on the nature of what is broken, and how long it has been in production before it was discovered. We tag and back up the production site before deploying any code, so one option is reverting to that backup. If that results in unacceptable data loss, though, it might be better to take the site offline if necessary and work through the issues to get it working correctly. It's a choice between data integrity and availability.

  With new sites, these are not really that critical at first. But if your site is successful, you're eventually going to need to face these issues and consider how to handle them -- far too often we see sites with no plans for this, and upgrades can be devastating if not properly planned.

  Adding proper code management and deployment planning moves you from the $10 month range to the $100 - $200/month range.

Handling traffic, higher loads, DOS attacks

Drupal scales to handle some of the largest, busiest sites on the Internet, but no busy site can handle that kind of traffic on a single server. And if you have to withstand a Denial Of Service attack, you need enough server capacity to deal with at least a moderate amount of traffic so a server admin can go in and block the attack. These kinds of things are becoming much more common, unfortunately. The plus side is that tuning a site to withstand attack also allows you to handle a lot more traffic. The downside is cost.

Eric, our Dev-Ops lead, is preparing a blog post to describe some options for expanding the capacity of the site, but overall here are the most common tactics:

• Upgrade to a virtual server with more RAM, SSD drives
• Add spam-fighting solutions at the application level
• Add spam-fighting solutions at the server/firewall level
• Deploy more advanced caching solutions, such as Varnish and Memcached, use intelligent caching headers
• Deploy Nginx/FPM to handle static content much quicker and serve more capacity
• Move static content to a CDN, tune web servers for dynamic content
• Move the database to a dedicated server(s)
• Add a load balancer and multiple front end servers, configure scripts to auto-provision as needed

  Aside from the spam-fighting stuff, all of these measures increase your monthly operation costs. An Amazon "Small" instance will run you about $55/month. At Digital Ocean, you can get a comparable 2GB "droplet" for $20/month. Increasing to an 8GB droplet takes you to $80/month.

  But the real performance gains come from splitting out the parts to different servers -- which means you might need to now pay for 3, 4, or 5 servers instead of one. That does start multiplying costs, but it allows you to handle exponentially more traffic.

  Digital Ocean (Affiliate link) is our new favorite for hosting sites that are not soon going to exceed the capacity of a single server. Amazon AWS still holds the lead for multiple servers, especially with the built-in CDN functionality, ElastiCache, RDS, and load balancers, just in terms of being able to easily add these with minimal configuration, when you need them. And we also do a lot with Rackspace Cloud Servers -- we think there are really good reasons in terms of ownership/risk reduction going with an open source platform like OpenStack (Rackspace is one of the major contributors to this project, and uses it for their own infrastructure).

Preventing data loss and handling service failures

Outages happen, even for the largest, most sophisticated companies. Murphy was an optimist. We don't think you should put all your eggs in one basket, or one host.

If your web site and infrastructure is entirely hosted with a single company, you are entirely at their mercy. A billing snafu, a technical glitch, a hardware failure, a DMCA takedown, or many other things might happen that could potentially take your account offline -- and do you know for a fact that your host is in good fiscal condition?

Cheap insurance. That's what we're talking about here. The best insurance, when it comes to IT is a sound backup practice that takes into account all of the risks you face. Vendor risk being one of them.

We don't think a nightly snapshot of your server, at the same host, is good enough. You absolutely need your data copied to either your own servers, or a third party service, on a schedule that makes sure that data is fresh enough that you're willing to live with losing everything else.

That's one part of this concern -- the other is, how long will you be down if there's a major problem with your server? Even if you have backups, it can take time to restore from them, and during that time your site may be partially or fully down. If the site is critical to your business, you'll need to plan for redundant systems at multiple vendors, with up-to-the minute replication of data and plans for cutting over in any kind of service interruption event.

This level of high availability can really drive up the cost. For most of our clients, there's a clear point of diminishing returns here -- most of our clients do not find that level of high availability to come at too steep a cost, and so they accept some risk of an outage that takes hours or days to recover from -- these events are, these days, pretty rare.

If you're comfortable with losing up to a couple days of data, and can live with an outage of up to a business day, getting decent backups and putting together a recovery plan is a one-time cost with very little ongoing cost. If you need "High Availability" in terms of either having fresher backups or less downtime (or both) your costs will go up pretty dramatically.

Securing your data

After the other concerns we've already addressed, security almost seems like an afterthought. Even so, there are several things to think about that might also affect your hosting choices as well as the cost implications:

• Does the server ever see credit card data, or sensitive personal data such as social security numbers, health conditions, etc?
• Does the server ever store this information?
• Who has access to code files, who might be able to modify them?
• Who can access the database directly?
• Is there any unknown person who has login access to the server?
• Is plain-text FTP ever used to connect to the server?
• Is there sensitive data in the backups, and are the backups secured?
• What password policies are in place?
• Is two-factor authentication required to access the server?
• What update policies and practices are being performed?
• How are intrusions/problems detected?
• Who disables access when a trusted user leaves each organization?

  Most hosting companies that run their own data centers (e.g. not resellers) have a pretty good handle on these issues, and can give you good answers. If you have particularly sensitive needs, you'll want to work with hosts that specialize in these kind of things -- we have a couple clients at Fiber Cloud for this reason.

If you're failing to plan, you're planning to fail

We've covered a lot of ground here. What I'm describing is coming to be known as "Dev-Ops" in the Drupal world -- a role that focuses in keeping sites running smoothly, and this is an area Freelock has always excelled.

One of the key reasons we like Drupal so much is how it is really designed and built to accommodate and manage change. That doesn't mean you get this for free -- it takes very careful planning and execution to manage any change effectively, and Drupal certainly has its shortcomings in this department. But it also has some huge strengths, if you take some time to plan ahead and anticipate the changes you want to leverage with your site.

If you have some risky changes you need to make to your Drupal site, or if you want to partner with a shop with an eye towards experimenting with different functionality to arrive at a huge win, drop us a line! Meanwhile, if you disagree with something I wrote, or have something to add, please leave a comment below!