Drupal 8 Configuration Management in the real world - workflow and gotchas

Panacea, or disaster? Drupal 8 Configuration Management was supposed to solve all our woes when it came to dealing with deploying configuration. In many ways it's a vast improvement, but in some ways it has almost made matters worse.

Configuration Management has made it possible to store all Drupal configuration in code. This is a huge win, of course. But there are several gotchas that make it challenging to work with:

• If you don't track individual changes, a configuration export/import is an all-or-nothing proposition. If you have people making changes in multiple environments, merging these changes takes manual effort and coordination.
• It's easy to clobber changes made in one environment, when you synchronize changes.
• It takes extra effort to maintain different settings in different environments.
• Some configuration relies upon content entities that may only be present in one environment (for example, custom block placement, panels). This configuration does not deploy successfully, and can break sites if you're not careful.

The extra complexity of working with Drupal configurations might prove to be one more factor raising the bar for amateur web developers, making it useful primarily if you have other tools available to help manage it all, and a strong process that enforces best practices.

Gotchas

We've had a couple of issues catch us by surprise and affect sites in production, but for the most part, we've managed to avoid disaster. Here are some scenarios we see as challenges around configuration management. Most of these we've caught before they break stuff, but a couple have been "learning experiences".

Client makes a configuration change on production

Many of the larger enterprise users of Drupal may actually remove admin interfaces and lock down configuration so administrators can't make quick changes on production. That's too cumbersome for many of our clients -- one of the advantages of Drupal is how easy it is for non-programmers to make quick changes. We have clients who edit views headers or footers, or add blocks, or update panels, with the day-to-day operations of their organizations.

Then, when we go to roll out things we work on, if our dev database has gotten out of sync with production, when we deploy and import the changed configuration, we clobber all the client's work, gone, gone gone.

That does not make for a happy client.

Content does not deploy with new functionality

On a dev site, you might build out screens for new functionality, and drop in panes of content to support it. In Drupal 8, every block you create, or any other entity for that matter, gets created with a UUID. If you create a block and show it on a panel, it exports the panel configuration to code just fine -- but with the UUID of the block you created.

As it stands now, there's no way to deploy that block to production. If you try to create the content blocks on production first, they have different UUIDs, and so panels won't find them.

This one caught us on one site -- we deployed some changes to a panels page, and suddenly was greeted on production with "Block not found" errors visible right on the home page of a site!

Development modules and settings enabled on production

Deploying a new module to production has become quite easy in Drupal 8 -- turn it on on a development site, configure it, export the config, and then roll out and import the config on production, and presto, it's live. However, this also means that unless you take some extra measures, ALL of your configuration between production and development gets synchronized. Which means production sites end up with modules useful for development deployed, enabled, and configured -- consuming resources, potentially opening up attack vectors, allowing clients to muck around with stuff they should be leaving alone.

So how do we deal with these gotchas?

Strategies for managing configuration

We've largely lifted our process from managing dozens of Drupal 6 and 7 sites using features, and applied it to Drupal 8. The same basic conflicts and issues arise there -- the key difference is that with Drupal 8, ALL configuration gets exported/imported, so you need to be far more disciplined in following a process.

Here are some of our strategies.

Keep all configuration in sync with code

The challenge here is that nothing in Drupal complains loudly if you change any configuration in a site. You have to go looking for these changes.

We run a nightly check on all of our production and dev (integration) sites. If this check finds any configuration that is out of sync with the code of the currently checked out branch, it nags us in the project's chat room.

This check alerts us when a client has made a configuration change on production that we need to bring back to dev and merge into the next release. Our developers learn to commit all of their work, including configuration changes, each night, and keep the dev configurations clean.

Configuration that is stored in code can be tracked, much more easily merged, reverted, or managed -- but this takes discipline.

Create content entities on production, and pull down database

This can lead to a multi-step deployment process. For example, to set up some memberships on a new site, first we have to deploy code to support the new membership entity types and entities. Then we need to create some of the content that supports those screens on production, hidden from public view. Then we need to copy down the database to our development server, and then finally we can put the pieces together and do the final deployment manually.

It sounds to me like a lot of shops just build everything up on development and copy up the database -- the only time we allow this is for an entirely new site, otherwise our cardinal rule is the production database never gets overwritten, and all production data gets sanitized when we copy it down to development.

Use local configuration overrides, and exclusions

The D8 settings.php file comes with a commented out include to load a "settings.local.yml" file meant for environment-specific configuration. The great thing about the configuration management system is that you can override any configuration in the settings file, hard-coding it to whatever value you want in use without affecting the actual configuration that will get deployed. This is the most powerful, and most used way to manage environment-specific configurations.

Drush also supports excluding modules from the configuration when exporting or importing configuration.

Our setup

To support using configuration management effectively, you need to do a bit of setup on each site up front. After installing the site, we do the following:

1. Edit the settings.php file to include settings.local.php
2. Move the $database array and other environment-specific configuration into the settings.local.php file
3. Set the $config_directories['sync'] variable to a directory inside sites/default but outside of sites/default/files, so that we can manage the site configuration in git
4. Set .gitignore to track settings.php but ignore settings.local.php
5. Set all the production settings for twig debugging, aggregation, etc using "drupal site:mode prod" and commit the services.yml file
6. Set .gitignore to ignore the services.yml file -- this keeps the file in git but ignores any changes to it, because development settings change this file
7. Add the list of modules we want to designate as "development" to an appropriate drushrc file's command-specific "skip-modules" list
8. Export the site configuration and commit.

Supporting workflows

When you want to actually use the configuration management system, you end up having several different workflows to support different scenarios.

Handle nightly check warnings

Our Matrix bot will alert us each night to sites that have configurations that are out of sync.

If it's the production site, and we haven't deployed there in the past day, we can be pretty certain that it's a change from a customer (or possibly an attacker! Side benefit of a nightly check -- detect unexpected changes!) We export the production configuration, evaluate the nature of the changes, and if they are expected, commit and push back up to the master branch. Then, after verifying the development site is clean, merge those into the development site and import the configuration there.

If it's the dev site, we essentially do the same thing -- export the configuration, review for expected/desired changes, and commit.

If it's both sites, we export/commit both sites before attempting to merge -- this generally prevents us from clobbering any work done on development and makes the merges as simple as possible.

The longer a site goes out of sync, the harder it is to merge changes like these as people forget why a change might be made. We've found that adding these nightly checks to be a huge benefit!

Deploy configuration

Our process is pretty similar to what is written elsewhere. We export the configuration on dev, commit it, deploy the code, and import it on the target environment. Easy peasy.

In our case, we have a few extra steps that are mostly automated -- all having to do with quality checks. First we commit on a develop branch and push to our central repo. This triggers our Behat test runs, which let us know what passes/fails when testing the development site with all our "Behavior Driven Design" (BDD) tests.

Then we check out the "release" branch and merge in the develop code, and push. This triggers a deployment to our stage environment, and kicks off our visual regression testing, which highlights every pixel difference betwen stage and production.

Finally, when all that looks good, we ask our bot to deploy to production. It grabs a snapshot of the current production database, tags the release, pushes the code to the production environment, and applies the configuration.

On the next nightly check, our bot automatically does a fresh database copy from production to stage, to clean the decks and get ready for the next release.

Deploy a complex feature that depends upon content

We treat our stage site as an ephemeral testing location, routinely blown away and recreated from a fresh, sanitized copy of the production database, and the code/configuration to be rolled out in the next release. Our bot also maintains release notes, where we can add any manual steps needed for a successful deployment or to complete the job, as well as a list of commits in the release.

So when something needs to be deployed that will include content entities with unique UUIDs, it becomes a multi-release process.

To do this, we stage a release as usual with all the supporting code necessary to build up the content. Then after verifying that the changes are not "leaking" to people who shouldn't see them yet, we roll out to production. We create all the necessary supporting content, and then copy the database back down to stage, and then to development. Then we build out the remaining functionality and roll another release.

CMI is great, but watch out for dragons

All in all, we really like the new configuration management in Drupal 8, but we've found it essential to use other tools to manage the actual process, and make sure we're not destroying work along the way. Your team needs to understand how configuration management works, what it's capable of, where its shortcomings are, and how to merge and resolve conflicts. And if you don't enforce a "clean environment" policy, you will end up eventually spending more time sorting out what has changed where, in a nasty place where the wrong move will burn...