Making software deployable instead of configurable
Making software configurable can be a sign of a problem when developing custom software. By configurable, I’m referring to the process of making the configuration of a software system changeable. Especially, making the behavior of the system changeable by users and/or administrators in an attempt to anticipate future needs or to circumvent change management procedures.
Ways in which people make systems configurable
What ends up configurable can vary quite drastically. Starting with the more benign are things such as credentials to external systems, which historically made sense to make alterable. Other examples are locations of external resources, for example files and web services that an application depends upon.
These are changes an administrator makes to a production installation of a system. These are aspects that can change, albeit infrequently. For example, credentials could be compromised and need to be updated. Or, the location of a remote web service could change. Because these might change in the future, traditionally, they’re made alterable in live systems. The rationale is that’s the fastest way to update them should they need to be changed.
Feature toggles are another example of configurable aspects of systems. It often makes sense to develop new features, but keep them disabled until they’re complete and maybe even approved. Feature toggles can disable new functionality until a user or administrator enables the feature. But, feature toggles can actually be used without making them configurable in a live system, though most of the time they’re made configurable by default.
Many systems have activities, often referred to as tasks or jobs that need to be executed on a routine basis. Scheduling these tasks is an area where configuration starts to grow more complex, often unnecessarily. I’ve stumbled upon many cases where schedules were made dynamic, alterable in live systems. In the majority of these situations, the schedules never changed. In the few situations where the schedule changed, it changed maybe once a year. And the change had no degree of urgency.
Templates are a gray area where what may seem like simply a feature of software, is actually a more troublesome form of configuration. There are many systems that produce some form of human digestible output. Perhaps a statement, a customer report, or a legal confirmation.
Often prematurely, someone will propose a system for users to create templates to handle different contexts. For example, the confirmation of a purchase may need to include details and verbiage specific to a product. Making these details and verbiage configurable with templates may very well be a feature for a company that sells thousands of products that each require customized verbiage.
But, in many situations, there are only a handful of products. And of that handful, usually very few differences in verbiage. Using templates to handle this situation is a form of configuration that’s often wasteful. Especially when the templates end up being complex or containing considerable business logic.
Then there’s a level of configuration that’s often a telltale sign that you’re on a sinking ship. This happens when the system is created with the ability for users to write code in the live system to alter its behavior. This often manifests as a text box where users can literally type in code. Even worse, the language is often proprietary to the system.
When this gets carried away, you often find the capability to configure a system to handle and store new data structures. Or even worse, to map out new workflows. In these situations, part of the process of authoring the system has shifted from programmers to end users. Think about that for a minute.
Some boundaries
When I say configurable software is a problem, I’m talking about custom software that’s created and used by an organization for its own purposes. If software is sold or shared freely with others, there’s often good reason to make it configurable. Though even when software is created for others to use, it may benefit from many of the considerations that follow.
Why do we make software configurable?
If software isn’t configurable in live systems, then changing it means updating it. Updating it means rolling out a new version of the software. Historically, updating software has been a manual, extremely time consuming process. When the time required to update a system is greater than the tolerance for failure, then it makes sense to make some aspects configurable.
This tradeoff requires understanding an acceptable duration of failure. Unfortunately, the tendency is to believe we can’t accept any amount of failure, and that somehow the only way to do this is to make a system configurable. Never mind that changing the configuration still takes some amount of time, including some validation of the effect. And, never mind that modern techniques to update a system automatically can be faster than changing configuration in a live system.
Sometimes, configuration is added to bypass a slow update process. If an update process is manual and tedious, and if there’s no impetus to improve the update process, there’s a tendency to anticipate future needs and make the system configurable in such a way to theoretically handle them. Assumptions drive what aspects become configurable.
Sometimes, a firm understanding of the purpose of the software hasn’t been established. Making a system configurable, especially with the ability to inject code into text boxes seems like the way to justify the system. It can theoretically do anything, so it must be able to do something of value.
Along with a slow release process, there’s often a poor process to handle change management. It often involves pile of paperwork that nobody enjoys filling out. And, it’s often a slow process that requires the involvement of many different people. These obstacles incentivize strategies that make it easier to change system in a live environment to avoid the horrendous process to make changes through normal channels.
Occasionally, stakeholders perceive that programmers take forever to make changes. This perception arises for many reasons: manual update procedures, slow update processes, poor development planning and poor development organization to name a few. This perception becomes problematic when it surpasses the acceptable duration to adapt to change. For example, a business wants to roll out a new product, but they don’t think programmers will be able to update the system with the necessary verbiage in time for the product launch. In this situation, stakeholders may see configurable templates as the only viable alternative.
The problems configuration causes
Making software configurable comes with a cost. Even simple configurable aspects like feature toggles require injecting the appropriate configuration to turn features on and off when releasing the software. Configuration becomes another artifact that must be managed. Typically it’s stored in configuration files, but may be kept in a database or remote service. It becomes yet another aspect of the process to update the application. By complicating the release process, it further inhibits the ability to quickly update software, seemingly justifying itself.
More complex forms of configuration come with even greater costs. Things like templates and extensible programming languages in a live system. These are extremely time consuming and expensive to create. It’s always important to discuss what level of change is expected before investing in these advanced forms of configuration as a means to quickly adapt to future needs.
The assumption that configuration is the best way to adjust to unanticipated situations and future needs shouldn’t be taken for granted. Just because something is changeable in a live system, doesn’t mean that’s the fastest or most effective way to implement the change. In fact, bypassing the normal process to release updates to the application means it’s one of the most vulnerable ways to alter a system.
There are a host of checks and balances that software should go through before being updated. This is codified in a change management process. Software should be put through a consistent process to avoid defects. It should be consistently tested and verified. We always want to ensure the changes we’ve made don’t break existing, expected behavior. When configuration changes are made in a live environment, we’re bypassing all the sanity checks we would normally run.
We’re also bypassing many of the mechanisms that allow us to audit changes so we can quickly diagnose and recover from problems. Changes made to a live system are rarely captured like changes made through a normal update process. Configuration changes should be subject to the same level of accountability as normal changes for regulatory reasons. And yet, in many situations where regulation plays a role in software development procedures, people think they can bypass safeguards because changes are accomplished outside of the normal software development process and are somehow not subject to the same level of scrutiny.
When change management procedures are burdensome, the solution isn’t to bypass the procedures with configurable software. Instead, invest in ways to maintain safeguards without the hassle. Many of the best change management processes are fully automated. Automation affords the highest level of accountability and reliability in updating software. And, automation tends to expedite the process too. It’s a win-win.
In more advanced forms of configurability, especially extensible programming languages, users may introduce entirely new functionality. Because this functionality bypasses normal verification procedures, it can jeopardize existing, valuable functionality. Furthermore, users don’t have the tools to add verification specific to the functionality they are creating. They’re often left experimenting with new functionality in live systems. They have to debug their new functionality, and test it, with real systems that a business relies upon.
Users usually have no way to codify new functionality to ensure that future updates to the application maintain their new functionality. If they don’t communicate effectively with programmers, it’s very possible they’re using the configurability of the system in unanticipated ways that may not be something programmers are aware they need to support. And even worse, efforts may end up duplicated as programmers work on similar functionality in the future.
I strongly recommend considering who has the best tools to develop new functionality in a system. Users or programmers? Do users have development environments? Do users have testing tools? Do users know about good development practices? Do users know about good testing practices? Do users know how the system is supposed to work? Do they really know what the configurable aspects were meant to support? Will users really understand why the configuration exists? Will they really understand how to use it? Or, will programmers be left to help users create functionality in live systems that programmers themselves are much better equipped to do through a normal software development process?
Configuration also requires extensive testing to be added to the regular process to update the software. Developers must ensure different permutations of the configuration don’t fundamentally alter the behavior of the software. Because this configuration exists as external artifacts, expensive integration and end to end testing is often the only way to test the configuration. Programmers have a hard time fully testing systems without introducing permutations of configuration files. As such, configurable aspects are almost always left untested.
Because configuration is stored in files, it’s not uncommon for changes to these files to be lost. Someone may tweak the configuration of a live system and then fail to appropriately communicate that change. It’s typically a good idea to manage and appropriately version the contents of configuration files. Configuration files are normally loaded from a definitive source during updates to the system. If changes haven’t been propagated back to the definitive source, the next update will overwrite the changes. Ironically, when this happens, programmers tend to stop keeping a definitive source and instead manually update configuration files during updates. One can begin to imagine the problems that ensue.
It’s astounding how much of a modern software release process is consumed by complications from configuring the custom software itself. If it weren’t for the configurability, releases would be ridiculously simple, non-events. It’s the configurability that typically turns a 5 minute deploy into an hour long ordeal with programmers manually patching configuration files.
And perhaps the most fundamental problem with making systems configurable is that it bypasses having discussions and communicating about what the software should accomplish and why. When new functionality is implemented by users, software tends to head in directions that leave programmers and others unaware of what’s really important. When users can implement radical changes to a system, conversations about what’s valuable are often missing. New, marginally valuable functionality may steal the focus of worthwhile functionality.
In fact, discussions of making a system configurable are often the departure point of understanding what makes a system worthwhile. In many situations, asking “why does this need to be configurable?” uncovers a host of conversations that would otherwise be left unmentioned. Even simple questions like, why can’t we update the software to enable a new feature, why do we need a configuration file to turn on a new feature in the live system? Often, these conversations uncover how wasteful configuration is, and steer the conversation back toward valuable specifications.
When someone asks if something can be configurable, I always like to counter with the question: “Will it really be as simple as flipping a switch?” I often find, running through a few potential scenarios indicates that we can’t predict the future. And we’re better off saving the time we would’ve invested in configuration for the future when we have a new situation to address. Trying to anticipate future needs is almost always a sign of waste in software development.
What’s the alternative?
Configurable systems come with quite a daunting list of potential problems. Fortunately, there’s a simple alternative. In the process of discussing problems I couldn’t help but hint at the solution: make the system deployable. Implement a process to release changes in a consistent, efficient manner.
If the process to update the system is fast enough, the benefits of configurability vanish. Programmers will always have the best tools to develop and verify changes to software. Users are always ill equipped to make changes. Therefore, the only bottleneck is the ability to quickly release updates. Stop investing in configurability and start investing in updatability.
Not only will this pay dividends in adapting to change that was historically dealt with through configuration files, it will enable a more efficient process to develop normal changes to the system. Get in the habit of regularly releasing software. Invest in automating as much of the release process as possible. Invest in automated verification and testing of critical aspects of the system. Be able to update quickly, with confidence.
But it’s not just automation and confidence, it may also entail changing how development is organized. I mentioned configurability is often used to address poor development planning and organization. When development is done in large batches, the process tends to slow down. The ability to release is hampered by a back log of new functionality that needs to be verified. Releases are much more risky. As such, not much is invested in an efficient release process.
Instead of tackling big batches of work and releasing quarterly or semi-annual updates, tackle smaller batches on the scale of weeks up to a month. Release the changes at the end of the period and move on to the next set of changes. When something unanticipated happens, you won’t have as much baggage in the way to quickly switch gears and address it.
If you had the option between a five minute update process or configurable features, which would you chose? Imagine that even simple features like a hard coded feature toggle could be flipped and flow through all the normal checks and balances before affecting the live system. And, if something goes south, the update process terminates, instead of the live system.
The lack of configurability may seem like a lack of flexibility. That couldn’t be further from the truth. In fact, how much more flexible are you if you can make any change imaginable and release it within five minutes of completion? Instead of only the changes you anticipated necessary and placed into a configuration file.
If for some reason configurability still seems valuable, you can always add it as a last resort. But it should be a last resort, not the default, nor a best practice. Hard code things and focus on your update process.
Certainly, you can’t always control the update process, for example if you sell software that others install and maintain. In these situations, configurability is still a valuable tool. But, in these situations, I would still consider the consequences of the advanced forms of configurability, versus a process to quickly release updates. Look at how many software packages these days are auto updating. I can’t help but wonder how many configuration headaches this helps avoid.
Consider this
Think about the various software packages you rely on, and the various packages you may be involved in creating. Write down some of the ways in which the systems are configurable. Think about how frequently the configuration is changed and in what circumstances.
If you had the choice between a highly configurable system and a rapidly updateable system, which would you chose?