Picture a packaging supplies distributor. Its dealers place orders through a web application. On Friday at six in the evening a new discount screen goes live; the developer has tried the screen and it works. On Monday morning accounting calls: none of the orders that came in over the weekend became invoices. The new field added for the discount renamed a column that the invoice export expected. Nobody looked at the invoice side, because the change "was only about discounts".
A company that lives through a morning like this usually reaches one of two conclusions: either it trusts its developer less, or it decides to release updates as rarely as possible. The second is more dangerous. The rarer updates become, the bigger each one gets, and a bigger update breaks more places at once.
Much of the problem comes not from the code that was written, but from the way the code reaches users. The seven habits below apply both to companies with their own team and to those whose software is built by an outside team.
1. Release changes small and often
In a big update every three months, dozens of changes go live on the same day. When something breaks, finding which one broke it takes hours, and rolling back means rolling all of them back at once.
In small releases once or twice a week, each one carries only a few changes. When a problem appears the list of suspects is short, and only that small piece needs to be rolled back. Users also see little change each time, which makes it easier to adjust.
This requires splitting work into small pieces. "The new discount module" is not one job; it can be split into separate releases: first the data field, then the calculation, then the screen, and last, opening it to dealers.
2. Make the test environment resemble production
What runs on a developer's computer may not run in production. There may be a different database version, a missing setting, another file path, or a different version of an external service you depend on.
Set up a separate environment where you try a change before release, and run it with the same software versions, the same settings and the same integrations as production. Its data can be a copy of production with personal details masked; on an empty database everything looks fast and trouble-free.
Flows that touch other systems, such as the invoice export, should be tried end to end in this environment too. The distributor's problem above would have been caught exactly here.
3. Put the release time on a calendar
A change released on Friday evening is, if it goes wrong, a problem nobody looks at all weekend. Month-end closing days, campaign launches and the busiest week of the season are bad times in the same way.
Set a release window for your company: for example Tuesday to Thursday, before noon, during hours when the developer and a user who knows the work can both be reached. Urgent security fixes are outside this rule; everything else can wait for the calendar.
4. Write down the way back before you release
"If something happens we'll roll it back" is not a plan if how to roll back is not written down. Before every release, answer these three questions:
- What steps take you back to the previous version, and how long will they take?
- Who will make that call?
- What happens to data entered with the new version when you go back to the old one?
In most modern release setups, returning to the previous version takes a few minutes. The hard part is usually the data, which is why the next point deserves its own heading.
5. Make database changes in two steps
Rolling back code is easy; rolling back a deleted column or a renamed table is not. The problem in the distributor example was a renamed column too.
The safer path is to split the change into two releases. In the first release the new field is added, the old field stays where it is, and for a while the application writes to both. Once you are sure everything uses the new field, and that reports and integrations have moved to it as well, the old field is removed in a separate release.
This approach takes a little more work, but if the first release causes a problem, the code is rolled back and the data is still in a shape the old version understands.
6. Plan the half hour after the release
A release does not end when the change goes live. Within the first half hour, someone who knows the work should go through the system's most important flows from a short list, in order: signing in with a dealer account, creating an order, the order passing to invoicing, opening a report.
This list should take no longer than ten minutes and be the same for every release. It is not a long test plan, just the five or six steps you would describe as "if these break, the company stops". The list is more accurate if it is prepared by someone who uses the system every day rather than by the developer.
During the same half hour, the error logs should be checked as well. An error shows up in the logs before a user reports it.
7. Tell users what changed
When a button moves on a screen, the user may take it for a bug. Every change nobody announced comes back as a call to the support line.
Write a short release note with every release: what changed, who it affects, and whether they need to do anything new. Two or three sentences are enough. Updates that change the daily work of dealers or the field team should be announced in advance, with a screenshot if possible.
Over time release notes serve another purpose: when a problem appears, the answer to "when did this change?" is there.
In short
- Split changes into small pieces and release them often.
- Use a test environment with the same versions, settings and integrations as production.
- Do not release on Friday evenings, at month end or on busy days.
- Write down the steps for going back, how long it takes and who decides, in advance.
- Split database changes into two separate releases.
- After a release, run a short, fixed check list and look at the error logs.
- Add a release note of two or three sentences to every release.
None of these habits needs a large investment; most become routine once they are discussed and written down. What you gain is a company that is not afraid of updates. And a company that is not afraid of updates keeps its software current and secure.
In our software maintenance and support work, we plan and release small change requests within the same routine and add a short release note to every version.
Our work on this: Custom Software Development
Planning a project like this?
Plan it in 10 steps