technical: Is Continuous Delivery a Good Idea?
As a consultant, most of my projects involve investigating problems or issues, fixing difficult problems, or providing other advice. However last year I was on a project to upgrade a client's Websphere MQ version (well, IBM MQ as it's called today). This was a nice change, going back to refresh my knowledge issues in upgrading z/OS software. And many of the issues are still the same: SMP/E installations, PSP buckets, change management, phasing in the new software.
However there was one choice to be made that surprised me: continuous delivery. Did I want to implement continuous delivery?
Continuous Delivery 101
Continuous Delivery (CD) is a new direction from IBM for the delivery of new function. Well, not exactly new, DB2, CICS and MQ have offered CD since around 2016.
Traditionally, new function was introduced with new releases. So if you wanted new DB2 functionality, you would wait for the next release. DB2 releases in the past have been around 3 years apart, so you could wait for some time for new function.
CD introduces new functionality via the service stream. Traditionally, PTFs would be regularly added to your software to fix problems found after a product had been released. CD opens the door to adding new functionality via this service stream. So you can get new functionality earlier. Sounds good.
The problem is reliability. Mainframes are reliable, and it's a brave mainframe manager that will do things to endanger that. All you'd need is your mainframe to crash a couple of times in a short period for your job to be in danger. So the idea of introducing new functionality via a service stream can seem to be reckless: new functionality can introduce new problems.
Choose the New Functions You Want
IBM has attempted to address some of the concerns about new functionality by providing features to selectively enable or disable new functions. Websphere MQ does this with 'mini-releases.' The Websphere MQ 9.0.4 release adding new RESTful APIs is an example.
In the CSQ6SYSP, the OPMODE parameter can be used to determine how MQ will work. For example, you could code OPMODE=(COMPAT,800). This would maintain MQ V8 compatibility. So any new function introduced after MQ V8 would not be enabled. To enable these functions, OPMODE=(NEWFUNC,900) enables new functionality for MQ V9. However any CD features would not be enabled.
OPMODE=(NEWFUNC,904) enables CD for MQ 9.0.4. Any new functionality introduced will be available.
DB2 does things a little differently. DB2 introduces 'Function Levels'. For example, function level V11R1 includes all functionality introduced with the first release of DB2 V11. Function level V12R1M500 introduces all functionality with DB2 V12, plus additional functionality in modification '500'.
The DB2 ACTIVATE FUNCTIONLEVEL command activates each new function level for the subsystem. Individual applications can set application levels when binding with the APPLCOMPAT BIND option. For dynamic SQL, the SET CURRENT APPLICATION COMPATIBILITY command does the same thing.
CICS is different again. From CICS TS 5.4, feature toggles can be used to switch features on and off. So adding the line
to the feature toggle configuration file will enable the BMS3270 Intrusion Detection Service.
Is This Really a New Idea?
Well, no. Small programming enhancements (SPEs) have been around for years. For example, here's a link from IBM listing new function APARs for CICS regions going back to CICS TS 4.1.
In general, these SPEs have been small enhancements. And there have been no parameters that can selectively turn them on or off. Rather, systems programmers would choose to apply a PTF if they wanted the new feature.
So, Is This a Good Idea?
It depends. Let's look at our MQ install. In this case, we chose not to implement CD. With this install, we upgraded around 70 queue managers shared by over 150 applications. It took 12 months from SMP/E installing the software on our sandbox system to upgrading the last production queue manager. Our number one priority: no outages. We gradually migrated the software into our environments over time. We meticulously created installation procedures to minimize the risks. We allowed code to operate in development environments before migrating to QA. And we tested in QA for at least three months before slowly migrating into production queue managers: a couple at a time. We also had many meetings with application groups so they knew what was happening, what new functions would become available, and what they had to do.
And we still found errors. Some required IBM APARs, others we found workarounds. And despite all this, there were production problems.
With most of my clients, the overall priority is to minimize downtime and risk. The pressure to implement new functions is far less. So, CD may not be a good idea, as it introduces additional risk.
Anyone looking to implement DevOps would disagree. This champions the idea of regularly introducing new function, so CD would seem to fit in well. We talk about the challenges of implementing DevOps in a traditional mainframe site in our previous article. There will also be areas where this new function is required ASAP. How can this work?
How It Can Work
So, we have two sides: one requiring stability and resilience, the other wanting new function quickly. To me, CD allows this, giving sites the power to set their own development cycle. So rather than a 3-year upgrade cycle for DB2, a site could choose a 12-month cycle: add new maintenance and function every 12 months.
Even better, this can be different for different subsystems. So we may upgrade our traditional IMS connected MQ queue managers that need stability every two years. However our queue managers for web and RESTful users requiring the latest features ASAP could be upgraded every 3-6 months.
CD can seem like madness. But with thought and planning, it gives sites more control over their upgrade cycles, and when they are ready to install new functionality.