management: Open Source and Mainframes: A Good Idea, or Craziness?
The discussion about the security and resilience/reliability of open source software has been going on for years. But for mainframes, open source hasn't been that big a thing. But with the release of the mainframe open source software Zowe, it's a good time to consider this question again, in the context of mainframe systems.
So is open source OK for mainframes, or is it just asking for trouble?
Open Source and z/OS
Although open source software has never been a big thing for z/OS, the idea is not new. Veteran systems programmers will remember the early days of the CBT Tapes, where you'd write away to get a tape with freeware goodies. Many sites today still use some of these goodies, from SHOWMVS that displays a wealth of z/OS information, to TAPEMAP that does exactly that. In fact, cbttape.org is still alive and well with a wealth of free, open source tools and software.
But normally, these are tools restricted to systems programmers and other administration staff. They're rarely used for critical applications or systems. So, if there are resilience or security issues, it's not a big deal.
More recently, other open source software has begun to creep in. From programming languages like PHP and Python, data mining like Apache Spark, MQ replacements like Apache ActiveMQ, and SSL (OpenSSL). Today, there's a lot of open source software available. But the reality is that open source software still isn't mainstream for mainframes.
Open Source Builds
Most of the open source software I do see on z/OS sites today has been 'pre-built' by a vendor. For example, if we want OpenSSL, we use the package available from Rocket Software. IBM also have a lot of such builds, from their HTTP Server (Apache) to Apache Spark for z/OS.
Using pre-built packages from trusted vendors avoids the work of loading, compiling and installing the software themselves. But perhaps more importantly, we rely on these vendors to create a stable, secure version. And this makes sense.
But most of the available open source software is not packaged in this way. Rather, we go to places like GitHub, and follow any instructions. Zowe is an example of this.
The Red Hat State of Enterprise Open Source survey of 2019 stated that 25% of respondents believed that "access to enterprise-level support" was a reason for choosing open source software. These respondents are unlikely to be working on z/OS, but this raises an interesting point.
Mainframe software has traditionally offered better support than non-mainframe software. Although it could be argued that some mainframe software vendors that don't fit this statement, the question of open source vs vendor support is valid.
On non-mainframe systems such as Linux, there is a large 'pool' of talent contributing to open source software. There is also a culture of engaging in open source software. If an administrator finds a problem, they report it (and may try to fix it). So, for popular non-mainframe open source software, reported problems are likely to be resolved quickly.
For mainframe open source, this will be different. Most mainframe sites I visit have shaved their technical head-count, and few have free time to spend on open source projects. Further, many mainframe sites are being 'de-skilled', and are losing staff with deep technical knowledge that may be required to debug open source problems.
Many of those supporting open source software may not be working in a related environment. For example, anyone with a Linux system can contribute to Linux open source projects, even if they're not working in a Linux related job. Mainframes are different. Anyone contributing will need a working z/OS system. These can't be found in your local mall.
To put some numbers to this, the Zowe project had 19 contributors on GitHub at the time of writing this article. OpenSSL had 411, Django had 1752.
So, there's a risk that mainframe open-source support will not meet our mainframe requirements.
Underlying Open Source
So not only are we relying on the Zowe open source project, we're also relying on the node.js open source project. Node.js isn't traditionally mainframe software. So, we're relying on someone in the open source community providing the z/OS-specific support.
IBM first published their System Integrity Statement in 1973, explaining their commitment to securing z/OS. All software vendors will perform security checking and will modify supported software if security issues are found.
Open Source software may not have the vendor backing, but if it is a 'live' project, then it has support that is as good, if not better. For example, the OpenSSL Heartbleed exposure was resolved in six days. What's more, there are more people looking for these exposures.
The flip side is that any exposure found is out in the open. In the case of Heartbleed, there existed an exposure that was public knowledge, and remained until organisations installed the fix. For z/OS systems, it is not unusual to wait for months to install new fixes into production environments.
Documentation is often overlooked, but a critical part of any software. With most vendor products, time will have been taken to create quality documentation. However open source software projects rarely have full-time technical writers looking at the documentation. In fact, the 2017 Github Open Source Survey found "incomplete or confusing documentation" was a complaint for 93% of responders. 60% stated that they rarely or never contributed to documentation.
The concept of open source software is exciting, and it has been proven to be a very successful model for developing and supporting software in non-mainframe environments, particularly UNIX. Some of the open source option for z/OS are equally exciting, including Zowe.
However, it remains to be seen if this open source model will meet the higher availability and security needs of mainframe systems. With a smaller pool of contributors, any site using open source software for critical mainframe applications will need sufficient local technical knowledge to provide some of their own support.
The good news is that with open source software, you have all the source code you need to do this. The bad news is that finding local staff with the time and technical knowledge will be more difficult.