LongEx Mainframe Quarterly - August 2015
Often when we think about monitors we think about software like IBM Omegamon XE, BMC MAINVIEW and ASG TMON. These software products do things like regularly hop around storage and control blocks to find out what's happening right now. They'll often also insert 'hooks' or other magic to get information on current performance and workloads. They may even have tracing features to give you more details about what's happening, and some historical features to find out not only what's happening now, but what was happening in the past. However there's another type of monitor: a sampling monitor. And the chances are that you have one. The First Sampling MonitorIn the 1970s, a company called Programart developed Strobe. If this sounds familiar, then it should. Compuware acquired Programart in 1999, and continue to market Strobe. Strobe is a sampling based monitor. This means that it will 'sample' an address space, meaning it will take lots of snapshots, to see what's happening. How often it samples is set when you define a sampling session. As a rough guide, I like to sample around 1,000-1,500 times per minute when analysing something for more than 5 minutes. So if we sample 1,000 times per minute, Strobe will check every 0.06 seconds to see what an address space is doing. Compuware isn't publishing how it does this, but we can make a few educated guesses. For a start, Strobe is checking to see if any task in the address space is currently executing on a processor. If it is, it will find out from the PSW the address, and from this (in most cases), the module. So we can get a report that looks something like this: MODULE SECTION LINE PROCEDURE/FUNCTION STARTING CPU TIME %
NAME NAME NUM NAME LOCATION SOLO TOTAL
.COMMONX EXTENDED COMMON AREA 1823FB48 5.34 5.71
.XES IXLR1LKR LOCK W/RECORD STRUCT 5.09 5.50
.SUPERVS IGC0013{ SUPERVISOR SERVICES 2.38 2.42
.XMEMORY .XMEMORX EXTENDED CROSS MEM 2.05 2.26
.CICS DFHAIP DFHEIP EXEC(CMD-LEVEL)INT 2.05 2.05
.SVC SVC 011 TIME 1.93 1.93
.LELIB CEEPLPKA CEEV#GTS GET HEAP STORAGE 1.81 1.85
.COBLIB IGZCPAC COBPACK GENERAL 1.81 1.85
.DB2 DSNXGRDS DSNXEPM PGM MGR 1.68 1.81
.DB2 DSNK2DM IDX MGR 1.77 1.81
OK, but what does this all mean? What it tell us is where the CPU is being burned - in which modules. For example, we can see that the TIME SVC (SVC11) is consuming around 2% of the total CPU, and Language Environment GET HEAP almost the same. From this, we can target the CPU users, and start to tune them. Whenever I'm in a project looking to reduce CPU, this report is an excellent first step. Strobe will even go into more detail. So you can find the DB2 DBRM, the offset in a COBOL module, or the CICS transaction consuming the CPU. Nice huh? Competition for StrobeIn 2004, Macro4 announced a new product as a direct competitor to Compuware Strobe: Freeze Frame. Shortly after in 2005, IBM announced an almost identical product: IBM Application Performance Analyzer. Like Compuware Strobe, these products are sampling monitors, producing reports on CPU usage for an address space. Reports break down CPU usage by module, and like Strobe, can go down further into the DBRM, Transaction and offset. However all three products go further. They analyse wait times - why an address space was waiting. FreezeFrame can product a report similar to the following. -------- Mean Time in Seconds -----
Name NTxns Description Error Execution+Suspend + Delay = Service
CSMI 25464 Mirror transactio 0.6% 0.000 0.165 0.046 0.213
> DFHMIRS CICS Program 0.000 0.161 0.044 0.206
> CICS System Services 0.000 0.161 0.039 0.201
> IRLINK Wait on InterRegion Link 0.000 0.081 0.000 0.081
LINKMM1
> FCIOWAIT Wait on VSAM File I/O 0.000 0.064 0.000 0.064
FILE001
> CICSDly CICS Dispatch Delay 0.000 0.000 0.039 0.039
> ZCIOWAIT Wait on Terminal I/O 0.000 0.015 0.000 0.015
> FCXCSUSP FILE001 FILE001 0.000 0.000 0.000 0.000
> LG_FORCE Wait on Log Buffer 0.000 0.000 0.000 0.000
being Flushed
> LG_DEFER Wait on Log Buffer 0.000 0.000 0.000 0.000
being Flushed
> LGWRITE Wait on Writing Log 0.000 0.000 0.000 0.000
Record
So you can see that for a CICS region, FreezeFrame has broken down the service time by transaction. So our CSMI transaction has most of its service time waiting on the InterRegion Link LINKMM1, with VSAM File I/O on file FILE001 coming a close second. Other InformationOur three sampling product go further than this. They include reports on things as varied as:
And more. A gold mine of tuning information. Cost vs AccuracyAll this information comes at a cost. For a start, a sampling session must be manually defined. You define the address space (job, region or started task), the monitoring options to enable, the number of samples to take, and how long to run. The number of samples and length of sampling run are important. Let's look at them: You want the length of the sample to be long enough for a representative sample if monitoring online or recurring workloads. For example, I like a good 30 minute sampling period when monitoring a CICS region. For batch, the chances are that you want to measure the entire step you're looking at. All products have a flag that can be set to monitor until step-end. As an aside, they also all offer scheduling controls so you can setup a sampling session today that will run tomorrow, or next week, or every time a batch job runs. The number of samples is also important. You don't want this too high, or you'll burn a lot of CPU monitoring the address space. If you have too many samples, the monitoring may also get in the way of the address space you're monitoring, skewing the results. On the other hand, you want enough samples to get a good look. If you only get 10 samples where a batch job is using CPU, the CPU reports are useless. All three products include a report that is something like this: | Total samples 45,000 | Duration 29 min 59.96 sec| | Sampling rate 25.00 per sec | Report dataspace 505.81MB | | CPU/WAIT samples 36,545 | Sample dataspace 863.95MB | | TCB samples 45,000 | Meas significance 81.21% | | CPU queued samples 8,455 | | In this example, we took 45,000 samples over a 30 minute period (well, 29 minutes, 59.96 seconds). During this period, we obtained 36,545 samples where the address space was executing, or waiting. This is good. However if this values was something like 50, you don't have enough samples. If I'm trying to perform a very short sample (say, for a job step running for 15 seconds), I'll increase the number of samples to get a good number of CPU/WAIT samples. If I can't get a few hundred samples, I won't trust the output. System z CPU Measurement FacilityIn their z10 Systems z processors, IBM added the System z CPU Measurement Facility (CPU MF for short). This allows you to perform your own sampling without any of the above products. The HIS started task in z/OS manages this facility: operator commands start, stop and define sampling session parameters. This started tasks also collects these samples, and writes them to a z/OS UNIX dataset for analysis. CPU MF also provides some counter functionality and a new SMF Type 113 record. But we'll concentrate on the sampling side. The CPU MF has the advantage over the previous products in that it is a hardware feature, so the sampling won't affect the results obtained. It will also sample everything that executes - all address spaces. It also uses less CPU than the sampling monitors. The disadvantages start with the fact that this facility cannot be limited to one address space. You sample the entire z/OS system, or nothing. This will provide a huge amount of information, so it's rare to exceed 10 minutes in a sampling session. So sampling that 30 minute batch job step won't help. Further, this facility dumps information into a z/OS HFS. You need to do the analysis. Phoenix Software International market the zHISR product that can format and analyse this information. CPU MF also won't provide much of the DB2, CICS Transaction, Websphere MQ, file I/O and other information provided by the above sampling monitors. SummarySampling monitors are excellent tools for analysing the performance of batch and online applications and systems, providing information on the top CPU consumers, and why things are waiting. The CPU MF facility is a free alternative. However it provides different information that is harder to analyse, and does not provide many of the add-on benefits of the other sampling monitors. |
