technical: Monitoring by Sampling

Often when we think about monitors we think about software like IBM Omegamon XE, BMC MAINVIEW and ASG TMON. These software products do things like regularly hop around storage and control blocks to find out what's happening right now. They'll often also insert 'hooks' or other magic to get information on current performance and workloads. They may even have tracing features to give you more details about what's happening, and some historical features to find out not only what's happening now, but what was happening in the past.

However there's another type of monitor: a sampling monitor. And the chances are that you have one.

The First Sampling Monitor

In the 1970s, a company called Programart developed Strobe. If this sounds familiar, then it should. Compuware acquired Programart in 1999, and continue to market Strobe. Strobe is a sampling based monitor. This means that it will 'sample' an address space, meaning it will take lots of snapshots, to see what's happening. How often it samples is set when you define a sampling session. As a rough guide, I like to sample around 1,000-1,500 times per minute when analysing something for more than 5 minutes. So if we sample 1,000 times per minute, Strobe will check every 0.06 seconds to see what an address space is doing. Compuware isn't publishing how it does this, but we can make a few educated guesses. For a start, Strobe is checking to see if any task in the address space is currently executing on a processor. If it is, it will find out from the PSW the address, and from this (in most cases), the module.

So we can get a report that looks something like this:

MODULE    SECTION    LINE  PROCEDURE/FUNCTION         STARTING  CPU TIME %
 NAME     NAME       NUM   NAME                      LOCATION  SOLO  TOTAL

.COMMONX                        EXTENDED COMMON AREA 1823FB48  5.34   5.71
.XES     IXLR1LKR               LOCK W/RECORD STRUCT           5.09   5.50
.SUPERVS IGC0013{               SUPERVISOR SERVICES            2.38   2.42
.XMEMORY .XMEMORX               EXTENDED CROSS MEM             2.05   2.26
.CICS    DFHAIP        DFHEIP   EXEC(CMD-LEVEL)INT             2.05   2.05
.SVC     SVC 011                TIME                           1.93   1.93
.LELIB   CEEPLPKA      CEEV#GTS GET HEAP STORAGE               1.81   1.85
.COBLIB  IGZCPAC                COBPACK  GENERAL               1.81   1.85
.DB2     DSNXGRDS      DSNXEPM  PGM MGR                        1.68   1.81
.DB2     DSNK2DM                IDX MGR                        1.77   1.81

OK, but what does this all mean? What it tell us is where the CPU is being burned - in which modules. For example, we can see that the TIME SVC (SVC11) is consuming around 2% of the total CPU, and Language Environment GET HEAP almost the same. From this, we can target the CPU users, and start to tune them. Whenever I'm in a project looking to reduce CPU, this report is an excellent first step. Strobe will even go into more detail. So you can find the DB2 DBRM, the offset in a COBOL module, or the CICS transaction consuming the CPU. Nice huh?

Competition for Strobe

In 2004, Macro4 announced a new product as a direct competitor to Compuware Strobe: Freeze Frame. Shortly after in 2005, IBM announced an almost identical product: IBM Application Performance Analyzer. Like Compuware Strobe, these products are sampling monitors, producing reports on CPU usage for an address space. Reports break down CPU usage by module, and like Strobe, can go down further into the DBRM, Transaction and offset.

However all three products go further. They analyse wait times - why an address space was waiting. FreezeFrame can product a report similar to the following.

                                      -------- Mean Time in Seconds -----
Name    NTxns Description        Error Execution+Suspend + Delay = Service
CSMI   25464  Mirror transactio   0.6%   0.000    0.165    0.046     0.213
> DFHMIRS     CICS Program               0.000    0.161    0.044     0.206
  > CICS      System Services            0.000    0.161    0.039     0.201
    > IRLINK  Wait on InterRegion Link   0.000    0.081    0.000     0.081
              LINKMM1
    > FCIOWAIT Wait on VSAM File I/O     0.000    0.064    0.000     0.064
               FILE001 
    > CICSDly CICS Dispatch Delay        0.000    0.000    0.039     0.039
    > ZCIOWAIT Wait on Terminal I/O      0.000    0.015    0.000     0.015
    > FCXCSUSP FILE001 FILE001           0.000    0.000    0.000     0.000
    > LG_FORCE Wait on Log Buffer        0.000    0.000    0.000     0.000
               being Flushed
    > LG_DEFER Wait on Log Buffer        0.000    0.000    0.000     0.000
               being Flushed
    > LGWRITE Wait on Writing Log        0.000    0.000    0.000     0.000
              Record

So you can see that for a CICS region, FreezeFrame has broken down the service time by transaction. So our CSMI transaction has most of its service time waiting on the InterRegion Link LINKMM1, with VSAM File I/O on file FILE001 coming a close second.

Other Information

Our three sampling product go further than this. They include reports on things as varied as:

• DASD and file I/O values, counts and performance
• SQL activity
• Websphere MQ statistics

And more. A gold mine of tuning information.

Cost vs Accuracy

All this information comes at a cost. For a start, a sampling session must be manually defined. You define the address space (job, region or started task), the monitoring options to enable, the number of samples to take, and how long to run. The number of samples and length of sampling run are important. Let's look at them:

You want the length of the sample to be long enough for a representative sample if monitoring online or recurring workloads. For example, I like a good 30 minute sampling period when monitoring a CICS region. For batch, the chances are that you want to measure the entire step you're looking at. All products have a flag that can be set to monitor until step-end. As an aside, they also all offer scheduling controls so you can setup a sampling session today that will run tomorrow, or next week, or every time a batch job runs.

The number of samples is also important. You don't want this too high, or you'll burn a lot of CPU monitoring the address space. If you have too many samples, the monitoring may also get in the way of the address space you're monitoring, skewing the results. On the other hand, you want enough samples to get a good look. If you only get 10 samples where a batch job is using CPU, the CPU reports are useless.

All three products include a report that is something like this:

| Total samples      45,000         | Duration           29 min 59.96 sec|
| Sampling rate      25.00 per sec  | Report dataspace   505.81MB        |
| CPU/WAIT samples   36,545         | Sample dataspace   863.95MB        |
| TCB samples        45,000         | Meas significance  81.21%          |
| CPU queued samples 8,455          |                                    |

In this example, we took 45,000 samples over a 30 minute period (well, 29 minutes, 59.96 seconds). During this period, we obtained 36,545 samples where the address space was executing, or waiting. This is good. However if this values was something like 50, you don't have enough samples.

If I'm trying to perform a very short sample (say, for a job step running for 15 seconds), I'll increase the number of samples to get a good number of CPU/WAIT samples. If I can't get a few hundred samples, I won't trust the output.

System z CPU Measurement Facility

In their z10 Systems z processors, IBM added the System z CPU Measurement Facility (CPU MF for short). This allows you to perform your own sampling without any of the above products. The HIS started task in z/OS manages this facility: operator commands start, stop and define sampling session parameters. This started tasks also collects these samples, and writes them to a z/OS UNIX dataset for analysis. CPU MF also provides some counter functionality and a new SMF Type 113 record. But we'll concentrate on the sampling side.

The CPU MF has the advantage over the previous products in that it is a hardware feature, so the sampling won't affect the results obtained. It will also sample everything that executes - all address spaces. It also uses less CPU than the sampling monitors.

The disadvantages start with the fact that this facility cannot be limited to one address space. You sample the entire z/OS system, or nothing. This will provide a huge amount of information, so it's rare to exceed 10 minutes in a sampling session. So sampling that 30 minute batch job step won't help. Further, this facility dumps information into a z/OS HFS. You need to do the analysis. Phoenix Software International market the zHISR product that can format and analyse this information.

CPU MF also won't provide much of the DB2, CICS Transaction, Websphere MQ, file I/O and other information provided by the above sampling monitors.

Summary

David Stephens

LongEx Quarterly is a quarterly eZine produced by Longpela Expertise. It provides Mainframe articles for management and technical experts. It is published every November, February, May and August.

The opinions in this article are solely those of the author, and do not necessarily represent the opinions of any other person or organisation. All trademarks, trade names, service marks and logos referenced in these articles belong to their respective companies.

Although Longpela Expertise may be paid by organisations reprinting our articles, all articles are independent. Longpela Expertise has not been paid money by any vendor or company to write any articles appearing in our e-zine.