technical: Working With Large Quantities of SMF Records

In the last edition, I wrote an article that described how I would start a CPU reduction project by producing a table of the highest CPU consuming programs during a peak 1-2 hour period. This table is created by processing SMF Type 30 interval records, which introduces a problem. One SMF Type 30 record is produced by every address space every recording interval (usually 15-30 minutes). So there are a lot of these records to work through. What's worse, these records are amongst the millions of SMF records produced constantly by z/OS. So this processing will take time and CPU seconds. Lots of it.

This is a very common problem when processing any SMF records. From producing security related reports to DASD performance or DB2 capacity planning. When you work with SMF records, you will almost always need to manage and process very large quantities of SMF records. So let's look at some tricks to make our lives easier.

Processing Raw SMF Records

z/OS continually writes SMF records to in-memory SMF buffers. These buffers are then regularly written to either VSAM datasets or log streams by z/OS (depending on how the systems programmer has set it up). Every site will have regular jobs that empty these VSAM datasets/log streams, archiving the SMF data to disk or (more likely) tape. So it sounds easy doesn't it? To get our SMF records, we just scan through these archived SMF records.

And products like Merrill's MXG, IBMs TDSz and CA-MICS help us out. They provide tools and facilities to map each SMF record, and extract the information we need. So if I'm at site with MXG, I can extract the SMF Type 30 records I need using a job like:

//SAS      EXEC SAS,
// CONFIG='MXG.SOURCLIB(CONFIG)'
//SASLOG   DD SYSOUT=*
//SASLIST  DD SYSOUT=*
//WORK     DD LRECL=27648,BLKSIZE=27648,SPACE=(CYL,(150,100))
//SOURCLIB DD DSN=MXG.SOURCLIB,DISP=SHR
//LIBRARY  DD DSN=MXG.FORMATS,DISP=SHR
//SMF      DD DISP=SHR,DSN=SMF.ARCHIVE
//SYSIN    DD *
%INCLUDE SOURCLIB(TYPE30);

DATA JOBS1;
   SET TYPE30_V;
   WHERE (TYPETASK='JOB' OR TYPETASK='STC') AND CPUTM > 1 AND
   HOUR(SMFTIME) > 9 AND HOUR(SMFTIME) < 11;

RUN;

Those familiar with SAS will see that I'm working through an SMF archive (the dataset SMF.ARCHIVE - either on tape or disk), and putting a list of all SMF Type 30 records produced between 9am and 10am into the JOBS1 SAS file.

When working with SMF records, the secret is to work with as few as possible. And this is what I'm doing in this job. If you look more closely, you can see that I'm only listing jobs and started tasks (no USS processes, or APPC address spaces), and skipping every SMF Type 30 record with less than one second CPU time (There will be many records with 0 CPU if the task has not been dispatched. I'm not interested in those).

However I'm doing this after the %INCLUDE SOURCLIB(TYPE30). This means that MXG has already moved SMF Type 30 records into SAS files before my JOBS1 data step starts. To limit the records that MXG will process, you can use the MXG macro facilities.

Now, most sites I've seen archive their SMF records into daily tape datasets. So if you were to run this job against the SMF archive for one day, you'll be working through every SMF record for that day. Or in other words, you'll be waiting a couple of hours, with a corresponding CPU usage bill. There are a few ways to reduce the CPU overhead, but it's never going to be cheap. An excellent reason to run these jobs out of peak-periods: usually overnight.

If this is an ad-hoc job, then this is no problem. However if this job is to run regularly, then it's going to be expensive. A classic example of this is your regular IBM SCRT job that process SMF Type 70 and 89 records.

Limiting Raw SMF Records

If a site is going to regularly process specific SMF records, it makes sense to create a separate SMF archive holding only those SMF record types. This way any job only has to work through the SMF record types they need, rather than scanning through them all. This is also important if you need to archive some SMF records types for long periods for compliance (like RACF Type 80). IBM helps us out here. The IBM utility that archives SMF records: IFASMFDP (for SMF VSAM) and IFASMFDL (for log streams) can be setup to do just this. For example, look at the following JCL:

//STEP1    EXEC PGM=IFASMFDP
//SYSPRINT DD  SYSOUT=* 
//DUMPIN   DD  DISP=SHR,DSN=SYS1.MAN1 
//SMFOUT   DD  DSN=DUMP.SMF.ALL,DISP=MOD
//SCRT     DD  DSN=DUMP.SMF.SCRT,DISP=MOD
//SEC      DD  DSN=DUMP.SMF.SECURITY,DISP=MOD
//SYSIN    DD  *  
 INDD(MAN,OPTIONS(ALL))
 OUTDD(SMFOUT,TYPE(0:255))
 OUTDD(SCRT,TYPE(70,89))
 OUTDD(SEC,TYPE(80))

This job dumps all SMF records into the dataset DUMP.SMF.ALL. Additionally, SMF records needed for the IBM SCRT (Type 70 and 89) go into DUMP.SMF.SCRT, and RACF records (Type 80) go into DUMP.SMF.SECURITY. The good news is the additional work writing to additional SMF dump datasets doesn't incur a large CPU overhead. There are also a few ways to further reduce CPU overhead of SMF dump jobs.

You can go further and split up SMF records by application. So CICS1 data would go into one SMF archive; CICSB into another. However this takes some assembler programming skills to create exits for IFASMFDP/IFASMFDL.

Summarizing SMF

Another options is to avoid accessing the raw SMF records altogether. This can be done by creating databases holding summarised data. For example, summarising SMF Type 70 records by hour, or by day. So instead of working through every record, you can just access the summarised results. This approach is done by most sites when handling performance and capacity planning records, such as SMF Type 70/72, 110 (CICS) and 101 (DB2). These databases can be stored in DB2 (for TDSz), SAS files (for MXG and CA-MICS), or any other database system you choose. TDS, MXG and CA-MICS all include features to produce these summarised databases, which are often called Performance Databases, or PDBs.

These PDBs are great, and usually satisfy most needs. However I find that I'm regularly going back to the raw SMF records. Usually this is because I need information from a non-summarised record (such as Type 30 for CPU usage, Type 80 for security, or Type 14/15 for a dataset open), or I need more detail.

Processing SMF On a Workstation

Even after getting my Type 30 records, the chances are that I'll have tens of thousands of them to process. In the old days, I would sit down and code SAS statements to produce the tables and graphs that I want. And this is still an option. But tools like Microsoft Excel produce results much faster. The trick is to get the data to them.

Let's take my Type 30 extract job above. What I tend to do is create a CSV file with the Type 30 records. I can do this in SAS by adding this code to the bottom of the job:

ODS _ALL_ CLOSE;
ODS CSV FILE=CSV RS=NONE;

PROC PRINT;
   VAR JOB DATE SYSTEM
   CPUSRBTM CPUTCBTM CPUTM PROGRAM CPUIFETM CPUZIETM;

RUN;

ODS CSV   CLOSE;

This will take the SMF records from the SAS file and output the figures I want to a CSV file. Transfer this to my PC, open with Excel, and I'm away. For smaller sites, this works great.

However I find for larger sites, there is still too much data to process with Excel. In these cases, I fire up the free MySQL database I have installed on my laptop. I'll import the CSV created above to MySQL using SQL like:

LOAD DATA LOCAL INFILE 'C:/SMF30V.csv' INTO TABLE SMF30 FIELDS 
TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n'
(job, datetime, system, cpu_srb, cpu_tcb, cpu_total, program, 
cpu_zaap_eligible, cpu_ziip_eligible);

I can then create subsets of the data using SQL statements, and import them into Excel for processing.

If this all seems too hard, then the Australian based Black Hill Software can help. Their software product Easy SMF for z/OS does all of this for you. Download the raw SMF, and Easy SMF will process it, and produces graphs or tables. You can also import this data into Microsoft Excel for your own processing. The disadvantage is that Easy SMF only supports a subset of SMF records.

Conclusion

If you start working with SMF records, the chances are that you will be handling large quantities of records. The good news is that there are ways to ease the pain of processing these records.

David Stephens

LongEx Quarterly is a quarterly eZine produced by Longpela Expertise. It provides Mainframe articles for management and technical experts. It is published every November, February, May and August.

The opinions in this article are solely those of the author, and do not necessarily represent the opinions of any other person or organisation. All trademarks, trade names, service marks and logos referenced in these articles belong to their respective companies.

Although Longpela Expertise may be paid by organisations reprinting our articles, all articles are independent. Longpela Expertise has not been paid money by any vendor or company to write any articles appearing in our e-zine.