topic Re: Getting data from Mainframe system?? in Splunk Search

Getting data from Mainframe system??

shri_27 — Mon, 20 May 2013 12:13:40 GMT

Hi all,
How to get data from Mainframe systems onto Splunk??

Re: Getting data from Mainframe system??

treinke — Mon, 20 May 2013 13:17:37 GMT

Can you syslog messages from the mainframe?

Re: Getting data from Mainframe system??

cramasta — Mon, 20 May 2013 14:06:56 GMT

I have looked into this in the past. You will more than likely need to use some 3rd party software to create metrics that Splunk will collect. There are currently no Splunk Apps/built in functionality (a side from using syslog) for doing this. This company seemed to have a solution that would plug into Splunk http://www.infosecinc.com/meas.php

Re: Getting data from Mainframe system??

dwaddle — Mon, 20 May 2013 14:30:22 GMT

There is no forwarder code for mainframe systems today. You could always submit a feature request asking for it. If you don't, then chances are it will never happen. (Splunk product management does not look at Splunkbase questions around a particular topic as a proxy for actual feature requests)

Also, you need to be explicit about what you are looking for. There are at least 3 common "mainframe" operating systems, and programs compiled for one WILL NOT work on the other. You have:

z/OS (MVS)
z/VM
Linux (s390 architecture)

All of these are "mainframe" operating systems, each with their own APIs and idiosyncracies. When 99% of all people think "mainframe" they are thinking of z/OS, but the alternatives exist. (Functionally speaking, Linux/s390 would be the least difficult for Splunk to port a forwarder to - the other two could be much worse)

Re: Getting data from Mainframe system??

a212830 — Fri, 21 Jun 2013 19:17:25 GMT

I'm pretty certain some forwarders for mainframes are coming in Splunk 6. You should check with your account rep.

Re: Getting data from Mainframe system??

sideview — Fri, 21 Jun 2013 23:35:33 GMT

I did a little exploration with a third party about getting performance metrics off of Nonstop/Tandem hardware. We wrote a program to collect the metrics and write them out using a vaguely sensible format to a socket. Then it was just a simple TCP input in the Splunk server and some extractions. It was quite a successful prototype and proof-of-concept although we didn't end up releasing the product.

Re: Getting data from Mainframe system??

BobFake — Mon, 19 Aug 2013 15:50:39 GMT

Hi - check out the Mainframe Event Acquisition System (MEAS) which will send mainframe data to Splunk. Events such as security activity, database accesses, CICS transaction activity, dataset access, FTP, TCPIP, RMF, SMP/E and more. You have the ability to filter so that you can send only what you really want to Splunk for further alerting and reporting. www.meas-info.com.

Re: Getting data from Mainframe system??

gfuente — Fri, 11 Oct 2013 13:31:05 GMT

Hello,

Splunk 6 has been released, and there is no UF for mainframe. Do you know if it is going to be released later this year?

Thanks

Re: Getting data from Mainframe system??

watsm10 — Fri, 11 Oct 2013 13:51:20 GMT

We FTP our Mainframe logs every 5 minutes to a text file on a heavy forwarder. The logs are forwarded on from there and load balanced across our indexers.

Re: Getting data from Mainframe system??

BobFake — Mon, 24 Feb 2014 16:27:30 GMT

Hi - if you need to get mainframe data (security, database, CICS, FTP, TCPIP, master console messages and much more), please see meas-info.com. Our Mainframe Event Acquisition System (MEAS)product will allow you to monitor, filter and forward - in real time - any/all events from the mainframe that you would like to see in Splunk. It take roughly 1/2 day to install and no IPL necessary. Please give us a call if you need any more information.

Re: Getting data from Mainframe system??

jreda — Wed, 04 Feb 2015 21:29:00 GMT

Ironstream from Syncsort can do all of this work for you. It will handle all of the issues related to SYSLOG, z/OS SMF records, log4j and flat files. It deals with the compression, the triplets, the binary data and converts the data from EBCDIC to ASCII. It does this very efficiently, even offloading a lot of the work to a zIIP engine in order to keep the MSU cost of this work to an absolute minimum. This is all done in real time to give you the best data latency possible while not impacting the existing workload on your system.

Re: Getting data from Mainframe system??

Graham_Hanningt — Fri, 11 Nov 2016 07:09:56 GMT

IBM Transaction Analysis Workbench for z/OS ("Workbench") can forward a wide range of logs from various z/OS subsystems to Splunk.

Recent enhancements to Workbench include features specifically for streaming logs in JSON Lines format off z/OS to a Splunk TCP data input.

Some key points about log forwarding with Workbench:

Workbench does not require special agent software to collect log data. Instead, Workbench uses the logs and other historical data that each subsystem generates during normal transaction processing and system operations.
Forward the fields you want, from the records you want, when you want:
- Workbench does not limit you to forwarding a fixed subset of fields from each record type. You can select as many or as few fields as you want.
- You can filter records for forwarding based on combinations of field values.
- Workbench log forwarding is not real-time. To forward logs with Workbench, you run batch jobs on z/OS. You decide when to run those jobs, and how often.
Simple, self-contained JCL. Streaming a log as JSON Lines to a Splunk TCP data input involves about a dozen lines of self-contained JCL.

Example JCL to forward CICS monitoring facility (CMF) performance class (SMF type 110) records from a dumped SMF data set:

//S1       EXEC PGM=FUWBATCH
//STEPLIB  DD   DISP=SHR,DSN=FUW.SFUWLINK
//LOGIN    DD   DISP=SHR,DSN=SMF.MVS1(-1)
//SYSPRINT DD   SYSOUT=*
//SYSIN    DD   *
STREAM NAME(SPLUNK) TRANSPORT(TCP) HOST(mysplunk) PORT(6789) +
       LINES FLAT OMITNULL NOTITLE FIELDCASE(LOWER) ASCII LF ZONE
JSON STREAM(SPLUNK) CODE(CMF)
FIELDS(
* Insert list of CMF fields you want to forward
)

Related Splunk app on GitHub (currently just one dashboard, for CICS performance).

Some relevant topics in the Workbench (version 1.3) product documentation:

Getting started ► Overview ► Features ► Log forwarding

Forwarding logs to analytics platforms ► Splunk

Extracting logs to CSV or JSON ► Streaming JSON Lines over TCP

Disclosure: I am the author of the Workbench product documentation.

Re: Getting data from Mainframe system??

bklutz — Thu, 01 Dec 2016 20:56:04 GMT

IBM has a product: IBM Common Data Provider for z Systems v1.1.
It provides for near real-time SMF and various log streaming or in batch.
check it out at: ibm.biz/CDPzInfo

Re: Getting data from Mainframe system??

tldenney — Fri, 23 Jun 2017 18:51:53 GMT

IBM Common Data Provider for z Systems can forward mainframe data to Splunk in near real-time.

It supports a wide variety of data including 140 data sources and 100+ SMF record types, and it can stream structured and unstructured data or use batch mode to collect data. IBM Common Data Provider for z Systems also has advanced filtering capabilities including RegEx and time filtering.

You can also learn more about IBM Common Data Provider directly on Splunkbase.

Re: Getting data from Mainframe system??

mattwhitbourne — Mon, 26 Jun 2017 18:48:21 GMT

I downvoted this post because not really true, there are a variety of forwarding options on the market like ibm's common data provider... although appreciate this post might have been correct when initially written!

Re: Getting data from Mainframe system??

DomenicoDAlteri — Tue, 27 Jun 2017 07:38:51 GMT

It is based on 30+ years of experienced IBM engineers, to learn more take a look here : ibm.biz/CDPzInfo

Re: Getting data from Mainframe system??

ehall0328 — Fri, 15 Sep 2017 15:48:21 GMT

With Syncsort Ironstream, you can collect log data from SMF, RMF, Syslog and other z/OS sources, and forward that data in real time to the Splunk® Enterprise analytics platform. That gives you visibility into your z/OS environment. Ironstream also integrates with Splunk’s Enterprise Security and IT Service Intelligence applications. This goes beyond IT operational analytics to give you a firmer grasp of potential security threats in your z/OS environment. It ensures that your critical business services are being delivered on time.
For more information on Ironstream: http://www.syncsort.com/en/Products/Mainframe/Ironstream

Re: Getting data from Mainframe system??

tldenney — Thu, 28 Sep 2017 18:41:12 GMT

Splunk and IBM recently hosted a joint webinar on how they are partnering to help customers gain access to mainframe machine data more quickly for real-time investigation, analytics, and pattern analysis. You can watch the replay here: https://onlinexperiences.com/scripts/Server.nxp?LASCmd=AI:4;F:QS!10100&ShowKey=43016&AffiliateData=Flyer

Re: Getting data from Mainframe system??

CurtisGannaway — Tue, 27 Jul 2021 21:32:14 GMT

Hi @Anonymous

Did you get your question answered?

If you need to get mainframe data (security, database, CICS, FTP, TCPIP, master console messages and much more), please see dgtechllc.com/meas. Our Mainframe Event Acquisition System (MEAS) product will allow you to monitor, filter and forward - in real time - any/all events from the mainframe that you would like to see in Splunk. It takes roughly 1/2 day to install and no IPL necessary. Please give us a call if you need anymore information.

Thanks!