Getting Data In

Can Splunk input IBM SMF records?

CZ1900Splunker
New Member

Can IBM SMF records be input to Splunk from z/OS? I am interested in indexing RACF data specifically.

Tags (3)
0 Karma

racfra2
New Member

Many thanks for the info. Interesting tool. We have different needs to cover joblogs(job/Step termination within batch jobs), syslog,RACF, data from user exits ... data from any file etc. to b sent to any kind of syslog servers. In many cases using sort exits is the smartest way for us, as a user can filter the data as needed.

0 Karma

racfra2
New Member

A clever and free way is using a SORT EXIT calling EZASMI to send anything to any syslog server.
Use it for console/syslog messages, RACF IRRADU00/IRRDBU00. The sort has the advantage that you can filter on anything.
Why paying big money to the big corporations for simple programs?

Easy to send jobs step information to Splunk etc.
<1> JCT scanned : MOBI USERID: IBMUSER READER: 2018-07-10 07:23:56:22 JOBSTART: 07:25:17:83 SMFID: APEX

PAL$TCP4-10 JOBNAME:MAXP001C JOB04459 STEP:COMPLINK ASM PGM:ASMA90 CODE: 0004

PAL$TCP4-10 JOBNAME:MAXP001C JOB04459 STEP:COMPLINK LKED PGM:IEWL CODE: 0000

0 Karma

jeastman
Path Finder

If you are just interested in SYSLOG data only, Syncsort's Ironstream has a "starter edition" that is FREE and only sends SYSLOG data.

http://www.syncsort.com/en/testdrive/ironstream-starter-edition

0 Karma

tldenney
Path Finder

IBM Common Data Provider for z Systems (CDPz) is the best option for sending SMF records to Splunk.

CDPz can send a wide variety of data including 140 data sources and 100+ SMF record types. More specifically, CDPz can support the following:

• SMF records
• SYSLOG (IBM z/OS System Log and USS SyslogD)
• JOBLOGs
• Application logs (IBM CICS Transaction Server logs and IBM WebSphere Application Server logs)

CDPz also has advanced filtering capabilities including RegEx and time filtering that can be set up using the built-in web configuration tool shown below.

alt text

More information on IBM Common Data Provider for z Systems can be found directly on Splunkbase.

tldenney
Path Finder

The following Splunk Blog outlines how Splunk and IBM are partnering to help customers integrate IBM Z (Mainframe) Data and Insights into Splunk software:

https://www.splunk.com/blog/2017/08/22/insane-in-the-mainframe-splunk-and-ibm-partner-to-provide-end...

0 Karma

raymondleroux
New Member

Hi how can I use ftp through splunk ?

0 Karma

jeastman
Path Finder

What data are you trying to get into Splunk? Do you mean FTP from a mainframe?

I believe you would simply FTP the file to the Splunk indexer or even just a machine with a universal forwarder on it. Then you can input the data from the file into Splunk through the manual interface via the browser or you can monitor the files in Splunk to upload when changes occur to that file.

More doc in data input from files here:
http://docs.splunk.com/Documentation/Splunk/6.5.0/Data/Monitorfilesanddirectories

If you need real time log data from the mainframe, as mentioned above, Ironstream is a good solution that will take care of data mapping and transformation and get that data over in real-time to Splunk:

http://www.syncsort.com/en/Products/Mainframe/Ironstream

0 Karma

jfeitosa
Path Finder

Here I am using IBM zSecure that is already installed on Mainframe and exporting via FTP to Splunk. So with the .txt file should do the parsing for regex.

But Ironstream can do this more easily.
http://www.syncsort.com/en/Products/Mainframe/Ironstream

jreda
Explorer

Ironstream from Syncsort can do all of this work for you. It will handle all of the issues related to z/OS SMF records. It deals with the compression, the triplets, the binary data and converts the data from EBCDIC to ASCII. It does this very efficiently, even offloading a lot of the work to a zIIP engine in order to keep the MSU cost of this work to an absolute minimum. This is all done in real time to give you the best data latency possible while not impacting the existing workload on your system.

If you have other data sources like SYSLOG, Log4j or flat files, Ironstream can handle those as well.

kristian_kolb
Ultra Champion

Yes it can. But rather, the question is; can you get your mainframe to dump the SMF records in a readable format (i.e. not EBCDIC) and transport it to a place where the file can be indexed by Splunk?

As you know, there is no forwarder for the mainframe platform (read z/OS), and 'syslog' is not necessarily part of the mainframe toolkit.

I have seen this done with the help of some JCL-code to dump relevant SMF records as XML (yes, I know that it's huge) and transport it via (S)FTP on a regular basis to a place where Splunk reads it as a file. I believe that the conversion from EBCDIC to ASCII was performed by the mainframe FTP utility (in a fairly automated manner).

Google your way to find sample code for dumping SMF records (offered on some IBM web sites), show it to your mainframe people and ask them to adapt it to their environment.

Best of luck,

/K

kristian_kolb
Ultra Champion

No. There is no pretrained sourcetype, if that is what you mean. But getting the output as XML will vastly simplify the parsing of the (variable length) SMF records, since the XML tags are created by the mainframe, and are thus done so correctly.

Then you will have to find out how the various XML-tagged fields will map to 'Failed Login' or 'Access Granted' etc.

0 Karma

CZ1900Splunker
New Member

Thanks Kristian. There's no problem getting the data to the server. But, does Splunk already know how SMF records are formatted or does something have to be done manually to index the data? I did not see SMF listed as a data source type.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...