Forwarding Mainframe logs to Splunk

[vkbinsplunk]

I know we can forward logs from a Linux box to Splunk (if we install Splunk forwarder on the Linux box). Similarly can we forward logs from Mainframe - CICS region to Splunk? Do we have any method to achieve that? Please suggest.

I heard that, we can forward the logs from Mainframe to Linux/windows box first and then to Splunk. If that is the case, how can we forward logs from Mainframe to Linux/windows? Please share any ideas?

Thank You.

[tldenney]

IBM Common Data Provider for z Systems (CDPz) is the best option for sending Mainframe logs to Splunk.

CDPz can send a wide variety of data including 140 data sources and 100+ SMF record types. More specifically, CDPz can support the following:

• SMF records
• SYSLOG (IBM z/OS System Log and USS SyslogD)
• JOBLOGs
• Application logs (IBM CICS Transaction Server logs and IBM WebSphere Application Server logs)

CDPz also has advanced filtering capabilities including RegEx and time filtering that can be set up using the built-in web configuration tool shown below.

More information on IBM Common Data Provider for z Systems can be found directly on Splunkbase.

[tldenney]

The following Splunk Blog outlines how Splunk and IBM are partnering to help customers integrate IBM Z (Mainframe) Data and Insights into Splunk software:

https://www.splunk.com/blog/2017/08/22/insane-in-the-mainframe-splunk-and-ibm-partner-to-provide-end...

[jreda]

Ironstream from Syncsort can do all of this work for you. It will handle all of the issues related to SYSLOG, z/OS SMF records, log4j and flat files. It deals with the compression, the triplets, the binary data and converts the data from EBCDIC to ASCII. It does this very efficiently, even offloading a lot of the work to a zIIP engine in order to keep the MSU cost of this work to an absolute minimum. This is all done in real time to give you the best data latency possible while not impacting the existing workload on your system.

[sureshchinta]

Has there been any development in this area since then. Is there a native app / forwarder in Splunk to send logs from mainframe to Splunk server.

[steveirogers]

While not a direct Splunk solution, I came across this information while doing some research: http://www.infosecinc.com/
Quote: "Mainframe Event Acquisition System™ (MEAS™) is designed to help clients collect real-time information pertaining to security events, database related events, transaction processing events and more.

This technology enables mainframe clients to collect, store, report and take action against the event data through integration with SIEM (Security Information and Event Management) and log management technologies. MEASTM is integrated with all popular, commercially available SIEM and log management products.."

[lguinn2]

If you can write the log files to a shared filesystem like NFS, then a forwarder on a Linux/Windows box could access the log files there.

If you have syslog on the mainframe, or a similar tool that can emit the data over a network port, Splunk can monitor the port.

You could use any other mechanism for transporting the log files from the mainframe to a Linux/Windows box. But be careful; your technique needs to be robust enough to deal with network outages, server restarts, log file rotation etc. It also needs to avoid sending the same data twice. So writing your own script to forward data is not a trivial task.