Syncsort Ironstream: Why does the TCP stream need no destination index configured?

[marrette]

We've configured our Splunk instance to receive data via TCP stream from the Syncsort Ironstream forwarder installed on a mainframe. What's strange with this setup is that the inputs.conf file does not specify an index, somehow it is specified in Ironstream on the mainframe because the data arrives it automatically goes into that, non-default, index.

The inputs.conf file looks like this:

[tcp://9998]
connection_host = dns
sourcetype = json

Yet when the data is received Splunk know's to put it into the 'Mainframe' index. How is this happening? Is there some sort of 'meta' data the wraps the data on the way in that Splunk uses to determine the index?

[ianhss]

Hi marrette,

I am happy to help and glad to hear you are using Ironstream successfully.

Yes, there is metadata involved. Bottom line, this is legitimate and has no impact or overhead on your indexing. The metadata, at least for target index, is not indexed. But this is Splunk internals, hence I am not at liberty to reveal the full workings.

But if you put on your lateral thinking hat and use a wire tracing tool (such as one of our own Syncsort ZEN Suite components) you will be able to see what is being sent.

Happy [mainframe] Splunking!

[ianhss]

Hi marrette. No magic, all legitimate Splunk use.

Splunk can use meta data for specifying the receiving index. This can be specified in the Ironstream mainframe configuration. The value will be used as a data destination, unless it is overridden in Splunk. Failing that, Splunk will send the data to the default index.

Are you a licensed customer or using our free Starter Edition? If a licensed customer, and you have further questions, our Support Team will be very happy to help. Please feel free contact them via our MySypport site.

[marrette]

Hi Ianhas,
Thanks for the quick reply. We are indeed using the licensed version of Ironstream and it is all working fine.

I'm really just asking this question academically as I always thought that when Splunk is configured to receive data from a TCP stream you'd see everything in Splunk - but it sounds like there is some 'metadata' being sent with the stream that is understood by Splunk but not indexed? What is this data, and how is it sent to Splunk (is it in the packet header for example)?