Data forwarded to third party system missing metadata

[calinm]

Hi,

I have an all in one enterprise splunk install (indexer, search head, file monitoring) with a number of universal forwarders forwarding data to it.

I need to forward everything I receive on the central splunk server to a third party system. I tried both the:

[syslog] (using TCP)
and the
[tcpout]

approaches described in the doc.

The log data is forwarding fine.

The problem is that the metadata is missing. The source,sourcetype and host are missing when using tcpout and the source and sourcetype are missing when using syslog. Without them, the data will make no sense to the receiving system.

Is it possible to use a transform to put the metadata back in the data? If so, How?

Thanks!

Calin

[Jeremiah]

Yes its missing unfortunately.

http://docs.splunk.com/Documentation/Splunk/6.3.3/Forwarding/Forwarddatatothird-partysystemsd

If you need to move data to another system with included metadata you might want to have a look at the dump command or the Hadoop Connect app. You can export to a mounted filesystem if you don't actually have an HDFS cluster running:

http://docs.splunk.com/Documentation/Splunk/6.3.3/SearchReference/Dump
http://www.splunk.com/en_us/solutions/solution-areas/big-data/splunk-hadoop-connect.html

[calinm]

That's interesting, Thanks Jeremiah. I did not know about the dump command, but I don't think it helps here because I need to stream data, not move data.