Splunk Search

Data forwarded to third party system missing metadata

calinm
Engager

Hi,

I have an all in one enterprise splunk install (indexer, search head, file monitoring) with a number of universal forwarders forwarding data to it.

I need to forward everything I receive on the central splunk server to a third party system. I tried both the:

[syslog] (using TCP)
and the
[tcpout]

approaches described in the doc.

The log data is forwarding fine.

The problem is that the metadata is missing. The source,sourcetype and host are missing when using tcpout and the source and sourcetype are missing when using syslog. Without them, the data will make no sense to the receiving system.

Is it possible to use a transform to put the metadata back in the data? If so, How?

Thanks!

Calin

0 Karma

Jeremiah
Motivator

Yes its missing unfortunately.

http://docs.splunk.com/Documentation/Splunk/6.3.3/Forwarding/Forwarddatatothird-partysystemsd

If you need to move data to another system with included metadata you might want to have a look at the dump command or the Hadoop Connect app. You can export to a mounted filesystem if you don't actually have an HDFS cluster running:

http://docs.splunk.com/Documentation/Splunk/6.3.3/SearchReference/Dump
http://www.splunk.com/en_us/solutions/solution-areas/big-data/splunk-hadoop-connect.html

0 Karma

calinm
Engager

That's interesting, Thanks Jeremiah. I did not know about the dump command, but I don't think it helps here because I need to stream data, not move data.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!