Forwarding Data To Third Party, Hadoop Connect Issues

[samj1313]

Hello,

We are trying to send some data to a third party system that is hitting our indexers from Universal Forwarders, thus we can't send it via syslog as described here: http://docs.splunk.com/Documentation/Splunk/6.2.1/Forwarding/Forwarddatatothird-partysystemsd because it is not going to Heavy Forwarders.

We have been trying to use Hadoop Connect to run a query to write the desired data to HDFS, but there are errors that cause the destination directory in HDFS to be cleaned up and all data is removed. For example, if the scheduled export search is to be performed every hour, the directory grows and grows, but after the hour is up all data is removed from that directory and it starts all over again. We have tried different intervals for export but nothing has worked. The errors we are getting revolve around "Reading errors while waiting for the indexers". We are trying to pull back ~30GB every hour and we suspect there to be an issue with Splunk keeping up with our search.

1) Any ideas of ways to make Hadoop Connect in this instance work or to debug it better?
2) Are there any alternatives to sending data to a third party system with this configuration?

Thanks.

[alacercogitatus]

Simply enough, you can set that config on your indexers. I currently have our indexers sending syslog data to QRadar based on host name. I'm using a props/transforms to route that outbound via udp syslog. I also just discovered that while I was using TCP out to make sure the data got to Qradar, the Qradar system stopped listening on the ports it needed to. Therefore, ALL of my indexer queues got backed up and indexing stopped. A quick turn to udp fixed that and now Security has to redo Qradar to accept udp inputs.

What is your thirdparty system? If it takes syslog, send it udp syslog from the indexers. It'll work 😄

[redman1138]

am trying to forward data to qradar/hadoop via syslog and the host field is being replaced by the indexer name. were you able to get around that issue? Could you post a copy of your props, transforms and outputs.conf file?

[samj1313]

That sounds like a great solution! Our third party system takes syslog so that won't be a problem. Are there any docs on that setup?

Thanks for the quick reply!