How to forward data from an indexer to a 3rd party server

[anton085]

Hi,

I have the following setup:

3rd Party Server <---- Splunk Enterprise (Indexer):9997 <---- [Splunk Enterprise (Heavy Forwarder)] OR [Universal Forwarder]

If the forwarder is monitoring a file, for example: /var/log/syslog, how can I forward the events from only that file it from the Indexer to the 3rd party server? My conf files in the Indexer are given below, and this settings don't work:

props.conf:
[source::/var/log/syslog]
TRANSFORMS-routing=send_to_syslog

transforms.conf:
[send_to_syslog]
DEST_KEY=_SYSLOG_ROUTING
FORMAT=syslog_abc
REGEX=.

outputs.conf:
[syslog:syslog_abc]
disabled=false
server=x.x.x.x:514
timestampformat=%b %e %H:%M:%S
type=tcp

Thanks

[traxxasbreaker]

I hope you found something for the actual routing in the time since you asked this, or would request to see any relevant events in your splunk.d log related to that config, but I also wanted to put a word of warning out there on TCP syslog forwarding from your indexers.

If your syslog destination is down, what will happen? Is the IP you put in there actually a VIP that will always point to an active syslog destination?

If not what I've seen happen in scenarios when a TCP syslog destination is down, Splunk continues to hold the data destined for it in it's internal queues. Over time, which is relatively short for a high volume of data the queues all fill up and eventually result in the indexer being blocked and unable to return search results. As the forwarders redirect to other indexers, they start taking the rest down too.

While I hear there's also some settings that could change the behavior of whether splunk continues to hold the data in queue while waiting on the TCP response, we've also switched to UDP syslog forwarding to prevent a problem on the destination from taking out our indexing cluster again.