topic Re: Best practice for forwarding data into Splunk through an intermediate forwarder in Getting Data In

Best practice for forwarding data into Splunk through an intermediate forwarder

rewritex — Thu, 09 Feb 2017 21:35:30 GMT

I am seeking the best practice option to send data to my Splunk instance through an intermediate forwarder with emphasis on not losing data. I use universal forwarders. The intermediate Forwarder is on a Syslog server.

I currently put a forwarder on my hosts which send data to an intermediate forwarder which listens on a port, then sends the data to the Splunk Indexing cluster. Current Setup: Host(w/ forwarder) -> Intermediate Forwarder (listens TCP/UDP) -> Sends to Cluster

My question is:
1Q: Any links/advice to setting up this configuration with emphasis on not losing the data?
2Q: Should my setup be: Host (w/ syslog-out) -> syslog_server -> Intermediate_forwarder (listen to syslog.log) -> Cluster
3Q: Can this work? Host (w/ forwarder) -> syslog_server -> Intermediate_forwarder(listen to syslog.log) -> cluster
4Q: When I restart the host or forwarder, which is the best setup to not lose data?

I've read quite a number of forum posts but I may have missed something. Thank You.

Re: Best practice for forwarding data into Splunk through an intermediate forwarder

richgalloway — Thu, 09 Feb 2017 22:23:44 GMT

Intermediate forwarders are not considered Best Practice. The preferred approach is to forward your syslogs to a syslog server then have a Universal Forwarder on the syslog server forward logs to your index cluster.

Re: Best practice for forwarding data into Splunk through an intermediate forwarder

rewritex — Thu, 09 Feb 2017 23:18:35 GMT

Thank you for the response.

Concerning the host setup best option.... can i use the universal forwarder to send data to the dedicate_syslog server or should I just syslog-out?

Re: Best practice for forwarding data into Splunk through an intermediate forwarder

starcher — Fri, 10 Feb 2017 02:36:05 GMT

Rich is suggesting you receive syslog to something like rsyslog or syslog-ng. Write to file. Monitor the files with the Universal Forward and it sends to Splunk.

Re: Best practice for forwarding data into Splunk through an intermediate forwarder

s2_splunk — Fri, 10 Feb 2017 17:59:14 GMT

This article may help, in addition to the other responses you already received.
Intermediary forwarders come with their own architectural pitfalls and you need to ensure that any intermediary forwarding tier is architected properly to not introduce single points of failure or cause event distribution issues across your indexers. Sometimes, network connectivity restrictions or other requirements make them necessary, but - like others have said - you shouldn't introduce any additional tiers in your forwarding architecture that are not needed. The KISS principle applies.