Best practice for forwarding data into Splunk through an intermediate forwarder

[rewritex]

I am seeking the best practice option to send data to my Splunk instance through an intermediate forwarder with emphasis on not losing data. I use universal forwarders. The intermediate Forwarder is on a Syslog server.

I currently put a forwarder on my hosts which send data to an intermediate forwarder which listens on a port, then sends the data to the Splunk Indexing cluster. Current Setup: Host(w/ forwarder) -> Intermediate Forwarder (listens TCP/UDP) -> Sends to Cluster

My question is:
1Q: Any links/advice to setting up this configuration with emphasis on not losing the data?
2Q: Should my setup be: Host (w/ syslog-out) -> syslog_server -> Intermediate_forwarder (listen to syslog.log) -> Cluster
3Q: Can this work? Host (w/ forwarder) -> syslog_server -> Intermediate_forwarder(listen to syslog.log) -> cluster
4Q: When I restart the host or forwarder, which is the best setup to not lose data?

I've read quite a number of forum posts but I may have missed something. Thank You.

[s2_splunk · Accepted solution]

This article may help, in addition to the other responses you already received.
Intermediary forwarders come with their own architectural pitfalls and you need to ensure that any intermediary forwarding tier is architected properly to not introduce single points of failure or cause event distribution issues across your indexers. Sometimes, network connectivity restrictions or other requirements make them necessary, but - like others have said - you shouldn't introduce any additional tiers in your forwarding architecture that are not needed. The KISS principle applies.

[richgalloway]

Intermediate forwarders are not considered Best Practice. The preferred approach is to forward your syslogs to a syslog server then have a Universal Forwarder on the syslog server forward logs to your index cluster.

[rewritex]

Thank you for the response.

Concerning the host setup best option.... can i use the universal forwarder to send data to the dedicate_syslog server or should I just syslog-out?

[starcher]

Rich is suggesting you receive syslog to something like rsyslog or syslog-ng. Write to file. Monitor the files with the Universal Forward and it sends to Splunk.