How do you convert an indexer into a heavy forwarder?

[PCT80000]

Hello all,

We are replacing our single Splunk indexer with a pair of new indexers and have migrated all the indexes except those filled by syslog sources.

We know that sending syslog straight to an indexer is not best practice, so we are now looking at directing this to SyslogNG first. However, we would like to make use of the old Splunk indexer server to take the output of syslogNG and load balance it to the two new indexers.

What we don't understand is if this is simply a matter of editing the old indexers outputs.conf or if the indexer will still need to function to take the different UDP data input ports and direct them to the correct indexes.

Thanks in advance!

[xavierashe · Accepted solution]

You are correct. All you have to do is set your outputs.conf to something like this:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = splunkindex1.corp.com:9997, splunkindex2.corp.com:9997

The reason you can do this is that indexAndForward is set to false by default. http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Configureforwarderswithoutputs.con...

If set to "false" (the default), the
forwarder forwards data but does not
index it.

[PCT80000]

would we need to change indexAndForward to true in order for the server to continue to resolve events from different syslog tcp ports into the correct indexes and then forward these on. Is this what defines a heavy forwarder?

[xavierashe]

Splunk separates outputs from inputs from indexing. You can have a Splunk server accept data or not. You can have a Splunk server forward data or not. You can have a Splunk server index data or not.

From what you have posted I gather you want to have your server accept syslog data, not index locally, and forward it to other Splunk servers.

If you configure a standard outputs.conf, by default Splunk does not index that data locally (indexAndForward=false). All your inputs will be forwarded to the outputs you define in outputs.conf.

The ability to "resolve events from different syslog tcp ports into the correct indexes" is defined by your inputs.conf.

[PCT80000]

What we need is to get the syslog event data to the two new indexers.

We were hoping that the existing old indexer would use its inputs.conf to process the many different syslog tcp ports into events for the correct indexes. We dont know if changing the outputs.conf to point at the new indexers would just mean that the syslog tcp would be sent directly to the new indexers, or if it will be event data.

[xavierashe]

It will be event data. Splunk assumes that if you specify an output, it's a Splunk server that you're outputting to. If you want to output raw syslog, you have to do more. This configuration should work for you.

[PCT80000]

thanks xavierashe. Would we still need to have the indexer running in this config?

[xavierashe]

The indexer processes will still be running, but nothing will be stored locally. That's what makes it a "Heavy" forwarder.

[PCT80000]

Brilliant, thanks for all your help

[xavierashe]

Is there a reason why you wouldn't install a universal forwarder on the new syslog-ng server?

[PCT80000]

We could do this, but it is an appliance supplied by Balabit (SyslogNG-StoreBox). I dont think we have access to install splunk.