Getting Data In

How do you convert an indexer into a heavy forwarder?

PCT80000
Explorer

Hello all,

We are replacing our single Splunk indexer with a pair of new indexers and have migrated all the indexes except those filled by syslog sources.

We know that sending syslog straight to an indexer is not best practice, so we are now looking at directing this to SyslogNG first. However, we would like to make use of the old Splunk indexer server to take the output of syslogNG and load balance it to the two new indexers.

What we don't understand is if this is simply a matter of editing the old indexers outputs.conf or if the indexer will still need to function to take the different UDP data input ports and direct them to the correct indexes.

Thanks in advance!

0 Karma
1 Solution

xavierashe
Contributor

You are correct. All you have to do is set your outputs.conf to something like this:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = splunkindex1.corp.com:9997, splunkindex2.corp.com:9997

The reason you can do this is that indexAndForward is set to false by default. http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Configureforwarderswithoutputs.con...

If set to "false" (the default), the
forwarder forwards data but does not
index it.

View solution in original post

0 Karma

xavierashe
Contributor

You are correct. All you have to do is set your outputs.conf to something like this:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = splunkindex1.corp.com:9997, splunkindex2.corp.com:9997

The reason you can do this is that indexAndForward is set to false by default. http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Configureforwarderswithoutputs.con...

If set to "false" (the default), the
forwarder forwards data but does not
index it.

0 Karma

PCT80000
Explorer

would we need to change indexAndForward to true in order for the server to continue to resolve events from different syslog tcp ports into the correct indexes and then forward these on. Is this what defines a heavy forwarder?

0 Karma

xavierashe
Contributor

Splunk separates outputs from inputs from indexing. You can have a Splunk server accept data or not. You can have a Splunk server forward data or not. You can have a Splunk server index data or not.

From what you have posted I gather you want to have your server accept syslog data, not index locally, and forward it to other Splunk servers.

If you configure a standard outputs.conf, by default Splunk does not index that data locally (indexAndForward=false). All your inputs will be forwarded to the outputs you define in outputs.conf.

The ability to "resolve events from different syslog tcp ports into the correct indexes" is defined by your inputs.conf.

0 Karma

PCT80000
Explorer

What we need is to get the syslog event data to the two new indexers.

We were hoping that the existing old indexer would use its inputs.conf to process the many different syslog tcp ports into events for the correct indexes. We dont know if changing the outputs.conf to point at the new indexers would just mean that the syslog tcp would be sent directly to the new indexers, or if it will be event data.

0 Karma

xavierashe
Contributor

It will be event data. Splunk assumes that if you specify an output, it's a Splunk server that you're outputting to. If you want to output raw syslog, you have to do more. This configuration should work for you.

0 Karma

PCT80000
Explorer

thanks xavierashe. Would we still need to have the indexer running in this config?

0 Karma

xavierashe
Contributor

The indexer processes will still be running, but nothing will be stored locally. That's what makes it a "Heavy" forwarder.

0 Karma

PCT80000
Explorer

Brilliant, thanks for all your help

0 Karma

xavierashe
Contributor

Is there a reason why you wouldn't install a universal forwarder on the new syslog-ng server?

0 Karma

PCT80000
Explorer

We could do this, but it is an appliance supplied by Balabit (SyslogNG-StoreBox). I dont think we have access to install splunk.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...