topic Re: Forwarder config with Multiple Indexers in Splunk Enterprise

Forwarder config with Multiple Indexers

shocko — Tue, 13 Mar 2018 11:34:00 GMT

HI guys, i'm forwarding events form my DCs to both my production and lab instances using the following inputs:

Prod:
[WinEventLog://Security]
disabled = 0
index = corp-dc
persistentQueueSize=500MB

Lab:
[WinEventLog://Security]
disabled = 0
index = lab
persistentQueueSize=500MB

And then my outputs:

Production Enterprise Indexer

[tcpout:production]
server = prod.net:7999

Lab Splunk developer Instance

[tcpout:lab]
server = lab.net:9997

Problem si when the lab indexer stops taking in events (disk full or whatever) the forwarder also stops sending to production. Any ideas?

Re: Forwarder config with Multiple Indexers

tiagofbmm — Tue, 13 Mar 2018 11:43:46 GMT

So you have two Universal Forwarders, one in your Lab DC and one on the PRODUCTION.

And you have two Splunk Enterprise, one Lab and one PRODUCTION. Is this correct?

Re: Forwarder config with Multiple Indexers

shocko — Tue, 13 Mar 2018 11:45:47 GMT

I have a single forwarder forwarding to both a LAB and production indexer. Splunk enterprise on prod and running Splunk Dev License on Lab 😉

Re: Forwarder config with Multiple Indexers

p_gurav — Tue, 13 Mar 2018 12:25:07 GMT

Hi,

Can you put this in outputs.conf:

[tcpout]
defaultGroup=indexer1,indexer2

[tcpout:indexer1]
server=<lab_server_ip>:9997

[tcpout:indexer2]
server=<production_ip>:9997

Refer this data for data cloning:
http://docs.splunk.com/Documentation/Forwarder/7.0.2/Forwarder/Configureforwardingwithoutputs.conf

Re: Forwarder config with Multiple Indexers

tiagofbmm — Tue, 13 Mar 2018 12:33:39 GMT

I don't think the user wants production data going to the dev environment in case the pro indexer fails...

Re: Forwarder config with Multiple Indexers

tiagofbmm — Tue, 13 Mar 2018 12:34:45 GMT

Ok so the reason is probably the Universal Forwarder has a Queue size that would fill up when your indexer fails, as it can't release any of the events in that queue

Re: Forwarder config with Multiple Indexers

p_gurav — Tue, 13 Mar 2018 12:50:24 GMT

Hi,

Yes, thats the case in most of the environments, but here user said " I'm forwarding events form my DCs to both my production and lab instances". Thats why look for this option of cloning.

Re: Forwarder config with Multiple Indexers

tiagofbmm — Tue, 13 Mar 2018 12:56:25 GMT

Agreed, maybe I'm overthinking, let's see what the user says

Re: Forwarder config with Multiple Indexers

shocko — Tue, 13 Mar 2018 13:01:29 GMT

I want data data to go to both but I don't want issues on my lab affecting the forwarding of data to my prod. That's whats happening.

Re: Forwarder config with Multiple Indexers

tiagofbmm — Tue, 13 Mar 2018 13:09:20 GMT

The thing is, if you have two separate instances and are sending segmented data to Prod or Lab, you don't want to get mixed data in either of the environments.

What I would suggest to avoid getting your tcpout queue full is explore this in outputs.conf:

dropEventsOnQueueFull = <integer>
* If set to a positive number, wait <integer> seconds before throwing out
  all new events until the output queue has space.
* Setting this to -1 or 0 will cause the output queue to block when it gets
  full, causing further blocking up the processing chain.
* If any target group's queue is blocked, no more data will reach any other
  target group.
* Using auto load-balancing is the best way to minimize this condition,
  because, in that case, multiple receivers must be down (or jammed up)
  before queue blocking can occur.
* Defaults to -1 (do not drop events).
* DO NOT SET THIS VALUE TO A POSITIVE INTEGER IF YOU ARE MONITORING FILES!

Using this this on the

[tcpout:lab]
server = lab.net:9997