Forwarder config with Multiple Indexers

[shocko]

HI guys, i'm forwarding events form my DCs to both my production and lab instances using the following inputs:

Prod:
[WinEventLog://Security]
disabled = 0
index = corp-dc
persistentQueueSize=500MB

Lab:
[WinEventLog://Security]
disabled = 0
index = lab
persistentQueueSize=500MB

And then my outputs:

Production Enterprise Indexer

[tcpout:production]
server = prod.net:7999

Lab Splunk developer Instance

[tcpout:lab]
server = lab.net:9997

Problem si when the lab indexer stops taking in events (disk full or whatever) the forwarder also stops sending to production. Any ideas?

[p_gurav]

Hi,

Can you put this in outputs.conf:

[tcpout]
defaultGroup=indexer1,indexer2

[tcpout:indexer1]
server=<lab_server_ip>:9997

[tcpout:indexer2]
server=<production_ip>:9997

Refer this data for data cloning:
http://docs.splunk.com/Documentation/Forwarder/7.0.2/Forwarder/Configureforwardingwithoutputs.conf

[tiagofbmm]

I don't think the user wants production data going to the dev environment in case the pro indexer fails...

[p_gurav]

Hi,

Yes, thats the case in most of the environments, but here user said " I'm forwarding events form my DCs to both my production and lab instances". Thats why look for this option of cloning.

[tiagofbmm]

Agreed, maybe I'm overthinking, let's see what the user says

[shocko]

I want data data to go to both but I don't want issues on my lab affecting the forwarding of data to my prod. That's whats happening.

[tiagofbmm]

The thing is, if you have two separate instances and are sending segmented data to Prod or Lab, you don't want to get mixed data in either of the environments.

What I would suggest to avoid getting your tcpout queue full is explore this in outputs.conf:

dropEventsOnQueueFull = <integer>
* If set to a positive number, wait <integer> seconds before throwing out
  all new events until the output queue has space.
* Setting this to -1 or 0 will cause the output queue to block when it gets
  full, causing further blocking up the processing chain.
* If any target group's queue is blocked, no more data will reach any other
  target group.
* Using auto load-balancing is the best way to minimize this condition,
  because, in that case, multiple receivers must be down (or jammed up)
  before queue blocking can occur.
* Defaults to -1 (do not drop events).
* DO NOT SET THIS VALUE TO A POSITIVE INTEGER IF YOU ARE MONITORING FILES!

Using this this on the

[tcpout:lab]
server = lab.net:9997

[tiagofbmm]

So you have two Universal Forwarders, one in your Lab DC and one on the PRODUCTION.

And you have two Splunk Enterprise, one Lab and one PRODUCTION. Is this correct?

[shocko]

I have a single forwarder forwarding to both a LAB and production indexer. Splunk enterprise on prod and running Splunk Dev License on Lab 😉

[tiagofbmm]

Ok so the reason is probably the Universal Forwarder has a Queue size that would fill up when your indexer fails, as it can't release any of the events in that queue