How do I configure my heavy forwarder to filter and route data as expected?

[herman_vb]

I'm having some issues with a heavy forwarder that I can't explain, and I was hoping someone could help me.

First question:
I have 1 heavy forwarder and 3 separate indexers. How can I define on the heavy forwarder which data stream needs to go where? Now I'm just receiving errors that a certain index does not exist on an indexer (which is correct). If I can just ignore the error, it's also fine, but I was hoping there is a cleaner way of doing it.

Second question:
On the heavy-forwarder I have defined the following

In props.conf:

[WinEventLog:Security]
TRANSFORMS-security = dropadlog, adlog

in transforms.conf

[adlog]
REGEX = (?msi)^EventCode=(4625|4768|4769|4771|4773|4776|4740)
DEST_KEY = queue
FORMAT = indexQueue

[dropadlog]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

So what I want to do here is only allow the eventcodes that are defined and drop the rest. What is weird is that it passes on 4768 and 4769. All other eventcodes get dropped (or at least, they do not reach the index).

Any ideas?
I know I can whitelist this on the Universal forwarder already, but I would like to do it on the heavy forwarder instead if possible.

[tommoore]

You need to put the [dropadlog] stanza BEFORE your [adlog] one, otherwise it will drop everything regardless of the tag. See "Keep specific events and discard the rest" on http://docs.splunk.com/Documentation/Splunk/7.1.1/Forwarding/Routeandfilterdatad

[ryanoconnor · Accepted solution]

I understand question 1 a bit better now thank you. It is definitely possible, You're going to want to look at routing and filtering data

http://docs.splunk.com/Documentation/Splunk/6.4.1/Forwarding/Routeandfilterdatad

As for your second question, is it possible that the other event codes just aren't being generated on your server? So for example you stated you only saw 4768 and 4769. Is auditing turned on so that events like 4625 are ending up in the Security logs on your Windows system?

[herman_vb]

The index exists on server1. It only needs to exist on server1, not on server2. So the question is, will I need 2 heavy-forwards to be able to filter my events sent to indexes. 1 per indexer(or index cluster)?
Why doesn't it exist on the other server? It doesn't need to be there, and won't it count it double to index the data to 2 separate indexers? in other words, I'll be paying for data I won't need.
As for the reason why we are splitting things up; performance and security. It makes it easier for us to physically split the servers and indexes for certain sources. Not everything.

As for the second: in the transforms bit, you can see I call both. If i first call the adlog and after that the dropadlog, it drops nothing. If I first call the dropadlog and after the adlog, it only passes on 2 eventcodes and drops the rest. Mystery.

[herman_vb]

The index exists on server1. It only needs to exist on server1, not on server2. So the question is, will I need 2 heavy-forwards to be able to filter my events sent to indexes. 1 per indexer(or index cluster)?
Why doesn't it exist on the other server? It doesn't need to be there, and won't it count it double to index the data to 2 separate indexers? in other words, I'll be paying for data I won't need.
As for the reason why we are splitting things up; performance and security. It makes it easier for us to physically split the servers and indexes for certain sources. Not everything.

As for the second: in the transforms bit, you can see I call both. If i first call the adlog and after that the dropadlog, it drops nothing. If I first call the dropadlog and after the adlog, it only passes on 2 eventcodes and drops the rest. Mystery.

[ryanoconnor]

Let's start with question number 1, is there a reason you don't want to create the index it's complaining about?

Question 2: it looks like your props.conf is referencing the dropadlog stanza and then you also have the adlog stanza. Is the adlog stanza being referenced anywhere?