Getting Data In

How do I configure my heavy forwarder to filter and route data as expected?

herman_vb
New Member

I'm having some issues with a heavy forwarder that I can't explain, and I was hoping someone could help me.

First question:
I have 1 heavy forwarder and 3 separate indexers. How can I define on the heavy forwarder which data stream needs to go where? Now I'm just receiving errors that a certain index does not exist on an indexer (which is correct). If I can just ignore the error, it's also fine, but I was hoping there is a cleaner way of doing it.

Second question:
On the heavy-forwarder I have defined the following

In props.conf:

[WinEventLog:Security]
TRANSFORMS-security = dropadlog, adlog

in transforms.conf

[adlog]
REGEX = (?msi)^EventCode=(4625|4768|4769|4771|4773|4776|4740)
DEST_KEY = queue
FORMAT = indexQueue

[dropadlog]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

So what I want to do here is only allow the eventcodes that are defined and drop the rest. What is weird is that it passes on 4768 and 4769. All other eventcodes get dropped (or at least, they do not reach the index).

Any ideas?
I know I can whitelist this on the Universal forwarder already, but I would like to do it on the heavy forwarder instead if possible.

0 Karma
1 Solution

ryanoconnor
Builder

I understand question 1 a bit better now thank you. It is definitely possible, You're going to want to look at routing and filtering data

http://docs.splunk.com/Documentation/Splunk/6.4.1/Forwarding/Routeandfilterdatad

As for your second question, is it possible that the other event codes just aren't being generated on your server? So for example you stated you only saw 4768 and 4769. Is auditing turned on so that events like 4625 are ending up in the Security logs on your Windows system?

View solution in original post

tommoore
Path Finder

You need to put the [dropadlog] stanza BEFORE your [adlog] one, otherwise it will drop everything regardless of the tag. See "Keep specific events and discard the rest" on http://docs.splunk.com/Documentation/Splunk/7.1.1/Forwarding/Routeandfilterdatad

0 Karma

ryanoconnor
Builder

I understand question 1 a bit better now thank you. It is definitely possible, You're going to want to look at routing and filtering data

http://docs.splunk.com/Documentation/Splunk/6.4.1/Forwarding/Routeandfilterdatad

As for your second question, is it possible that the other event codes just aren't being generated on your server? So for example you stated you only saw 4768 and 4769. Is auditing turned on so that events like 4625 are ending up in the Security logs on your Windows system?

herman_vb
New Member

The index exists on server1. It only needs to exist on server1, not on server2. So the question is, will I need 2 heavy-forwards to be able to filter my events sent to indexes. 1 per indexer(or index cluster)?
Why doesn't it exist on the other server? It doesn't need to be there, and won't it count it double to index the data to 2 separate indexers? in other words, I'll be paying for data I won't need.
As for the reason why we are splitting things up; performance and security. It makes it easier for us to physically split the servers and indexes for certain sources. Not everything.

As for the second: in the transforms bit, you can see I call both. If i first call the adlog and after that the dropadlog, it drops nothing. If I first call the dropadlog and after the adlog, it only passes on 2 eventcodes and drops the rest. Mystery.

0 Karma

herman_vb
New Member

The index exists on server1. It only needs to exist on server1, not on server2. So the question is, will I need 2 heavy-forwards to be able to filter my events sent to indexes. 1 per indexer(or index cluster)?
Why doesn't it exist on the other server? It doesn't need to be there, and won't it count it double to index the data to 2 separate indexers? in other words, I'll be paying for data I won't need.
As for the reason why we are splitting things up; performance and security. It makes it easier for us to physically split the servers and indexes for certain sources. Not everything.

As for the second: in the transforms bit, you can see I call both. If i first call the adlog and after that the dropadlog, it drops nothing. If I first call the dropadlog and after the adlog, it only passes on 2 eventcodes and drops the rest. Mystery.

0 Karma

ryanoconnor
Builder

Let's start with question number 1, is there a reason you don't want to create the index it's complaining about?

Question 2: it looks like your props.conf is referencing the dropadlog stanza and then you also have the adlog stanza. Is the adlog stanza being referenced anywhere?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...