Getting Data In

Universal Forwarder sending two sets of Windows Security Logs to two different indexer

minliang
Loves-to-Learn Lots

I need the Universal Forwarders to send Windows Security Logs to two different indexers but the data I want to send has different criteria.  

I need to send all win security events without a whitelist to Indexer1 and I need to send win security events with a whitelist to indexer2.  

Indexer2 is in another country which will provide 24/7 SOC support and there's a bandwidth limitation.  

Is this possible? 

 

Labels (1)
0 Karma

PickleRick
Ultra Champion

Unfortunately, the "filtering" is applied to a particular input (there is no filtering capability as such on Universal Forwarder - the black/whitelisting is a functionality of this particular input). So you can't get two different data streams from a single input.

And most of the metadata is specified also at the input level (no advanced manipulation on UF, so no props/transforms) so the most you can do - as others already pointed out is to route the events to two destinations. It's the destination HF/indexer that you can try to manipulate the metadata on further (i.e. rewrite the index, source, sourcetype and so on).

0 Karma

sombhtr239
Explorer

Please try to route the data first based upon your tcpoutputproc, for example:

[WinEventLog://System]

_TCP_ROUTING = splunkidxA,splunkidxB

indexer = A

 

As the data is received in both the indexer, try to drop unecessary event from the indexer using props and transforms. Hope this helps.

0 Karma

sushi
Explorer

Can you configure indexer2?
If possible, you set nullqueue on indexer2. If not possible, I think you need to deploy HF between UF and indexers.
The following posts may be helpful.
https://community.splunk.com/t5/Getting-Data-In/Filtering-events-using-NullQueue/m-p/66392 

Unfortunately, Bandwidth control is not possible for each indexer...
UF have thruput in limits.conf, but
this parameter should be global.

0 Karma

minliang
Loves-to-Learn Lots

Okay, I took the suggestion and basically setup index and forwarding.  

I have this in my transforms.prof in indexer1 to forward logs to my second index using TCP_ROUTING.

My new question is that how can I specify the index name to send the data to in Indexer2. 

[DataToForward]

REGEX= XYZ

DEST_KEY = _TCP_ROUTING

FORMAT = INDEXER2

 

 

0 Karma

sushi
Explorer

Umm...I think it is not possible if 2 indexers have different index name...
You can only index on one indexer.

I tried to set following conf.
In default, testsourcetype is indexed to indexname1 on INDEXER1.
INDEXER1 don't have indexname2.
So, INDEXER1 cannot index events which forwarded to indexname2 on INDEXER2.

If you have same indexname on INDEXER1 and INDEXER2, you can index on both indexers.

# inputs.conf
[monitor://testfile]
sourcetype = testsourcetype
index = indexname1

# props.conf
[testsourcetype]
TRANSFORMS-hogehoge = DataToForward,changeIndex

# transforms.conf
[DataToForward]
REGEX= XYZ
DEST_KEY = _TCP_ROUTING
FORMAT = INDEXER2

[changeIndex]
SOURCE_KEY = _TCP_ROUTING
REGEX = INDEXER2
DEST_KEY = _MetaData:Index
FORMAT = indexname2

0 Karma
Get Updates on the Splunk Community!

Maximize the Value from Microsoft Defender with Splunk

<P style=" text-align: center; "><span class="lia-inline-image-display-wrapper lia-image-align-center" ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

<FONT size="5"><FONT size="5" color="#FF00FF">Get the latest news and updates from the Splunk Community ...