topic Re: Universal Forwarder sending two sets of Windows Security Logs to two different indexer in Getting Data In

Universal Forwarder sending two sets of Windows Security Logs to two different indexer

minliang — Mon, 01 Nov 2021 23:03:14 GMT

I need the Universal Forwarders to send Windows Security Logs to two different indexers but the data I want to send has different criteria.

I need to send all win security events without a whitelist to Indexer1 and I need to send win security events with a whitelist to indexer2.

Indexer2 is in another country which will provide 24/7 SOC support and there's a bandwidth limitation.

Is this possible?

Re: Universal Forwarder sending two sets of Windows Security Logs to two different indexer

sushi — Tue, 02 Nov 2021 01:17:33 GMT

Can you configure indexer2?
If possible, you set nullqueue on indexer2. If not possible, I think you need to deploy HF between UF and indexers.
The following posts may be helpful.
https://community.splunk.com/t5/Getting-Data-In/Filtering-events-using-NullQueue/m-p/66392

Unfortunately, Bandwidth control is not possible for each indexer...
UF have thruput in limits.conf, but this parameter should be global.

Re: Universal Forwarder sending two sets of Windows Security Logs to two different indexer

minliang — Tue, 02 Nov 2021 15:44:43 GMT

Okay, I took the suggestion and basically setup index and forwarding.

I have this in my transforms.prof in indexer1 to forward logs to my second index using TCP_ROUTING.

My new question is that how can I specify the index name to send the data to in Indexer2.

[DataToForward]

REGEX= XYZ

DEST_KEY = _TCP_ROUTING

FORMAT = INDEXER2

Re: Universal Forwarder sending two sets of Windows Security Logs to two different indexer

sushi — Fri, 05 Nov 2021 08:11:48 GMT

Umm...I think it is not possible if 2 indexers have different index name...
You can only index on one indexer.

I tried to set following conf.
In default, testsourcetype is indexed to indexname1 on INDEXER1.
INDEXER1 don't have indexname2.
So, INDEXER1 cannot index events which forwarded to indexname2 on INDEXER2.

If you have same indexname on INDEXER1 and INDEXER2, you can index on both indexers.

# inputs.conf
[monitor://testfile]
sourcetype = testsourcetype
index = indexname1

# props.conf
[testsourcetype]
TRANSFORMS-hogehoge = DataToForward,changeIndex

# transforms.conf
[DataToForward]
REGEX= XYZ
DEST_KEY = _TCP_ROUTING
FORMAT = INDEXER2

[changeIndex]
SOURCE_KEY = _TCP_ROUTING
REGEX = INDEXER2
DEST_KEY = _MetaData:Index
FORMAT = indexname2

Re: Universal Forwarder sending two sets of Windows Security Logs to two different indexer

sombhtr239 — Fri, 05 Nov 2021 09:21:15 GMT

Please try to route the data first based upon your tcpoutputproc, for example:

[WinEventLog://System]

_TCP_ROUTING = splunkidxA,splunkidxB

indexer = A

As the data is received in both the indexer, try to drop unecessary event from the indexer using props and transforms. Hope this helps.

Re: Universal Forwarder sending two sets of Windows Security Logs to two different indexer

PickleRick — Fri, 05 Nov 2021 12:36:17 GMT

Unfortunately, the "filtering" is applied to a particular input (there is no filtering capability as such on Universal Forwarder - the black/whitelisting is a functionality of this particular input). So you can't get two different data streams from a single input.

And most of the metadata is specified also at the input level (no advanced manipulation on UF, so no props/transforms) so the most you can do - as others already pointed out is to route the events to two destinations. It's the destination HF/indexer that you can try to manipulate the metadata on further (i.e. rewrite the index, source, sourcetype and so on).