topic How do you filter data from a log and send them to 2 splunk instances while discarding the rest? in Getting Data In

How do you filter data from a log and send them to 2 splunk instances while discarding the rest?

melvinfuglem — Wed, 30 Sep 2020 03:17:30 GMT

I'm trying to filter data from a log and send them to 2 splunk instances while discarding the rest.
I've tried a little different config but can't quite get it to work.

Here is my props.conf and transforms.conf

props.conf
[cisco:estreamer:data]
TRANSFORMS-estreamrouting=estreamDiscard,estreamKeep,estreamRouting1,estreamRouting2

transforms.conf
[estreamDiscard]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[estreamKeep]
REGEX = .1.|.2.
DEST_KEY = queue
FORMAT = indexQueue

[estreamRouting1]
SOURCE_KEY = _raw
REGEX = .1.
DEST_KEY = _TCP_ROUTING
FORMAT = Splunk_Old

[estreamRouting2]
SOURCE_KEY = _raw
REGEX = .2.
DEST_KEY = _TCP_ROUTING
FORMAT = Splunk_Test

I have tried various discards and regex filters but i always get 2 results:

Data discard: OK
Splunk instanse 1 : Data 1
Splunk Instanse 2 : No Data
OR
Data Discard: OK
Splunk instanse 1 : Data 1
Splunk instanse 2 : Data 1 and 2

Wanted Result:
Data Discard: OK
Splunk instanse 1 : Data 1
Splunk instanse 2 : Data 2

Any tips?

Re: How do you filter data from a log and send them to 2 splunk instances while discarding the rest?

oscar84x — Mon, 09 Dec 2019 17:11:36 GMT

Could you share a little more about your environment? Are you deploying this configuration on a HF and trying to send to two different indexers?
Can you also share your outputs.conf with your tcpout definitions?

Another idea is if you define an index for each different data type on a specific instance, you could route by index.

[estreamRouting1]
REGEX = .1.
DEST_KEY = _MetaData:Index
FORMAT = <index_on_instance1>

[estreamRouting2]
REGEX = .2.
DEST_KEY = _MetaData:Index
FORMAT = <index_on_instance2>

Re: How do you filter data from a log and send them to 2 splunk instances while discarding the rest?

melvinfuglem — Tue, 10 Dec 2019 08:14:59 GMT

I am configuring this on a HF and we are trying to forward this to 2 other splunk instances owned by other companies while not sending it to our own splunk instance.

My tcpout definitions look like this:

[tcpout]
defaultGroup = splunkInternal1

[tcpout: splunkInternal1]
server = IP:9998, IP:9998

[tcpout:Splunk_Old]
server = IP:9997

[tcpout:Splunk_Test]
server = IP:9997

Re: How do you filter data from a log and send them to 2 splunk instances while discarding the rest?

oscar84x — Tue, 10 Dec 2019 15:18:37 GMT

Thanks. How did you change the configuration between the two different results you mentioned above?
Have you run btool to ensure each piece of the configuration is being applied?

Could you also share a few sample events with the data you're filtering by?

Re: How do you filter data from a log and send them to 2 splunk instances while discarding the rest?

melvinfuglem — Wed, 30 Sep 2020 03:58:13 GMT

sorry for late answer, but i found the problem. One of the splunk instances i was testing on was configured as a search head on the other splunk instance.

This worked.

[estreamRouting1]
SOURCE_KEY = _raw
REGEX = .1.
DEST_KEY = _TCP_ROUTING
FORMAT = Splunk_Old

[estreamRouting2]
SOURCE_KEY = _raw
REGEX = .2.
DEST_KEY = _TCP_ROUTING
FORMAT = Splunk_Test