topic Re: Filtering and sending only specific data in Getting Data In

Filtering and sending only specific data

JPValadas — Mon, 28 Sep 2020 11:40:06 GMT

Hi again,

I got one question in filtering and routing to indexer.

i got my props like this:

pros.conf

[WinEventLog:System]

TRANSFORMS-wmi = WantedEventCodes

outputs.conf

[tcpout]

defaultGroup = splunk.host.pt_9997

[tcpout:splunk.host.pt_9997]

autoLB = true

server = splunk.host.pt:9997

transforms.conf

[WantedEventCodes]

REGEX = (?msi)^EventCode=(7040)

DEST_KEY= _TCP_ROUTING

FORMAT = splunk.host.pt_9997

My question is, my forwarder still sending everything to my inxeder, but shouldnt he filter and send only my event code 7040 to indexer?

Thank you.

Re: Filtering and sending only specific data

kristian_kolb — Wed, 11 Apr 2012 17:25:17 GMT

I think that you have a default configuration of sending logs to splunk.host.pt:9997, and then you specify that you want to send a subset of these (those matching your EventCode regex) to ... the same place.

Didn't the nullQueue config work out for you?
http://splunk-base.splunk.com/answers/44583/splunk-filter-on-windows

UPDATE:

Is there perhaps an element of confusion here?

On a heavy forwarder (or indexer), you can use the nullQueue approach to stop some events from being indexed by dropping them, according to your regex.

The _TCP_ROUTING approach can be used when you normally send all your data to one (group of) indexer(s), but want to send some of the data to a different (group of) indexer(s).

Configuring _TCP_ROUTING is not needed for normal forwarding/indexing of events to your ordinary indexer(s).

UPDATE 2:

No. Sending events to the nullQueue will not count towards the license.

If you only have one indexer, you should use the nullQueue to filter out events that you do not want to index. If you have a Heavy forwarder you put your configs there, if you have a Lightweight or Universal forwarder, you do the configuration on the indexer.

/kristian

Re: Filtering and sending only specific data

jbsplunk — Wed, 11 Apr 2012 17:41:40 GMT

Kristian is correct, I think, with his assessment of the configuration. This configuration would forward everything to splunk.host.pt:9997. Additionally, this configuration would only a subset of events from a heavy forwarder to an indexer, or from an indexer to another indexer. If you're trying to do this on a Universal/Lightweight forwarder, it isn't going to work because no data is parsed on those instances.

Re: Filtering and sending only specific data

JPValadas — Wed, 11 Apr 2012 17:45:50 GMT

That worked perfectly, but the qeues was on forwarder.

Now i want to send the data from hosts to indexer, but i want to specify what kind of data do i want to send to Indexer.

In this way i will not have problems with exceding space limits (500megas)

In other example i was filtering but not sending data do indexer now i´m trying to use TCP_ROUTING to send data to indexer.

Re: Filtering and sending only specific data

jbsplunk — Wed, 11 Apr 2012 17:48:24 GMT

You can do this on the indexer with the instructions found here:

http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Routeandfilterdatad

You're looking for the section that says 'Keep specific events and discard the rest'. This is intended for the exact use case you describe, to avoid exceeding the license volume allowed.

Re: Filtering and sending only specific data

JPValadas — Wed, 11 Apr 2012 17:57:11 GMT

jbsplunk and Kristian thank you for the answer,

jbsplunk i´m doing this with a heavy forwarder and i should do the changes in forwarder right? not in indexer.

I created that props and transforms on forwarder to make him send only Event Code 7040 to indexer, TCP Routing in forwarder should not send to indexer?

Re: Filtering and sending only specific data

jbsplunk — Wed, 11 Apr 2012 19:01:22 GMT

If you're using a heavy forwarder, then the changes will be made on the forwarder, that is correct. If you're using a heavy forwarder, you can just throw all the events you don't want to nullQueue at the forwarder level, and what's left will be sent on.

I am not sure I understand what you mean by your last question, but the configuration you were previously using sent everything to a single indexer, and then a subset of events to that same indexer, which seems redundant.

Re: Filtering and sending only specific data

kristian_kolb — Thu, 12 Apr 2012 08:37:02 GMT

see update above. /k

Re: Filtering and sending only specific data

JPValadas — Thu, 12 Apr 2012 09:47:57 GMT

My last question was:

sending to null queue in forwarder will not count? in size?

My afraid is that sending too a Queue even if a Null Queue that will be taking space and exceed the limit in forwarder right? because there is data in a Queue and i only have 500 mgs.

What i was trying was, in forwarder he see the data that i want and send it to indexer, and trash data will not put in queues, and i will not have problems with space.

Re: Filtering and sending only specific data

kristian_kolb — Thu, 12 Apr 2012 10:25:58 GMT

see update above. /k