Getting Data In

Filtering and sending only specific data

JPValadas
Explorer

Hi again,

I got one question in filtering and routing to indexer.

i got my props like this:


pros.conf

[WinEventLog:System]

TRANSFORMS-wmi = WantedEventCodes


outputs.conf

[tcpout]

defaultGroup = splunk.host.pt_9997

[tcpout:splunk.host.pt_9997]

autoLB = true

server = splunk.host.pt:9997


transforms.conf

[WantedEventCodes]

REGEX = (?msi)^EventCode=(7040)

DEST_KEY= _TCP_ROUTING

FORMAT = splunk.host.pt_9997


My question is, my forwarder still sending everything to my inxeder, but shouldnt he filter and send only my event code 7040 to indexer?

Thank you.

Tags (3)
0 Karma

kristian_kolb
Ultra Champion

I think that you have a default configuration of sending logs to splunk.host.pt:9997, and then you specify that you want to send a subset of these (those matching your EventCode regex) to ... the same place.

Didn't the nullQueue config work out for you?
http://splunk-base.splunk.com/answers/44583/splunk-filter-on-windows


UPDATE:

Is there perhaps an element of confusion here?

On a heavy forwarder (or indexer), you can use the nullQueue approach to stop some events from being indexed by dropping them, according to your regex.

The _TCP_ROUTING approach can be used when you normally send all your data to one (group of) indexer(s), but want to send some of the data to a different (group of) indexer(s).

Configuring _TCP_ROUTING is not needed for normal forwarding/indexing of events to your ordinary indexer(s).


UPDATE 2:

No. Sending events to the nullQueue will not count towards the license.

If you only have one indexer, you should use the nullQueue to filter out events that you do not want to index. If you have a Heavy forwarder you put your configs there, if you have a Lightweight or Universal forwarder, you do the configuration on the indexer.

/kristian

kristian_kolb
Ultra Champion

see update above. /k

0 Karma

JPValadas
Explorer

My last question was:

sending to null queue in forwarder will not count? in size?

My afraid is that sending too a Queue even if a Null Queue that will be taking space and exceed the limit in forwarder right? because there is data in a Queue and i only have 500 mgs.

What i was trying was, in forwarder he see the data that i want and send it to indexer, and trash data will not put in queues, and i will not have problems with space.

0 Karma

kristian_kolb
Ultra Champion

see update above. /k

0 Karma

jbsplunk
Splunk Employee
Splunk Employee

If you're using a heavy forwarder, then the changes will be made on the forwarder, that is correct. If you're using a heavy forwarder, you can just throw all the events you don't want to nullQueue at the forwarder level, and what's left will be sent on.

I am not sure I understand what you mean by your last question, but the configuration you were previously using sent everything to a single indexer, and then a subset of events to that same indexer, which seems redundant.

0 Karma

JPValadas
Explorer

jbsplunk and Kristian thank you for the answer,

jbsplunk i´m doing this with a heavy forwarder and i should do the changes in forwarder right? not in indexer.

I created that props and transforms on forwarder to make him send only Event Code 7040 to indexer, TCP Routing in forwarder should not send to indexer?

0 Karma

jbsplunk
Splunk Employee
Splunk Employee

You can do this on the indexer with the instructions found here:

http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Routeandfilterdatad

You're looking for the section that says 'Keep specific events and discard the rest'. This is intended for the exact use case you describe, to avoid exceeding the license volume allowed.

0 Karma

JPValadas
Explorer

That worked perfectly, but the qeues was on forwarder.

Now i want to send the data from hosts to indexer, but i want to specify what kind of data do i want to send to Indexer.

In this way i will not have problems with exceding space limits (500megas)

In other example i was filtering but not sending data do indexer now i´m trying to use TCP_ROUTING to send data to indexer.

0 Karma

jbsplunk
Splunk Employee
Splunk Employee

Kristian is correct, I think, with his assessment of the configuration. This configuration would forward everything to splunk.host.pt:9997. Additionally, this configuration would only a subset of events from a heavy forwarder to an indexer, or from an indexer to another indexer. If you're trying to do this on a Universal/Lightweight forwarder, it isn't going to work because no data is parsed on those instances.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...