Filtering and sending only specific data

[JPValadas]

Hi again,

I got one question in filtering and routing to indexer.

i got my props like this:

---

pros.conf

[WinEventLog:System]

TRANSFORMS-wmi = WantedEventCodes

---

outputs.conf

[tcpout]

defaultGroup = splunk.host.pt_9997

[tcpout:splunk.host.pt_9997]

autoLB = true

server = splunk.host.pt:9997

---

transforms.conf

[WantedEventCodes]

REGEX = (?msi)^EventCode=(7040)

DEST_KEY= _TCP_ROUTING

FORMAT = splunk.host.pt_9997

---

My question is, my forwarder still sending everything to my inxeder, but shouldnt he filter and send only my event code 7040 to indexer?

Thank you.

[kristian_kolb]

I think that you have a default configuration of sending logs to splunk.host.pt:9997, and then you specify that you want to send a subset of these (those matching your EventCode regex) to ... the same place.

Didn't the nullQueue config work out for you?
http://splunk-base.splunk.com/answers/44583/splunk-filter-on-windows

---

UPDATE:

Is there perhaps an element of confusion here?

On a heavy forwarder (or indexer), you can use the nullQueue approach to stop some events from being indexed by dropping them, according to your regex.

The _TCP_ROUTING approach can be used when you normally send all your data to one (group of) indexer(s), but want to send some of the data to a different (group of) indexer(s).

Configuring _TCP_ROUTING is not needed for normal forwarding/indexing of events to your ordinary indexer(s).

---

UPDATE 2:

No. Sending events to the nullQueue will not count towards the license.

If you only have one indexer, you should use the nullQueue to filter out events that you do not want to index. If you have a Heavy forwarder you put your configs there, if you have a Lightweight or Universal forwarder, you do the configuration on the indexer.

/kristian

[kristian_kolb]

see update above. /k

[JPValadas]

My last question was:

sending to null queue in forwarder will not count? in size?

My afraid is that sending too a Queue even if a Null Queue that will be taking space and exceed the limit in forwarder right? because there is data in a Queue and i only have 500 mgs.

What i was trying was, in forwarder he see the data that i want and send it to indexer, and trash data will not put in queues, and i will not have problems with space.

[kristian_kolb]

see update above. /k

[jbsplunk]

If you're using a heavy forwarder, then the changes will be made on the forwarder, that is correct. If you're using a heavy forwarder, you can just throw all the events you don't want to nullQueue at the forwarder level, and what's left will be sent on.

I am not sure I understand what you mean by your last question, but the configuration you were previously using sent everything to a single indexer, and then a subset of events to that same indexer, which seems redundant.

[JPValadas]

jbsplunk and Kristian thank you for the answer,

jbsplunk i´m doing this with a heavy forwarder and i should do the changes in forwarder right? not in indexer.

I created that props and transforms on forwarder to make him send only Event Code 7040 to indexer, TCP Routing in forwarder should not send to indexer?

[jbsplunk]

You can do this on the indexer with the instructions found here:

http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Routeandfilterdatad

You're looking for the section that says 'Keep specific events and discard the rest'. This is intended for the exact use case you describe, to avoid exceeding the license volume allowed.

[JPValadas]

That worked perfectly, but the qeues was on forwarder.

Now i want to send the data from hosts to indexer, but i want to specify what kind of data do i want to send to Indexer.

In this way i will not have problems with exceding space limits (500megas)

In other example i was filtering but not sending data do indexer now i´m trying to use TCP_ROUTING to send data to indexer.

[jbsplunk]

Kristian is correct, I think, with his assessment of the configuration. This configuration would forward everything to splunk.host.pt:9997. Additionally, this configuration would only a subset of events from a heavy forwarder to an indexer, or from an indexer to another indexer. If you're trying to do this on a Universal/Lightweight forwarder, it isn't going to work because no data is parsed on those instances.