Filtering fields in log before forward to indexing

[SplunkCSIT]

Hi,
If i need to filtering some data in the log before forward to indexing, how to go abt doing it? thks

[thesteve]

There's a page of documentation dedicated to just this question: http://docs.splunk.com/Documentation/Splunk/6.0.1/Forwarding/Routeandfilterdatad

If you want to filter before forwarding, you have to use a heavy forwarder. The reason for this is that a light forwarder doesn't use transforms.conf and therefore does very limited parsing of the source data before sending it on.

Discard specific events and keep the rest

This example discards all sshd events in /var/log/messages by sending them to nullQueue:

1. In props.conf, set the TRANSFORMS-null attribute:

   [source::/var/log/messages]
   TRANSFORMS-null= setnull

2. Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":

   [setnull]
   REGEX = [sshd]
   DEST_KEY = queue
   FORMAT = nullQueue

That does it.

Keep specific events and discard the rest

Here's the opposite scenario. In this example, you use two transforms to keep only the sshd events. One transform routes sshd events to indexQueue, while another routes all other events to nullQueue.

Note: In this example, the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.

1. In props.conf:

   [source::/var/log/messages]
   TRANSFORMS-set= setnull,setparsing

2. In transforms.conf:

   [setnull]
   REGEX = .
   DEST_KEY = queue
   FORMAT = nullQueue

   [setparsing]
   REGEX = [sshd]
   DEST_KEY = queue
   FORMAT = indexQueue

Alternatively, you can use SED command processing to replace all text in a given line with nothing (s/something//). Splunk doesn't support SED delete line commands, but it shouldn't index blank lines, so replacing a line of text with nothing should do it.

[mdzmuran]

Hi. Are these features also available in Splunk Light (version 6.5.1)?

[dshpritz]

Good answer, but I'm not sure it fits the use case. He doesn't want to filter entire events (nullQueue) but wants to strip out most of the event, and leave parts.

[dshpritz]

If you want to avoid indexing part of an event, you should look into using a SED entry in your props.conf file. You would need a regular expression matching the portion of the data you want to remove, then in your props.conf, you have an entry like:

[mysourcetype]
SED-remove_data = s/<your regex here>//g

If you want that to occur before the data is sent over the network, that would need to be done on a heavy forwarder, or another parsing system. Note that the SED entry is done at index time, so it would need to be on your indexers or other parsing systems.

HTH,

Dave

[dshpritz]

The SED command goes into the props.conf file on the first parsing system (indexer or heavy forwarder).

[MuS]

read Dave's answer and you find every information you need 😉

[SplunkCSIT]

thks for the valuable info, where will SED command be implement, at forwarder or the indexer and at which file? thks

[dshpritz]

In this case, you can run two SED commands, one to stip away what is in front of and one for after :

SED-remove_before = s/(?s).*(?=)//g
SED-remove_after = s/(?s)(?<=).*//g

[mmsull4]

Sorry for these silly question, I'm new to SED modification. 1) This doesn't seem to be working for me (I'm using splunk 6.2 on Windows) which leads me to 2) where do I find doc on remove_data, remove_before or remove_after?

[SplunkCSIT]

If i want to forward the below xml file to index, but before forward to index, i want the data field and values (this is the data i want only)to be forwarded and the rest to be filter off, how to configure in the props.conf? thks

file.xml testing focusing on the inputs this is the data i want only this problem

[SplunkCSIT]

If i want to forward the below xml file to index, but before forward to index, i want the data field and values (this is the data i want only)to be forwarded and the rest to be filter off, how to configure in the props.conf? thks

file.xml
testing
focusing on the inputs
this is the data i want only
this problem