topic Filtering fields in log before forward to indexing in Getting Data In

Filtering fields in log before forward to indexing

SplunkCSIT — Fri, 14 Feb 2014 03:50:59 GMT

Hi,
If i need to filtering some data in the log before forward to indexing, how to go abt doing it? thks

Re: Filtering fields in log before forward to indexing

dshpritz — Fri, 14 Feb 2014 05:35:34 GMT

If you want to avoid indexing part of an event, you should look into using a SED entry in your props.conf file. You would need a regular expression matching the portion of the data you want to remove, then in your props.conf, you have an entry like:

[mysourcetype]
SED-remove_data = s/<your regex here>//g

If you want that to occur before the data is sent over the network, that would need to be done on a heavy forwarder, or another parsing system. Note that the SED entry is done at index time, so it would need to be on your indexers or other parsing systems.

HTH,

Dave

Re: Filtering fields in log before forward to indexing

SplunkCSIT — Sat, 15 Feb 2014 03:58:19 GMT

If i want to forward the below xml file to index, but before forward to index, i want the data field and values (this is the data i want only)to be forwarded and the rest to be filter off, how to configure in the props.conf? thks

file.xml
testing
focusing on the inputs
this is the data i want only
this problem

Re: Filtering fields in log before forward to indexing

SplunkCSIT — Mon, 17 Feb 2014 07:07:53 GMT

file.xml testing focusing on the inputs this is the data i want only this problem

Re: Filtering fields in log before forward to indexing

dshpritz — Mon, 17 Feb 2014 07:43:30 GMT

In this case, you can run two SED commands, one to stip away what is in front of and one for after :

SED-remove_before = s/(?s).*(?=)//g
SED-remove_after = s/(?s)(?<=).*//g

Re: Filtering fields in log before forward to indexing

SplunkCSIT — Tue, 18 Feb 2014 13:07:10 GMT

thks for the valuable info, where will SED command be implement, at forwarder or the indexer and at which file? thks

Re: Filtering fields in log before forward to indexing

MuS — Tue, 18 Feb 2014 14:19:22 GMT

read Dave's answer and you find every information you need 😉

Re: Filtering fields in log before forward to indexing

thesteve — Tue, 18 Feb 2014 14:54:08 GMT

There's a page of documentation dedicated to just this question: http://docs.splunk.com/Documentation/Splunk/6.0.1/Forwarding/Routeandfilterdatad

If you want to filter before forwarding, you have to use a heavy forwarder. The reason for this is that a light forwarder doesn't use transforms.conf and therefore does very limited parsing of the source data before sending it on.

Discard specific events and keep the rest

This example discards all sshd events in /var/log/messages by sending them to nullQueue:

In props.conf, set the TRANSFORMS-null attribute:

[source::/var/log/messages]
TRANSFORMS-null= setnull
Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":

[setnull]
REGEX = [sshd]
DEST_KEY = queue
FORMAT = nullQueue

That does it.

Keep specific events and discard the rest

Here's the opposite scenario. In this example, you use two transforms to keep only the sshd events. One transform routes sshd events to indexQueue, while another routes all other events to nullQueue.

Note: In this example, the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.

In props.conf:

[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing
In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = [sshd]
DEST_KEY = queue
FORMAT = indexQueue

Alternatively, you can use SED command processing to replace all text in a given line with nothing (s/something//). Splunk doesn't support SED delete line commands, but it shouldn't index blank lines, so replacing a line of text with nothing should do it.

Re: Filtering fields in log before forward to indexing

dshpritz — Tue, 18 Feb 2014 17:32:29 GMT

The SED command goes into the props.conf file on the first parsing system (indexer or heavy forwarder).

Re: Filtering fields in log before forward to indexing

dshpritz — Tue, 18 Feb 2014 17:34:03 GMT

Good answer, but I'm not sure it fits the use case. He doesn't want to filter entire events (nullQueue) but wants to strip out most of the event, and leave parts.

Re: Filtering fields in log before forward to indexing

mmsull4 — Mon, 28 Sep 2020 18:38:01 GMT

Sorry for these silly question, I'm new to SED modification. 1) This doesn't seem to be working for me (I'm using splunk 6.2 on Windows) which leads me to 2) where do I find doc on remove_data, remove_before or remove_after?

Re: Filtering fields in log before forward to indexing

mdzmuran — Mon, 19 Jun 2017 11:54:38 GMT

Hi. Are these features also available in Splunk Light (version 6.5.1)?