topic Re: Extract Key=value before indexing , and index only the extracted key/values in Splunk Search

Extract Key=value before indexing , and index only the extracted key/values

aelnaggar — Tue, 03 Dec 2013 15:06:49 GMT

I have data input which returns key=value delimited with space, so I don't need to index all of them , so how can I index some of them and avoid the others..
Also I want when I search for something only the important key/values to show

Re: Extract Key=value before indexing , and index only the extracted key/values

kristian_kolb — Tue, 03 Dec 2013 16:25:24 GMT

Permanently removing (parts of) event data prior to indexing can be done by means of index-time transformations or SEDCMD, read more here;

http://docs.splunk.com/Documentation/Splunk/6.0/Data/Anonymizedatausingconfigurationfiles

The definition of 'important' is hard for anyone but you to make. But changing the search mode might be what you're after;

http://docs.splunk.com/Documentation/Splunk/6.0/Search/Changethesearchmode

This will control how fields will be extracted, if at all. You can probably do this in a more manual fashion, by setting KV_MODE=none for your sourcetype, and making explicit EXTRACTs;

http://docs.splunk.com/Documentation/Splunk/6.0/Knowledge/Createandmaintainsearch-timefieldextractionsthroughconfigurationfiles#Steps_for_defining_basic_search-time_field_extractions_with_props.conf

http://docs.splunk.com/Documentation/Splunk/6.0/admin/Propsconf

Re: Extract Key=value before indexing , and index only the extracted key/values

aelnaggar — Mon, 09 Dec 2013 11:28:37 GMT

Hi Kristian, thanks for your answer , the main goal here is to exclude unneeded data from being stored in splunk, so I needed to store only the important one to me..

so thinking about how to do this , and extract some parts of the incoming messages to Splunk store it and indexing it..

still your answer valid in that case ?

Re: Extract Key=value before indexing , and index only the extracted key/values

aelnaggar — Mon, 09 Dec 2013 11:28:45 GMT

Hi Kristian, thanks for your answer , the main goal here is to exclude unneeded data from being stored in splunk, so I needed to store only the important one to me..

so thinking about how to do this , and extract some parts of the incoming messages to Splunk store it and indexing it..

still your answer valid in that case ?

Re: Extract Key=value before indexing , and index only the extracted key/values

aelnaggar — Mon, 09 Dec 2013 11:28:53 GMT

Hi Kristian, thanks for your answer , the main goal here is to exclude unneeded data from being stored in splunk, so I needed to store only the important one to me..

so thinking about how to do this , and extract some parts of the incoming messages to Splunk store it and indexing it..

still your answer valid in that case ?

Re: Extract Key=value before indexing , and index only the extracted key/values

sc0tt — Mon, 09 Dec 2013 15:06:42 GMT

I recently filtered unwanted data at index time by using the filter and route method. I think this will allow you to accomplish what you need.

Re: Extract Key=value before indexing , and index only the extracted key/values

kristian_kolb — Tue, 10 Dec 2013 06:48:06 GMT

The first link in my answer above, demonstrate a few options for removing unwanted data from within each event prior to indexing.

The link provided by @sc0tt in his answer shows how to discard/keep whole events based on individual event content.