topic Re: Filter data and Extract field before indexed in Getting Data In

Filter data and Extract field before indexed

gurveer_singh88 — Fri, 19 May 2017 11:09:13 GMT

To avoid over index usage, I want to filter before it indexed and I also want extract field before indexed as well.
As of now, I am trying to configure extract field, so later based on field values I can filter data.

Following is my configuration, I don't why props.conf and transforms.conf isn't working.

Configuration on Splunk Universal Forward is as follow:

[monitor://C:\myfoler\mylogfile.log]
disabled = false 
index = test_index
sourcetype = Logfile

Splunk Indexer configuration is as follow:

cat $SPLUNK_HOME/etc/system/local/props.conf

[source:://C:\\myfoler\\mylogfile.log]
TRANSFORMS-MyLOG = MyLOG

cat $SPLUNK_HOME/etc/system/local/transforms.conf

[MyLOG] 
DEST_KEY = _raw 
REGEX = (?<MyType>\w+)\s+\|\s+jvm\s+1\s+\|\s+(?<Date>\d+\/\d+\/\d+)\s+(?<Time>\d+\:\d+\:\d+)\s+\|\s+(?<EventTimeStamp>.*)-\s+(?<Message>.*)
FORMAT = $1 $2 $3 $4 $5

Regex is working perfectly fine when I use this in splunk search.

Please also guide, how I can filter data, for example
if fieldx == Value1 // Index this data
if fieldx == Value2 // Drop this data

Thanks

Re: Filter data and Extract field before indexed

davebrooking — Fri, 19 May 2017 13:38:09 GMT

I think your main problem is that you are trying to do this on the UF. It needs to be done using a heavy forwarder or on the indexer.

For filtering take a look at Filter event data and send to queues in the documentation

Dave

Re: Filter data and Extract field before indexed

gurveer_singh88 — Mon, 22 May 2017 04:21:31 GMT

Thanks for your response.

Using UF, I am just forwarding logs.
props.conf and transforms.conf are placed on indexer (Splunk Enterprise).

Re: Filter data and Extract field before indexed

woodcock — Fri, 26 May 2017 01:56:33 GMT

Use this instead:

[source:://C:\\myfoler\\mylogfile.log] 
SEDCMD-trim_raw = s%\(\w+\)\s+\|\s+jvm\s+1\s+\|\s+\(\d+\/\d+\/\d+\)\s+\(\d+\:\d+\:\d+\)\s+\|\s+\(.*\)-\s+\(.*\)%\1 \2 \3 \4 \5%

Re: Filter data and Extract field before indexed

gurveer_singh88 — Fri, 26 May 2017 06:32:13 GMT

I used following configuration but I was getting error when i restart Splunk Enterprise.

cat etc/system/local/transforms.conf
[MyLOG]
DEST_KEY = _raw
SEDCMD-trim_raw = s%\(\w+\)\s+\|\s+jvm\s+1\s+\|\s+\(\d+\/\d+\/\d+\)\s+\(\d+\:\d+\:\d+\)\s+\|\s+\(.*\)-\s+\(.*\)%\1 \2 \3 \4 \5%
FORMAT = $MyType $Data $Time $EventTimeStamp $Message

Error is as following:
Invalid key in stanza [fMyLOG] in $SPLUNK_HOME/etc/system/local/transforms.conf, line 3: SEDCMD-trim_raw  (value:  s%\(\w+\)\s+\|\s+jvm\s+1\s+\|\s+\(\d+\/\d+\/\d+\)\s+\(\d+\:\d+\:\d+\)\s+\|\s+\(.*\)-\s+\(.*\)%\1 \2 \3 \4 \5%).

Am I doing something wrong here?

Re: Filter data and Extract field before indexed

woodcock — Sat, 27 May 2017 14:24:29 GMT

Yes, you do not need the DEST_KEY, nor the FORMAT; you only need the single SEDCMD line under the stanza header and it needs to be in props.conf, not transforms.conf. It needs to be deployed to your Indexers and all Splunk instances restarted there. This will only effect events which are indexed AFTER the restarts.