Filter data and Extract field before indexed

[gurveer_singh88]

To avoid over index usage, I want to filter before it indexed and I also want extract field before indexed as well.
As of now, I am trying to configure extract field, so later based on field values I can filter data.

Following is my configuration, I don't why props.conf and transforms.conf isn't working.

Configuration on Splunk Universal Forward is as follow:

[monitor://C:\myfoler\mylogfile.log]
disabled = false 
index = test_index
sourcetype = Logfile

Splunk Indexer configuration is as follow:

cat $SPLUNK_HOME/etc/system/local/props.conf

[source:://C:\\myfoler\\mylogfile.log]
TRANSFORMS-MyLOG = MyLOG

cat $SPLUNK_HOME/etc/system/local/transforms.conf

[MyLOG] 
DEST_KEY = _raw 
REGEX = (?<MyType>\w+)\s+\|\s+jvm\s+1\s+\|\s+(?<Date>\d+\/\d+\/\d+)\s+(?<Time>\d+\:\d+\:\d+)\s+\|\s+(?<EventTimeStamp>.*)-\s+(?<Message>.*)
FORMAT = $1 $2 $3 $4 $5

Regex is working perfectly fine when I use this in splunk search.

Please also guide, how I can filter data, for example
if fieldx == Value1 // Index this data
if fieldx == Value2 // Drop this data

Thanks

[woodcock]

Use this instead:

[source:://C:\\myfoler\\mylogfile.log] 
SEDCMD-trim_raw = s%\(\w+\)\s+\|\s+jvm\s+1\s+\|\s+\(\d+\/\d+\/\d+\)\s+\(\d+\:\d+\:\d+\)\s+\|\s+\(.*\)-\s+\(.*\)%\1 \2 \3 \4 \5%

[gurveer_singh88]

I used following configuration but I was getting error when i restart Splunk Enterprise.

cat etc/system/local/transforms.conf
[MyLOG]
DEST_KEY = _raw
SEDCMD-trim_raw = s%\(\w+\)\s+\|\s+jvm\s+1\s+\|\s+\(\d+\/\d+\/\d+\)\s+\(\d+\:\d+\:\d+\)\s+\|\s+\(.*\)-\s+\(.*\)%\1 \2 \3 \4 \5%
FORMAT = $MyType $Data $Time $EventTimeStamp $Message

Error is as following:
Invalid key in stanza [fMyLOG] in $SPLUNK_HOME/etc/system/local/transforms.conf, line 3: SEDCMD-trim_raw  (value:  s%\(\w+\)\s+\|\s+jvm\s+1\s+\|\s+\(\d+\/\d+\/\d+\)\s+\(\d+\:\d+\:\d+\)\s+\|\s+\(.*\)-\s+\(.*\)%\1 \2 \3 \4 \5%).

Am I doing something wrong here?

[woodcock]

Yes, you do not need the DEST_KEY, nor the FORMAT; you only need the single SEDCMD line under the stanza header and it needs to be in props.conf, not transforms.conf. It needs to be deployed to your Indexers and all Splunk instances restarted there. This will only effect events which are indexed AFTER the restarts.

[davebrooking]

I think your main problem is that you are trying to do this on the UF. It needs to be done using a heavy forwarder or on the indexer.

For filtering take a look at Filter event data and send to queues in the documentation

Dave

[gurveer_singh88]

Thanks for your response.

Using UF, I am just forwarding logs.
props.conf and transforms.conf are placed on indexer (Splunk Enterprise).