topic Re: Discarding specific event and keeping the rest in Splunk Search

Discarding specific event and keeping the rest

pdash — Tue, 19 Feb 2013 23:27:18 GMT

Below is the raw data that am getting. I want to extract the events where category is Error.
For this am doing this in props.conf.

[source::d:\TGNI\Logs*.log]
TRANSFORMS-null= setnull

And transform.conf:
[setnull]
REGEX = Category: Error
DEST_KEY = queue
FORMAT = nullQueue

Is this the right way to do?

1 » 2/18/13
6:48:54.000 PM

Timestamp: 2/18/2013 6:48:54 PM
Category: Error
Machine: devmundia01
IP Address:
Customer ID:
Request URL:
Referrer URL:
Browser Name:
Browser Version:
User Agent:
Show all 24 lines
host=devmundia01 Options| sourcetype=mundiaerr Options| source=d:\TGNI\Logs\tgni-mundia.2013-02-19.log Options

2 » 2/18/13
6:48:53.000 PM

Timestamp: 2/18/2013 6:48:53 PM
Category: Info
Machine: devmundia01
IP Address: 10.6.8.28 (3yu4xv0x5bbyk5345sqcbegq)
Customer ID:
Request URL: http://10.13.65.105/
Referrer URL:
Browser Name: Jakarta Commons-HttpClient
Browser Version: 0.0
User Agent: Jakarta Commons-HttpClient/3.0.1
Show all 16 lines
host=devmundia01 Options| sourcetype=mundiaerr Options| source=d:\TGNI\Logs\tgni-mundia.2013-02-19.log Options

Re: Discarding specific event and keeping the rest

pdash — Tue, 19 Feb 2013 23:40:45 GMT

Correction: props.conf
[source::d:\TGNI\Logs*.log]
TRANSFORMS-null= setnull

Correction to transforms.conf
[setnull]
REGEX = Category\:\sInfo
DEST_KEY = queue
FORMAT = nullQueue

I cant still discard the info events. Where am i doing mistake?

Re: Discarding specific event and keeping the rest

lguinn2 — Wed, 20 Feb 2013 02:45:03 GMT

Splunk applies your transformation as part of the parsing process - before the raw data is written to the data store. Therefore, your transform will only apply to new data. Data that has already been indexed will not be removed.

You will have to remove the data from the index and reindex it, if you want your changes to apply to existing data.

Second, your props.conf and transforms.conf files must reside on the Splunk server(s) doing the parsing. Usually this means that the .conf files go on the indexer(s). However, if you are using a heavy forwarder, it will be doing the parsing and so the .conf files will go there instead.

Finally, it is possible that you need to update the regex to

REGEX = (?m)Category\:\sInfo