Splunk Search

Index selected events of log file

carljohan
Path Finder

I have a log file namned: wrapper.log
This log file has two different type of events defined with the prefix INFO or ERROR.
I want to index only the ERROR events but am not getting it to work.
Im on a mac.

Here is my log file:

ERROR | jvm 1 | 2013/05/03 10:47:52 Test_error

INFO | jvm 1 | 2013/05/03 10:48:52 Test

ERROR | jvm 1 | 2013/05/03 10:49:52 Test_error

INFO | jvm 1 | 2013/05/03 10:50:52 Test

ERROR | jvm 1 | 2013/05/03 10:51:52 Test_error

INFO | jvm 1 | 2013/05/03 10:52:52 Test

ERROR | jvm 1 | 2013/05/03 10:53:52 Test_error

inputs.conf:

[monitor:///Users/carljohan/logs/wrapper.log]

disabled=false

sourcetype = ESB_Wrapper

props.conf:

[ESB_Wrapper]

SHOULD_LINEMERGE=false

LINE_BREAKER = ([\r\n]+)[0-9]+-[0-9]+-[0-9]+\s+

TIME_FORMAT = %Y/%m/%d %H:%M:%S,%3N

TRANSFORMS-set= setnull,setparsing

tranfsforms.conf

[setnull]

REGEX = .

DEST_KEY = queue

FORMAT = nullQueue

[setparsing]

REGEX = (\W|^)ERROR(\W|$)

DEST_KEY = queue

FORMAT = indexQueue

With this setup all events are still being indexed.
What am I doing wrong?

Tags (2)
0 Karma
1 Solution

lguinn2
Legend

I think that your LINE_BREAKER may be causing part of the problem. You should not need a LINE_BREAKER if you have set SHOULD_LINEMERGE to false

props.conf:

[ESB_Wrapper]
SHOULD_LINEMERGE=false
TIME_FORMAT = %Y/%m/%d %H:%M:%S,%3N
TRANSFORMS-set= setnull,setparsing

Finally, this REGEX should be sufficient.

REGEX = ^ERROR

You don't need to match the whole event. So a trailing '.*' is unnecessary and more expensive.

ONE MORE THING:

Where are your props.conf and transforms.conf? They need to be located where the data is parsed - usually the indexer(s). And, you will need to restart the indexer(s) to make these changes effective. And, your parsing changes are not retroactive - they will only affect new data as it is parsed.

View solution in original post

lguinn2
Legend

I think that your LINE_BREAKER may be causing part of the problem. You should not need a LINE_BREAKER if you have set SHOULD_LINEMERGE to false

props.conf:

[ESB_Wrapper]
SHOULD_LINEMERGE=false
TIME_FORMAT = %Y/%m/%d %H:%M:%S,%3N
TRANSFORMS-set= setnull,setparsing

Finally, this REGEX should be sufficient.

REGEX = ^ERROR

You don't need to match the whole event. So a trailing '.*' is unnecessary and more expensive.

ONE MORE THING:

Where are your props.conf and transforms.conf? They need to be located where the data is parsed - usually the indexer(s). And, you will need to restart the indexer(s) to make these changes effective. And, your parsing changes are not retroactive - they will only affect new data as it is parsed.

linu1988
Champion

why not ERROR.+ only?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Mind trying this for REGEX

"(?m)^ERROR.*"

0 Karma

lukejadamec
Super Champion

Run a search that pulls the logs listed above, and test the regex like this:
| regex ^ERROR
It should only show log entries that start with ERROR. If it does not, adjust the regex.

0 Karma

carljohan
Path Finder

They are indexed on the local splunk instance. No forwarders are included in the setup.

0 Karma

Ayn
Legend

Are you indexing these events on a local Splunk instance or are you forwarding these from your machine to a separate indexer?

0 Karma

carljohan
Path Finder

I tried with ^ERROR which gives me the following transforms.conf and restarted Splunk but it did not work.

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^ERROR
DEST_KEY = queue
FORMAT = indexQueue

0 Karma

lukejadamec
Super Champion

I'm not a regex wizard, but I should think ^ERROR should work. You will need to restart splunkd on the indexer for the change to take effect.

0 Karma

carljohan
Path Finder

I posted the complete .conf content.
What should I change in the REGEX?

0 Karma

lukejadamec
Super Champion

The configs look good except for the REGEX. Is what you posted missing characters?

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...