topic Re: Index selected events of log file in Splunk Search

Index selected events of log file

carljohan — Mon, 28 Sep 2020 15:16:40 GMT

I have a log file namned: wrapper.log
This log file has two different type of events defined with the prefix INFO or ERROR.
I want to index only the ERROR events but am not getting it to work.
Im on a mac.

Here is my log file:

ERROR | jvm 1 | 2013/05/03 10:47:52 Test_error

INFO | jvm 1 | 2013/05/03 10:48:52 Test

ERROR | jvm 1 | 2013/05/03 10:49:52 Test_error

INFO | jvm 1 | 2013/05/03 10:50:52 Test

ERROR | jvm 1 | 2013/05/03 10:51:52 Test_error

INFO | jvm 1 | 2013/05/03 10:52:52 Test

ERROR | jvm 1 | 2013/05/03 10:53:52 Test_error

inputs.conf:

[monitor:///Users/carljohan/logs/wrapper.log]

disabled=false

sourcetype = ESB_Wrapper

props.conf:

[ESB_Wrapper]

SHOULD_LINEMERGE=false

LINE_BREAKER = ([\r\n]+)[0-9]+-[0-9]+-[0-9]+\s+

TIME_FORMAT = %Y/%m/%d %H:%M:%S,%3N

TRANSFORMS-set= setnull,setparsing

tranfsforms.conf

[setnull]

REGEX = .

DEST_KEY = queue

FORMAT = nullQueue

[setparsing]

REGEX = (\W|^)ERROR(\W|$)

DEST_KEY = queue

FORMAT = indexQueue

With this setup all events are still being indexed.
What am I doing wrong?

Re: Index selected events of log file

lukejadamec — Thu, 14 Nov 2013 13:57:16 GMT

The configs look good except for the REGEX. Is what you posted missing characters?

Re: Index selected events of log file

carljohan — Thu, 14 Nov 2013 13:57:27 GMT

I posted the complete .conf content.
What should I change in the REGEX?

Re: Index selected events of log file

lukejadamec — Thu, 14 Nov 2013 14:06:44 GMT

I'm not a regex wizard, but I should think ^ERROR should work. You will need to restart splunkd on the indexer for the change to take effect.

Re: Index selected events of log file

carljohan — Thu, 14 Nov 2013 14:12:24 GMT

I tried with ^ERROR which gives me the following transforms.conf and restarted Splunk but it did not work.

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^ERROR
DEST_KEY = queue
FORMAT = indexQueue

Re: Index selected events of log file

Ayn — Thu, 14 Nov 2013 14:33:19 GMT

Are you indexing these events on a local Splunk instance or are you forwarding these from your machine to a separate indexer?

Re: Index selected events of log file

carljohan — Thu, 14 Nov 2013 14:34:14 GMT

They are indexed on the local splunk instance. No forwarders are included in the setup.

Re: Index selected events of log file

lukejadamec — Thu, 14 Nov 2013 14:42:34 GMT

Run a search that pulls the logs listed above, and test the regex like this:
| regex ^ERROR
It should only show log entries that start with ERROR. If it does not, adjust the regex.

Re: Index selected events of log file

somesoni2 — Thu, 14 Nov 2013 17:00:36 GMT

Mind trying this for REGEX

"(?m)^ERROR.*"

Re: Index selected events of log file

linu1988 — Thu, 14 Nov 2013 17:19:31 GMT

why not ERROR.+ only?

Re: Index selected events of log file

lguinn2 — Thu, 14 Nov 2013 20:06:05 GMT

I think that your LINE_BREAKER may be causing part of the problem. You should not need a LINE_BREAKER if you have set SHOULD_LINEMERGE to false

props.conf:

[ESB_Wrapper]
SHOULD_LINEMERGE=false
TIME_FORMAT = %Y/%m/%d %H:%M:%S,%3N
TRANSFORMS-set= setnull,setparsing

Finally, this REGEX should be sufficient.

REGEX = ^ERROR

You don't need to match the whole event. So a trailing '.*' is unnecessary and more expensive.

ONE MORE THING:

Where are your props.conf and transforms.conf? They need to be located where the data is parsed - usually the indexer(s). And, you will need to restart the indexer(s) to make these changes effective. And, your parsing changes are not retroactive - they will only affect new data as it is parsed.