I have a log file namned: wrapper.log
This log file has two different type of events defined with the prefix INFO or ERROR.
I want to index only the ERROR events but am not getting it to work.
Im on a mac.
Here is my log file:
ERROR | jvm 1 | 2013/05/03 10:47:52 Test_error
INFO | jvm 1 | 2013/05/03 10:48:52 Test
ERROR | jvm 1 | 2013/05/03 10:49:52 Test_error
INFO | jvm 1 | 2013/05/03 10:50:52 Test
ERROR | jvm 1 | 2013/05/03 10:51:52 Test_error
INFO | jvm 1 | 2013/05/03 10:52:52 Test
ERROR | jvm 1 | 2013/05/03 10:53:52 Test_error
inputs.conf:
[monitor:///Users/carljohan/logs/wrapper.log]
disabled=false
sourcetype = ESB_Wrapper
props.conf:
[ESB_Wrapper]
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)[0-9]+-[0-9]+-[0-9]+\s+
TIME_FORMAT = %Y/%m/%d %H:%M:%S,%3N
TRANSFORMS-set= setnull,setparsing
tranfsforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = (\W|^)ERROR(\W|$)
DEST_KEY = queue
FORMAT = indexQueue
With this setup all events are still being indexed.
What am I doing wrong?
I think that your LINE_BREAKER may be causing part of the problem. You should not need a LINE_BREAKER if you have set SHOULD_LINEMERGE to false
props.conf:
[ESB_Wrapper]
SHOULD_LINEMERGE=false
TIME_FORMAT = %Y/%m/%d %H:%M:%S,%3N
TRANSFORMS-set= setnull,setparsing
Finally, this REGEX should be sufficient.
REGEX = ^ERROR
You don't need to match the whole event. So a trailing '.*' is unnecessary and more expensive.
ONE MORE THING:
Where are your props.conf and transforms.conf? They need to be located where the data is parsed - usually the indexer(s). And, you will need to restart the indexer(s) to make these changes effective. And, your parsing changes are not retroactive - they will only affect new data as it is parsed.
I think that your LINE_BREAKER may be causing part of the problem. You should not need a LINE_BREAKER if you have set SHOULD_LINEMERGE to false
props.conf:
[ESB_Wrapper]
SHOULD_LINEMERGE=false
TIME_FORMAT = %Y/%m/%d %H:%M:%S,%3N
TRANSFORMS-set= setnull,setparsing
Finally, this REGEX should be sufficient.
REGEX = ^ERROR
You don't need to match the whole event. So a trailing '.*' is unnecessary and more expensive.
ONE MORE THING:
Where are your props.conf and transforms.conf? They need to be located where the data is parsed - usually the indexer(s). And, you will need to restart the indexer(s) to make these changes effective. And, your parsing changes are not retroactive - they will only affect new data as it is parsed.
why not ERROR.+ only?
Mind trying this for REGEX
"(?m)^ERROR.*"
Run a search that pulls the logs listed above, and test the regex like this:
| regex ^ERROR
It should only show log entries that start with ERROR. If it does not, adjust the regex.
They are indexed on the local splunk instance. No forwarders are included in the setup.
Are you indexing these events on a local Splunk instance or are you forwarding these from your machine to a separate indexer?
I tried with ^ERROR which gives me the following transforms.conf and restarted Splunk but it did not work.
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = ^ERROR
DEST_KEY = queue
FORMAT = indexQueue
I'm not a regex wizard, but I should think ^ERROR should work. You will need to restart splunkd on the indexer for the change to take effect.
I posted the complete .conf content.
What should I change in the REGEX?
The configs look good except for the REGEX. Is what you posted missing characters?