- Splunk Answers
- :
- Using Splunk
- :
- Splunk Search
- :
- Re: Index selected events of log file
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have a log file namned: wrapper.log
This log file has two different type of events defined with the prefix INFO or ERROR.
I want to index only the ERROR events but am not getting it to work.
Im on a mac.
Here is my log file:
ERROR | jvm 1 | 2013/05/03 10:47:52 Test_error
INFO | jvm 1 | 2013/05/03 10:48:52 Test
ERROR | jvm 1 | 2013/05/03 10:49:52 Test_error
INFO | jvm 1 | 2013/05/03 10:50:52 Test
ERROR | jvm 1 | 2013/05/03 10:51:52 Test_error
INFO | jvm 1 | 2013/05/03 10:52:52 Test
ERROR | jvm 1 | 2013/05/03 10:53:52 Test_error
inputs.conf:
[monitor:///Users/carljohan/logs/wrapper.log]
disabled=false
sourcetype = ESB_Wrapper
props.conf:
[ESB_Wrapper]
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)[0-9]+-[0-9]+-[0-9]+\s+
TIME_FORMAT = %Y/%m/%d %H:%M:%S,%3N
TRANSFORMS-set= setnull,setparsing
tranfsforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = (\W|^)ERROR(\W|$)
DEST_KEY = queue
FORMAT = indexQueue
With this setup all events are still being indexed.
What am I doing wrong?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I think that your LINE_BREAKER may be causing part of the problem. You should not need a LINE_BREAKER if you have set SHOULD_LINEMERGE to false
props.conf:
[ESB_Wrapper]
SHOULD_LINEMERGE=false
TIME_FORMAT = %Y/%m/%d %H:%M:%S,%3N
TRANSFORMS-set= setnull,setparsing
Finally, this REGEX should be sufficient.
REGEX = ^ERROR
You don't need to match the whole event. So a trailing '.*' is unnecessary and more expensive.
ONE MORE THING:
Where are your props.conf and transforms.conf? They need to be located where the data is parsed - usually the indexer(s). And, you will need to restart the indexer(s) to make these changes effective. And, your parsing changes are not retroactive - they will only affect new data as it is parsed.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I think that your LINE_BREAKER may be causing part of the problem. You should not need a LINE_BREAKER if you have set SHOULD_LINEMERGE to false
props.conf:
[ESB_Wrapper]
SHOULD_LINEMERGE=false
TIME_FORMAT = %Y/%m/%d %H:%M:%S,%3N
TRANSFORMS-set= setnull,setparsing
Finally, this REGEX should be sufficient.
REGEX = ^ERROR
You don't need to match the whole event. So a trailing '.*' is unnecessary and more expensive.
ONE MORE THING:
Where are your props.conf and transforms.conf? They need to be located where the data is parsed - usually the indexer(s). And, you will need to restart the indexer(s) to make these changes effective. And, your parsing changes are not retroactive - they will only affect new data as it is parsed.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
why not ERROR.+ only?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Mind trying this for REGEX
"(?m)^ERROR.*"
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Run a search that pulls the logs listed above, and test the regex like this:
| regex ^ERROR
It should only show log entries that start with ERROR. If it does not, adjust the regex.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
They are indexed on the local splunk instance. No forwarders are included in the setup.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Are you indexing these events on a local Splunk instance or are you forwarding these from your machine to a separate indexer?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I tried with ^ERROR which gives me the following transforms.conf and restarted Splunk but it did not work.
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = ^ERROR
DEST_KEY = queue
FORMAT = indexQueue
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm not a regex wizard, but I should think ^ERROR should work. You will need to restart splunkd on the indexer for the change to take effect.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I posted the complete .conf content.
What should I change in the REGEX?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The configs look good except for the REGEX. Is what you posted missing characters?
Troubleshooting the OpenTelemetry Collector
Adoption of Infrastructure Monitoring at Splunk
Modern way of developing distributed application using OTel
