- Splunk Answers
- :
- Splunk Administration
- :
- Getting Data In
- :
- Re: Why am I seeing inconsistent behavior with BRE...
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Why am I seeing inconsistent behavior with BREAK_ONLY_BEFORE with my sourcetype configuration?
I have a sourcetype of j_out that breaks the lines properly for jboss java log file.
The event breaks here:
60487.098: [Full GC (Ergonomics) there's more after this
My j_out sourcetype configuration is this:
MAX_TIMESTAMP_LOOKAHEAD=30
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=(?i)^\d\d:\d\d:\d\d,\d\d\d|(?i)^\d+\.\d\d\d
TRUNCATE=0
MAX_EVENTS=5000
The logs all start with either one of these:
15:41:41,136 ...
116.624: [Full GC (Metadata GC Threshold) ...
The first is a real time stamp, the second is a second counter since java was started. This would explain my regex above.
Any idea why it would randomly break incorrectly? (inconsistent)
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I would suggest to try following configurations
props.conf on Indexers/Heavy forwarders
[YourSourceType]
MAX_TIMESTAMP_LOOKAHEAD=30
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)(?=(\d+:\d+:\d+,\d+)|(\d+\.\d+))
TRUNCATE=0
MAX_EVENTS=5000
Also, I don't see any proper/fix timestamp for the events, so you can use current time for the events, by adding following attribute
DATETIME_CONFIG=CURRENT
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Why Line_breaker instead? It would truncate the time/second values doing it that way.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I added this and recycled the indexer and I'm still seeing the behavior.
9/18/15 1:00:08.000 AM 86333.133: [Full GC
9/18/15 1:00:12.000 AM [PSYoungGen: 7690K->0K(228352K)] [ParOldGen: 839127K->336242K(1280000K)] 846818K->336242K(1508352K) [PSPermGen: 229705K->211702K(441856K)], 3.9622840 secs] [Times: user=7.84 sys=0.48, real=3.96 secs]
j.out file on the server shows:
86333.133: [Full GC [PSYoungGen: 7690K->0K(228352K)] [ParOldGen: 839127K->336242K(1280000K)] 846818K->336242K(1508352K) [PSPermGen: 229705K->211702K(441856K)], 3.9622840 secs] [Times: user=7.84 sys=0.48, real=3.96 secs]
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This will not truncate as I've put lookup-ahead regex ('?='). Did you get a chance to test it? You can check that in Preview to start with.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Ok, thanks. Im attempting your suggestion. Will wait a day to see if it happens on this indexer and get back to you.
Detecting Remote Code Executions With the Splunk Threat Research Team
Enter the Splunk Community Dashboard Challenge for Your Chance to Win!
.conf24 | Session Scheduler is Live!!
