Here is a sampling of the log data. Lines 1-3 had no issues, but line 4 issues a warning and dumps the problem onto line 5. Splunk needs to treat line 5 as part of Line 4 and not index the timestamp for line 5. Suggestions please.
2015-09-11 04:24:05,568 INFO [nexj.model.class.LDAPGRP] (WorkManager(2)-91:dispatcher ) End getSnapshot for LDAPGRP 2015-09-11 04:24:05,568 INFO [nexj.model.class.LDAPUSR] (WorkManager(2)-91:dispatcher ) Start getSnapshot for LDAPUSR. 2015-09-11 04:24:05,977 INFO [nexj.model.class.LDAPUSR] (WorkManager(2)-91:dispatcher ) End getSnapshot for LDAPUSR 2015-09-11 04:42:47,652 WARN [nexj.core.rpc.http.session.SessionManager] (servername%2F-IPAdress:/project-8009-7:/apex) Acquring lock exceeded max wait time of 30720s. Diagnostic dump: 2015-09-10 11:39:47.312: (Thread: servername%2F-IPAdress:/project-8009-15 Owner: Thread[servername%2F-IPAdress:/project-8009-15,5,jboss] Lock Count: 1 Ref Count: 0 Session: 488E74B020DAC9F33CD503F058947C6A.server) Acqured lock
Hope this helps:
2015-09-11 04:24:05,977 INFO :
Give this a try in your sourcetype definition in props.conf (on Indexer/Heavy Forwarder)
[YourSourceType] SHOULD_LINEMERGE=false NO_BINARY_CHECK=true LINE_BREAKER=([\r\n]+)(?=\d+-\d+-\d+ \d+:\d+:\d+,\d+ \w+) TIME_FORMAT=%Y-%m-%d %H:%M:%S,%Q MAX_TIMESTAMP_LOOKAHEAD=30
This did work. Can i get a breakdown of the various lines and what they do?
I get Time_format but the others are unclear.
LINEBREAKER - provides a delimiter which will break input stream into separate events, default is new line.
I've set it to match new line char + string with this format 2015-09-11 04:24:05,568 INFO. Thus excluding intermediate timestamp which doesn't have log level.
SHOULDLINEMERGE=provide where multiple lines should be merged into one events. Since I'm using custom LINEBREAKER, it should be false
MAXTIMESTAMP_LOOKAHEAD=After Timestamp is location, for how many characters the timestamp value should be looked upon. Based on your date (23 char), I just set to near by whole figure of 30.