I have a log file with the following lines;
2019/07/08 11:40:01 mess5 list_frozen_.sh mess5b stream 125 is Frozen.
2019/07/08 11:40:01 mess5 list_frozen_.sh mess5b stream 126 is Frozen.
2019/07/08 11:40:01 mess5 list_frozen_.sh mess5b stream 514 is Frozen.
my props.conf looks like this;
[source::/logs/Alerts_.log]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ^\d{4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2},
MAX_TIMESTAMP_LOOKAHEAD = 19
TIME_PREFIX = ^
TIME_FORMAT = %Y/%m/%d %H:%M:%S
MAX_EVENTS = 10000
TRUNCATE = 0
however my searches return the lines unsplit.
is this due to the lines being almost identicle in the search we have used mvexpand to get round this problem, however i would like to resolve this at the indexers.
any help much would be much appreciated.
Even if you get yours to work, throw it away and use this because it is more efficient:
[source::/logs/Alerts_.log]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 19
TIME_PREFIX = ^
TIME_FORMAT = %Y/%m/%d %H:%M:%S
MAX_EVENTS = 10000
TRUNCATE = 0
Even if you get yours to work, throw it away and use this because it is more efficient:
[source::/logs/Alerts_.log]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 19
TIME_PREFIX = ^
TIME_FORMAT = %Y/%m/%d %H:%M:%S
MAX_EVENTS = 10000
TRUNCATE = 0
Also, I would use a sourcetype
-based stanza header, instead of your source
-based one.
He does, but as you can see in his latest comments, he needed to override that for a specific source.
correct, this is an over-ride as the date format is different in this log
Agree, using LINE_BREAKER (with perhaps a slightly more specific linebreaker than this) is the better choice.
And you can also make that work with both formats:
LINE_BREAKER = ([\r\n]+)\d{2,4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2}
Thanks woodcock this worked.
True, but I am presuming that the events are as presented: 1 line = 1 event. If there are multi-line events, then, yes, use the LINE_BREAKER
that @FrankVl provided.
additional information
This is a source of a sourcetype that is already declared in props.conf
i dont know if that is causing an issue?
This log has a different date to the other logs in the sourcetype, hence a new entry.
[mess5]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ^\d{2}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2}\s
MAX_TIMESTAMP_LOOKAHEAD = 17
TIME_PREFIX = ^
TIME_FORMAT = %d/%m/%y %H:%M:%S
MAX_EVENTS = 10000
You have a ,
behind the BREAK_ONLY_BEFORE
regex. If that is there in your actual config file, that doesn't match your events, so it doesn't break.
Hi FrankVI,
That was a typo. Good spot!
A typo in your question, or in your config? In other words, did this resolve your problem?
no this did not solve the problem
This is a source of a sourcetype that is already declared in props.conf
i dont know if that is causing an issue?
This log has a different date to the other logs in the sourcetype, hence a new entry.
[mess5]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ^\d{2}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2}\s
MAX_TIMESTAMP_LOOKAHEAD = 17
TIME_PREFIX = ^
TIME_FORMAT = %d/%m/%y %H:%M:%S
MAX_EVENTS = 10000
Well, in theory source based settings should override sourcetype based settings. So that should work. Are you sure the source value you use accurately matches the source value on the events?