Unable to line break

[ssaenger]

I have a log file with the following lines;
2019/07/08 11:40:01 mess5 list_frozen_.sh mess5b stream 125 is Frozen.
2019/07/08 11:40:01 mess5 list_frozen_.sh mess5b stream 126 is Frozen.
2019/07/08 11:40:01 mess5 list_frozen_.sh mess5b stream 514 is Frozen.

my props.conf looks like this;
[source::/logs/Alerts_.log]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ^\d{4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2},
MAX_TIMESTAMP_LOOKAHEAD = 19
TIME_PREFIX = ^
TIME_FORMAT = %Y/%m/%d %H:%M:%S
MAX_EVENTS = 10000
TRUNCATE = 0

however my searches return the lines unsplit.
is this due to the lines being almost identicle in the search we have used mvexpand to get round this problem, however i would like to resolve this at the indexers.

any help much would be much appreciated.

[woodcock · Accepted solution]

Even if you get yours to work, throw it away and use this because it is more efficient:

[source::/logs/Alerts_.log]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 19
TIME_PREFIX = ^
TIME_FORMAT = %Y/%m/%d %H:%M:%S
MAX_EVENTS = 10000
TRUNCATE = 0

[woodcock]

Also, I would use a sourcetype-based stanza header, instead of your source-based one.

[FrankVl]

He does, but as you can see in his latest comments, he needed to override that for a specific source.

[ssaenger]

correct, this is an over-ride as the date format is different in this log

[FrankVl]

Agree, using LINE_BREAKER (with perhaps a slightly more specific linebreaker than this) is the better choice.

And you can also make that work with both formats:

LINE_BREAKER = ([\r\n]+)\d{2,4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2}

[ssaenger]

Thanks woodcock this worked.

[woodcock]

True, but I am presuming that the events are as presented: 1 line = 1 event. If there are multi-line events, then, yes, use the LINE_BREAKER that @FrankVl provided.

[ssaenger]

additional information

This is a source of a sourcetype that is already declared in props.conf
i dont know if that is causing an issue?
This log has a different date to the other logs in the sourcetype, hence a new entry.

[mess5]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ^\d{2}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2}\s
MAX_TIMESTAMP_LOOKAHEAD = 17
TIME_PREFIX = ^
TIME_FORMAT = %d/%m/%y %H:%M:%S
MAX_EVENTS = 10000

[FrankVl]

You have a , behind the BREAK_ONLY_BEFORE regex. If that is there in your actual config file, that doesn't match your events, so it doesn't break.

[ssaenger]

Hi FrankVI,

That was a typo. Good spot!

[FrankVl]

A typo in your question, or in your config? In other words, did this resolve your problem?

[ssaenger]

no this did not solve the problem

[ssaenger]

This is a source of a sourcetype that is already declared in props.conf
i dont know if that is causing an issue?
This log has a different date to the other logs in the sourcetype, hence a new entry.

[mess5]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ^\d{2}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2}\s
MAX_TIMESTAMP_LOOKAHEAD = 17
TIME_PREFIX = ^
TIME_FORMAT = %d/%m/%y %H:%M:%S
MAX_EVENTS = 10000

[FrankVl]

Well, in theory source based settings should override sourcetype based settings. So that should work. Are you sure the source value you use accurately matches the source value on the events?