topic Re: Unable to line break in Getting Data In

Unable to line break

ssaenger — Wed, 30 Sep 2020 01:11:37 GMT

I have a log file with the following lines;
2019/07/08 11:40:01 mess5 list_frozen_.sh mess5b stream 125 is Frozen.
2019/07/08 11:40:01 mess5 list_frozen_.sh mess5b stream 126 is Frozen.
2019/07/08 11:40:01 mess5 list_frozen_.sh mess5b stream 514 is Frozen.

my props.conf looks like this;
[source::/logs/Alerts_.log]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ^\d{4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2},
MAX_TIMESTAMP_LOOKAHEAD = 19
TIME_PREFIX = ^
TIME_FORMAT = %Y/%m/%d %H:%M:%S
MAX_EVENTS = 10000
TRUNCATE = 0

however my searches return the lines unsplit.
is this due to the lines being almost identicle in the search we have used mvexpand to get round this problem, however i would like to resolve this at the indexers.

any help much would be much appreciated.

Re: Unable to line break

FrankVl — Mon, 08 Jul 2019 11:58:21 GMT

You have a , behind the BREAK_ONLY_BEFORE regex. If that is there in your actual config file, that doesn't match your events, so it doesn't break.

Re: Unable to line break

ssaenger — Mon, 08 Jul 2019 12:19:29 GMT

Hi FrankVI,

That was a typo. Good spot!

Re: Unable to line break

FrankVl — Mon, 08 Jul 2019 12:20:14 GMT

A typo in your question, or in your config? In other words, did this resolve your problem?

Re: Unable to line break

ssaenger — Mon, 08 Jul 2019 12:24:54 GMT

no this did not solve the problem

Re: Unable to line break

ssaenger — Wed, 30 Sep 2020 01:11:42 GMT

This is a source of a sourcetype that is already declared in props.conf
i dont know if that is causing an issue?
This log has a different date to the other logs in the sourcetype, hence a new entry.

[mess5]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ^\d{2}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2}\s
MAX_TIMESTAMP_LOOKAHEAD = 17
TIME_PREFIX = ^
TIME_FORMAT = %d/%m/%y %H:%M:%S
MAX_EVENTS = 10000

Re: Unable to line break

ssaenger — Wed, 30 Sep 2020 01:11:45 GMT

additional information

This is a source of a sourcetype that is already declared in props.conf
i dont know if that is causing an issue?
This log has a different date to the other logs in the sourcetype, hence a new entry.

Re: Unable to line break

woodcock — Mon, 08 Jul 2019 13:02:39 GMT

Even if you get yours to work, throw it away and use this because it is more efficient:

[source::/logs/Alerts_.log]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 19
TIME_PREFIX = ^
TIME_FORMAT = %Y/%m/%d %H:%M:%S
MAX_EVENTS = 10000
TRUNCATE = 0

Re: Unable to line break

FrankVl — Mon, 08 Jul 2019 13:30:47 GMT

Well, in theory source based settings should override sourcetype based settings. So that should work. Are you sure the source value you use accurately matches the source value on the events?

Re: Unable to line break

FrankVl — Mon, 08 Jul 2019 13:33:07 GMT

Agree, using LINE_BREAKER (with perhaps a slightly more specific linebreaker than this) is the better choice.

And you can also make that work with both formats:

LINE_BREAKER = ([\r\n]+)\d{2,4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2}

Re: Unable to line break

woodcock — Mon, 08 Jul 2019 14:33:13 GMT

True, but I am presuming that the events are as presented: 1 line = 1 event. If there are multi-line events, then, yes, use the LINE_BREAKER that @FrankVl provided.

Re: Unable to line break

woodcock — Mon, 08 Jul 2019 14:34:25 GMT

Also, I would use a sourcetype-based stanza header, instead of your source-based one.

Re: Unable to line break

FrankVl — Mon, 08 Jul 2019 14:52:14 GMT

He does, but as you can see in his latest comments, he needed to override that for a specific source.

Re: Unable to line break

ssaenger — Tue, 09 Jul 2019 07:19:58 GMT

Thanks woodcock this worked.

Re: Unable to line break

ssaenger — Tue, 09 Jul 2019 07:20:40 GMT

correct, this is an over-ride as the date format is different in this log