Getting Data In

Unable to line break

Path Finder

I have a log file with the following lines;
2019/07/08 11:40:01 mess5 listfrozen.sh mess5b stream 125 is Frozen.
2019/07/08 11:40:01 mess5 listfrozen.sh mess5b stream 126 is Frozen.
2019/07/08 11:40:01 mess5 listfrozen.sh mess5b stream 514 is Frozen.

my props.conf looks like this;
[source::/logs/Alerts.log]
SHOULD
LINEMERGE = true
BREAKONLYBEFOREDATE = false
BREAK
ONLYBEFORE = ^\d{4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2},
MAX
TIMESTAMPLOOKAHEAD = 19
TIME
PREFIX = ^
TIMEFORMAT = %Y/%m/%d %H:%M:%S
MAX
EVENTS = 10000
TRUNCATE = 0

however my searches return the lines unsplit.
is this due to the lines being almost identicle in the search we have used mvexpand to get round this problem, however i would like to resolve this at the indexers.

any help much would be much appreciated.

0 Karma
1 Solution

Esteemed Legend

Even if you get yours to work, throw it away and use this because it is more efficient:

[source::/logs/Alerts_.log]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 19
TIME_PREFIX = ^
TIME_FORMAT = %Y/%m/%d %H:%M:%S
MAX_EVENTS = 10000
TRUNCATE = 0

View solution in original post

0 Karma

Esteemed Legend

Even if you get yours to work, throw it away and use this because it is more efficient:

[source::/logs/Alerts_.log]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 19
TIME_PREFIX = ^
TIME_FORMAT = %Y/%m/%d %H:%M:%S
MAX_EVENTS = 10000
TRUNCATE = 0

View solution in original post

0 Karma

Esteemed Legend

Also, I would use a sourcetype-based stanza header, instead of your source-based one.

0 Karma

Ultra Champion

He does, but as you can see in his latest comments, he needed to override that for a specific source.

0 Karma

Path Finder

correct, this is an over-ride as the date format is different in this log

0 Karma

Ultra Champion

Agree, using LINE_BREAKER (with perhaps a slightly more specific linebreaker than this) is the better choice.

And you can also make that work with both formats:

LINE_BREAKER = ([\r\n]+)\d{2,4}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2}
0 Karma

Path Finder

Thanks woodcock this worked.

0 Karma

Esteemed Legend

True, but I am presuming that the events are as presented: 1 line = 1 event. If there are multi-line events, then, yes, use the LINE_BREAKER that @FrankVl provided.

Path Finder

additional information

This is a source of a sourcetype that is already declared in props.conf
i dont know if that is causing an issue?
This log has a different date to the other logs in the sourcetype, hence a new entry.

[mess5]
SHOULDLINEMERGE = true
BREAK
ONLYBEFOREDATE = false
BREAKONLYBEFORE = ^\d{2}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2}\s
MAXTIMESTAMPLOOKAHEAD = 17
TIMEPREFIX = ^
TIME
FORMAT = %d/%m/%y %H:%M:%S
MAX_EVENTS = 10000

0 Karma

Ultra Champion

You have a , behind the BREAK_ONLY_BEFORE regex. If that is there in your actual config file, that doesn't match your events, so it doesn't break.

0 Karma

Path Finder

Hi FrankVI,

That was a typo. Good spot!

0 Karma

Ultra Champion

A typo in your question, or in your config? In other words, did this resolve your problem?

0 Karma

Path Finder

no this did not solve the problem

0 Karma

Path Finder

This is a source of a sourcetype that is already declared in props.conf
i dont know if that is causing an issue?
This log has a different date to the other logs in the sourcetype, hence a new entry.

[mess5]
SHOULDLINEMERGE = true
BREAK
ONLYBEFOREDATE = false
BREAKONLYBEFORE = ^\d{2}\/\d{2}\/\d{2}\s\d{2}:\d{2}:\d{2}\s
MAXTIMESTAMPLOOKAHEAD = 17
TIMEPREFIX = ^
TIME
FORMAT = %d/%m/%y %H:%M:%S
MAX_EVENTS = 10000

0 Karma

Ultra Champion

Well, in theory source based settings should override sourcetype based settings. So that should work. Are you sure the source value you use accurately matches the source value on the events?

0 Karma