topic props.conf Line Breaking in Getting Data In

props.conf Line Breaking

Svill321 — Mon, 17 Jul 2017 21:06:56 GMT

Hello everyone,

I have several events with different time stamps that I'm trying to breakup. The props file I'm using is as follows:

sourcetype=applogs
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=false
BREAK_ONLY_BEFORE = ^(\d{2}\:\d{2}\:\d{2}\||\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}|\d\/\d{2}\/\d{4}\s+?\d{2}:\d{2}:\d{2}:)
disabled = false
pulldown_type = true

Alternatively, I also use:

sourcetype=applogs
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)( \d{2}\:\d{2}\:\d{2}\||\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}|\d\/\d{2}\/\d{4}\s+?\d{2}:\d{2}:\d{2}:)
disabled = false
pulldown_type = true

The logs look like this:

14:49:27| stuff
14:49:27|stuff
14:49:28|stuff

Another log looks like
7/17/2017 17:02:12:stuff
7/17/2017 17:02:12:stuff
7/17/2017 17:02:12:stuff

And yet another log looks like this:

7/17/2017 17:01:58.345, stuff
7/17/2017 17:01:58.355, stuff
7/17/2017 17:01:58.376, stuff

Could someone let me know what I'm doing wrong in my prop.conf files? For some reason they're not breaking up the file

Re: props.conf Line Breaking

MuS — Tue, 18 Jul 2017 01:02:04 GMT

Hi Svill321,

here is a quick check list:

What are you applying this on? source, sourcetype or host? Check for typos
Where is this props.conf applied? Must be on the parsing layer, either heavyweight forwarder or indexer
Check with $SPLUNK_HOME/bin/splunk btool props list if your config on the parsing instance is applied correctly
your first regex does not match, try this \d{2}\:\d{2}\:\d{2}\||\d\/\d{2}\/\d{4}\s+\d{2}:\d{2}:\d{2}:|\d\/\d{2}\/\d{4}\s+\d{2}:\d{2}:\d{2}\.\d{3},
BREAK_ONLY_BEFORE only works if Splunk encounters a new line that matches the regular expression.
same with the second regex, try this instead `([\r\n]+)\d{2}:\d{2}:\d{2}||\d\/\d{2}\/\d{4}\s+\d{2}:\d{2}:\d{2}:|\d\/\d{2}\/\d{4}\s+\d{2}:\d{2}:\d{2}.\d{3},
did you restart the parsing Splunk instance after the changes?
This will only be applied to new arriving events, it is not applied to any existing already indexed events

Hope this helps to trace down the error ...

cheers, MuS

Re: props.conf Line Breaking

fredzhang — Tue, 29 Sep 2020 14:57:42 GMT

Suggest use different stanzas with source::.... to do line break for each different type logs.
E.g. for each source:

[source::...source1...]
BREAK_ONLY_BEFORE_DATE = true
DATETIME_CONFIG = none
TIME_FORMAT = XXXX // set as log shows
...