line breaking partial success

[jsmith_splunk]

I'm trying to bring in some custom source log files and initially no line breaking was occurring so all of the events in each log file would arrive as 1 event.
I wrote a line break based on each new record starting with a 7 digit date string.

SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)\d{7}\s+

Once that took effect, any log files that strictly contained only contained lines starting with a 7 digit string parsed correctly. However any log files that anywhere in the file has a line not starting with a 7 digit string, the entire log file is still entered as 1 event. As all lines not starting with the 7 digit string should be treated as the data for the previous digit string I'm unsure why this is failing.

[jsmith_splunk · Accepted solution]

Thanks all, I ended up resolving the issue by writing the following in props.conf

[sourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = ^\d{7}

[lguinn2]

If every line is a single event, you can do this very simply:

SHOULD_LINEMERGE = false

That is all that is needed. Do not set the LINE_BREAKER if every line is a single event.

If your input has timestamps that are not always at the beginning of a line, then (as @woodcock points out), you need to parse out the timestamp. There are a variety of settings for this. For multi-line events, Splunk breaks to begin each event with a line containing a timestamp. If Splunk isn't parsing your timestamps properly, it can't break multi-line events properly. And even if your input is all single-line events, you do need to get the timestamp right, too!

As @somesoni2 mentioned, samples of your data will help us help you.

Finally - if you are collecting multiple log files with a single input stanza, that's great. But if the log files have different formats, you will need to treat them differently for parsing by using separate props.conf stanzas.

[woodcock]

Generally the solution is to set your timestamp and break there. Do your events start with a timestamp? Share a few sample events.

[somesoni2]

Could you post some sample data , including both type of events, one starts with 7 digit string and one without it,