topic Re: line breaking partial success in Getting Data In

line breaking partial success

jsmith_splunk — Wed, 05 Aug 2015 14:12:24 GMT

I'm trying to bring in some custom source log files and initially no line breaking was occurring so all of the events in each log file would arrive as 1 event.
I wrote a line break based on each new record starting with a 7 digit date string.

SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)\d{7}\s+

Once that took effect, any log files that strictly contained only contained lines starting with a 7 digit string parsed correctly. However any log files that anywhere in the file has a line not starting with a 7 digit string, the entire log file is still entered as 1 event. As all lines not starting with the 7 digit string should be treated as the data for the previous digit string I'm unsure why this is failing.

Re: line breaking partial success

somesoni2 — Wed, 05 Aug 2015 15:40:33 GMT

Could you post some sample data , including both type of events, one starts with 7 digit string and one without it,

Re: line breaking partial success

woodcock — Wed, 05 Aug 2015 16:22:30 GMT

Generally the solution is to set your timestamp and break there. Do your events start with a timestamp? Share a few sample events.

Re: line breaking partial success

lguinn2 — Wed, 05 Aug 2015 16:51:27 GMT

If every line is a single event, you can do this very simply:

SHOULD_LINEMERGE = false

That is all that is needed. Do not set the LINE_BREAKER if every line is a single event.

If your input has timestamps that are not always at the beginning of a line, then (as @woodcock points out), you need to parse out the timestamp. There are a variety of settings for this. For multi-line events, Splunk breaks to begin each event with a line containing a timestamp. If Splunk isn't parsing your timestamps properly, it can't break multi-line events properly. And even if your input is all single-line events, you do need to get the timestamp right, too!

As @somesoni2 mentioned, samples of your data will help us help you.

Finally - if you are collecting multiple log files with a single input stanza, that's great. But if the log files have different formats, you will need to treat them differently for parsing by using separate props.conf stanzas.

Re: line breaking partial success

jsmith_splunk — Wed, 05 Aug 2015 17:40:14 GMT

Thanks all, I ended up resolving the issue by writing the following in props.conf

[sourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = ^\d{7}