Getting Data In
Highlighted

line breaking partial success

Splunk Employee
Splunk Employee

I'm trying to bring in some custom source log files and initially no line breaking was occurring so all of the events in each log file would arrive as 1 event.
I wrote a line break based on each new record starting with a 7 digit date string.

SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)\d{7}\s+

Once that took effect, any log files that strictly contained only contained lines starting with a 7 digit string parsed correctly. However any log files that anywhere in the file has a line not starting with a 7 digit string, the entire log file is still entered as 1 event. As all lines not starting with the 7 digit string should be treated as the data for the previous digit string I'm unsure why this is failing.

0 Karma
Highlighted

Re: line breaking partial success

SplunkTrust
SplunkTrust

Could you post some sample data , including both type of events, one starts with 7 digit string and one without it,

0 Karma
Highlighted

Re: line breaking partial success

Esteemed Legend

Generally the solution is to set your timestamp and break there. Do your events start with a timestamp? Share a few sample events.

0 Karma
Highlighted

Re: line breaking partial success

Legend

If every line is a single event, you can do this very simply:

SHOULD_LINEMERGE = false

That is all that is needed. Do not set the LINE_BREAKER if every line is a single event.

If your input has timestamps that are not always at the beginning of a line, then (as @woodcock points out), you need to parse out the timestamp. There are a variety of settings for this. For multi-line events, Splunk breaks to begin each event with a line containing a timestamp. If Splunk isn't parsing your timestamps properly, it can't break multi-line events properly. And even if your input is all single-line events, you do need to get the timestamp right, too!

As @somesoni2 mentioned, samples of your data will help us help you.

Finally - if you are collecting multiple log files with a single input stanza, that's great. But if the log files have different formats, you will need to treat them differently for parsing by using separate props.conf stanzas.

0 Karma
Highlighted

Re: line breaking partial success

Splunk Employee
Splunk Employee

Thanks all, I ended up resolving the issue by writing the following in props.conf

[sourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = ^\d{7}

View solution in original post

0 Karma