How to determine what Props.conf settings affect line merging

[MasterOogway]

I have two versions of Splunk, v4.3.1 & v4.1.4 Indexing the same data, but only v4.3.1 indexes as a single line event, which is correct.
What is the best method to determine what Props.conf file settings are causing the data to stay in multiline events, and not single as needed.

I have used "splunk cmd btool props list --debug"

I have used "splunk test sourcetype /tmp/testfile.txt" to understand more of what is being used for sourcetype.

The data is being pushed from a client installation on a LWF, so I am depending on the Indexer to correctly recognize the timestamp.

I have the following setup in default Props already: "BREAK_ONLY_BEFORE_DATE = True" and with this very straight forward data having the timestamp as the first characters it seems like it shouldn't be confused on how to break lines....but it does. My default installations are pretty generic and don't usually have to do weird stuff to make Splunk work. This is the exception.

V4.1.4 - exhibits merged events

[1334880312] PASSIVE HOST CHECK: blah.com;0;Host state confirmed by hostchecker - TCP OK - 0.115 second response time on port 22

<--- no break in events here --- it just formats weird online.

[1334880312] HOST ALERT: blah.com;UP;HARD;1;Host state confirmed by hostchecker - TCP OK - 0.115 second response time on port 22

v4.3.1 - exhibits single line events (correct way to be indexed)
[1334954305] SERVICE NOTIFICATION: theparser;blahblah.com;sinfo;CRITICAL;notify-by-parser;Disk_Space_[D:92.0%>=90%] Disk_Space_[E:97.0%>=90%]

[1334954305] SERVICE NOTIFICATION: theparser;blahblah.com;sinfo;CRITICAL;notify-by-parser;Disk_Space_[E:82.0%>=80%]

[CarolinaHB]

Hi @MasterOogway 

You were able to solve this problem, I currently have this detail with version 7.2.4.2.

I work in a development environment but when I migrate production environment the sourcetype doesn't recognize me. I already copied and pasted the props but it doesn't work

Your solution would help me a lot.

https://community.splunk.com/t5/Splunk-Cloud/MultiLine-Event-Line-Breaker/m-p/504155#M28  

Regards

[lguinn2]

I don't know exactly what is going wrong here, here are some ideas to fix it

1. Explicitly tell Splunk to use epoch time

2. Add MAX_TIMESTAMP_LOOKAHEAD to your settings

3. Tell Splunk that this log contains only single line events

This will keep Splunk clear about where the timestamp appears in the data, and it will make it process the data more efficiently, too.

[yoursourcetypehere]
SHOULD_LINEMERGE = false
TIME_FORMAT=%s
MAX_TIMESTAMP_LOOKAHEAD = 15

would probably do it. All of these settings go in the props.conf file on your indexer. You can put this in your usual local directory - if you are not sure, I suggest $SPLUNK_HOME/etc/apps/search/local

You could probably just get by with SHOULD_LINEMERGE = false

[MasterOogway]

I looked back at my btool output and added a statement for [source:///var/log/nagios/nagios.log]
SHOULD_LINEMERGE = false
explicitly for the nagios logs. Funny thing is I am getting (1) of my (4) serves to index correctly but the other (3) are not. I compared the btool output for each nagios stanza to the server working and they are each identical. Humphhh!

Thoughts?

I will keep digging to see if I can see any precedence set from another stanza.

Thanks.