I have two versions of Splunk, v4.3.1 & v4.1.4 Indexing the same data, but only v4.3.1 indexes as a single line event, which is correct.
What is the best method to determine what Props.conf file settings are causing the data to stay in multiline events, and not single as needed.
I have used "splunk cmd btool props list --debug"
I have used "splunk test sourcetype /tmp/testfile.txt" to understand more of what is being used for sourcetype.
The data is being pushed from a client installation on a LWF, so I am depending on the Indexer to correctly recognize the timestamp.
I have the following setup in default Props already: "BREAK_ONLY_BEFORE_DATE = True" and with this very straight forward data having the timestamp as the first characters it seems like it shouldn't be confused on how to break lines....but it does. My default installations are pretty generic and don't usually have to do weird stuff to make Splunk work. This is the exception.
V4.1.4 - exhibits merged events
 PASSIVE HOST CHECK: blah.com;0;Host state confirmed by hostchecker - TCP OK - 0.115 second response time on port 22
<--- no break in events here --- it just formats weird online.
 HOST ALERT: blah.com;UP;HARD;1;Host state confirmed by hostchecker - TCP OK - 0.115 second response time on port 22
v4.3.1 - exhibits single line events (correct way to be indexed)
 SERVICE NOTIFICATION: theparser;blahblah.com;sinfo;CRITICAL;notify-by-parser;Disk_Space_[D:92.0%>=90%] Disk_Space_[E:97.0%>=90%]
 SERVICE NOTIFICATION: theparser;blahblah.com;sinfo;CRITICAL;notify-by-parser;Disk_Space_[E:82.0%>=80%]
would probably do it. All of these settings go in the props.conf file on your indexer. You can put this in your usual local directory - if you are not sure, I suggest $SPLUNK_HOME/etc/apps/search/local
You could probably just get by with SHOULD_LINEMERGE = false
I looked back at my btool output and added a statement for [source:///var/log/nagios/nagios.log]
SHOULD_LINEMERGE = false
explicitly for the nagios logs. Funny thing is I am getting (1) of my (4) serves to index correctly but the other (3) are not. I compared the btool output for each nagios stanza to the server working and they are each identical. Humphhh!
I will keep digging to see if I can see any precedence set from another stanza.