A few events from a single log source file are getting stuck together at indexing - why?

[pj]

I am indexing a log file of about 50,000 single line events and for the most part the events are indexed fine. This runs every 24hrs and takes in the events.

However, each day, there are about 4 indexed events in Splunk that actually contain many events within the single indexed event (e.g. an over 250 line event). This is pretty annoying. The source file looks fine and the events are all be on their own lines, so not sure why splunk is taking a few of them and indexing them as one event.

Any ideas?

[Simeon]

Splunk has the concept of a line and event breaker. The line breaker will separate the lines within a source, whereas the event breaker will dictate when an event is multi-line or not. For your scenario, it sounds as thought you need to tune the event breaker to properly separate the events. In some situations, Splunk will try to combine events if it does not see a timestamp on each line. Again, this depends on how you have set the event breaker. For your scenario, you have said that Splunk shows these events have hundreds of lines implying that the line breaker is working correctly. To turn off line merging, you can set the SHOULD_LINEMERGE to False under your specific sourcetype within your $SPLUNK_HOME/etc/apps/search/local/props.conf file:

[your_sourcetype]
SHOULD_LINEMERGE = False

For more detail on settings, specifically for what dictates multi-line events:

http://www.splunk.com/base/Documentation/latest/Admin/Indexmulti-lineevents

[Lowell]

Can you provide some sample events. Are the events that get stuck together in any way different from the ones that do not? Can you post your props settings related to this source/souretype. (Please add this info to your existing question. Use the the "edit" link.)