Some Single Line Messages Are Merged into a Single Event

[fandingo]

I'm working with data that looks like this:

QA4 :: 1354371771 :: 020_grid_progress :: M020_grid_progress :: alert :: Grid recovery completed on Sat Dec 1 09:22:49 2012: There were 17 active application(s) when the grid controller went down. 3 application(s) have been recovered. The state of 11 applications has been reacquired.3 application(s) failed to be recovered. See the controller system log for details. QA4 :: 350399612 :: 050_filer_status :: M050_filer_status :: info :: Internal condition 'filer status' occurred. This condition should not affect the operation of your grid. Please notify support that this error has occurred and reference SCR2301.

Each event ends with a UNIX newline (\n), and I've verified that the newline is always properly set.

The weird part is that Splunk sometimes merges events. Here is how Splunk has interpreted the data. I used the JSON export from Splunk because it shows the newline character.

{"preview":false,"result":{"raw":"QA4 :: 1354382431 :: 070_vol_maint :: M070_vol_maint :: info :: Volume check completed on Sat Dec 1 12:20:30 2012. Volume maintenance is required. Found 8 unused volumes.\nQA4 :: 1354370459 :: 500_3tctlmon_report :: M500_3tctlmon_report :: alert :: Controller restarted on Sat Dec 1 09:00:10 2012 because of an unexpected shutdown. Please note that this failure has no effect on the applications that may be running on the grid. Please contact technical support. ","_time":"2012-12-01T12:20:30.000-0600","date_hour":"12","date_mday":"1","date_minute":"20","date_month":"december","date_second":"30","date_wday":"saturday","date_year":"2012","date_zone":"local","host":"dc2opsdashqa01","index":"main","linecount":"2","punct":"::::::::::______::..._:::","source":"/var/log/applogic-dashboard-messages.log","sourcetype":"applogic-dashboard-msg","splunk_server":"dc2mgmtsplqa01","timeendpos":"114","timestartpos":"94"}}

{"preview":false,"result":{"raw":"QA2 :: 1354382375 :: 070_vol_maint :: M070_vol_maint :: info :: Volume check completed on Sat Dec 1 12:19:34 2012. Volume maintenance is required. Found 74 unused volumes.","_time":"2012-12-01T12:19:34.000-0600","date_hour":"12","date_mday":"1","date_minute":"19","date_month":"december","date_second":"34","date_wday":"saturday","date_year":"2012","date_zone":"local","host":"dc2opsdashqa01","index":"main","linecount":"1","punct":"::::::::::______::..__.","source":"/var/log/applogic-dashboard-messages.log","sourcetype":"applogic-dashboard-msg","splunk_server":"dc2mgmtsplqa01","timeendpos":"114","timestartpos":"94"}}

Notice how the first event actually includes two. (Look for "\nQA4" in it.)

Why has Splunk combined the first two messages, but properly splits the third one into a separate event? Is there anything I can do to force a split on "\n"?

Thanks,

[yannK]

setup a sourcetype for your events, that disable the multiline detection.
in prop.conf

[mysourcetype]
SHOULD_LINEMERGE=false

see http://docs.splunk.com/Documentation/Splunk/5.0.1/Data/IndexMulti-lineEvents

[fandingo]

I have been clearing the data every time, and the re-indexed messages aren't affected. I've also run the data through "| sort -R" on the shell before Splunk picks it up. Each time, it's completely different messages that are merged, so there's nothing weird happening with the line endings.

[lguinn2]

Once Splunk has indexed data, it will not change it. So you will need to clean the events from the index and re-index the source data in order to make the changes.

./splunk clean eventdata -index yourindex

will do the trick - although Splunk will re-index everything in that index and this might be an issue for your license.

[fandingo]

We only have a single indexer, and these logs are only present on one server.

I worked some with engineers in efnet and this updated props does not work either. (I modified the log format to have the epoch timestamp first.)

etc/users/admin/search/local/props.conf

[applogic-dashboard-msg]
SHOULD_LINEMERGE=false
TIME_FORMAT=%s
EXTRACT-timestamp-grid-id-name-severity-text = ^[0-9]+ :: (?P[^ ]+) :: (?P[^ ]+) :: (?P[^ ]+) :: (?P[^ ]+) :: (?P[^\n]+)

[yannK]

Do you have multiple forwarders and indexers ?
The props.conf has to be on the indexer (for index time parameters)

[fandingo]

Thanks for the reply, but that did not fix the problem. My props.conf is now:

[applogic-msg]
SHOULD_LINEMERGE=false
EXTRACT-grid-timestamp-id-name-severity = ^(?P[^ ]+) :: (?P[0-9]+) :: (?P[^ ]+) :: (?P[^ ]+) :: (?P[^ ]+) :: (?P[^\n]+)

I appended the messages from earlier to this file, but some of them (including the example in my question) are still merged.