I'm working with data that looks like this:
QA4 :: 1354371771 :: 020_grid_progress :: M020_grid_progress :: alert :: Grid recovery completed on Sat Dec 1 09:22:49 2012: There were 17 active application(s) when the grid controller went down. 3 application(s) have been recovered. The state of 11 applications has been reacquired.3 application(s) failed to be recovered. See the controller system log for details. QA4 :: 350399612 :: 050_filer_status :: M050_filer_status :: info :: Internal condition 'filer status' occurred. This condition should not affect the operation of your grid. Please notify support that this error has occurred and reference SCR2301.
Each event ends with a UNIX newline (\n), and I've verified that the newline is always properly set.
The weird part is that Splunk sometimes merges events. Here is how Splunk has interpreted the data. I used the JSON export from Splunk because it shows the newline character.
{"preview":false,"result":{"raw":"QA4 :: 1354382431 :: 070_vol_maint :: M070_vol_maint :: info :: Volume check completed on Sat Dec 1 12:20:30 2012. Volume maintenance is required. Found 8 unused volumes.\nQA4 :: 1354370459 :: 500_3tctlmon_report :: M500_3tctlmon_report :: alert :: Controller restarted on Sat Dec 1 09:00:10 2012 because of an unexpected shutdown. Please note that this failure has no effect on the applications that may be running on the grid. Please contact technical support. ","_time":"2012-12-01T12:20:30.000-0600","date_hour":"12","date_mday":"1","date_minute":"20","date_month":"december","date_second":"30","date_wday":"saturday","date_year":"2012","date_zone":"local","host":"dc2opsdashqa01","index":"main","linecount":"2","punct":"::::::::::______::..._:::","source":"/var/log/applogic-dashboard-messages.log","sourcetype":"applogic-dashboard-msg","splunk_server":"dc2mgmtsplqa01","timeendpos":"114","timestartpos":"94"}}
{"preview":false,"result":{"raw":"QA2 :: 1354382375 :: 070_vol_maint :: M070_vol_maint :: info :: Volume check completed on Sat Dec 1 12:19:34 2012. Volume maintenance is required. Found 74 unused volumes.","_time":"2012-12-01T12:19:34.000-0600","date_hour":"12","date_mday":"1","date_minute":"19","date_month":"december","date_second":"34","date_wday":"saturday","date_year":"2012","date_zone":"local","host":"dc2opsdashqa01","index":"main","linecount":"1","punct":"::::::::::______::..__.","source":"/var/log/applogic-dashboard-messages.log","sourcetype":"applogic-dashboard-msg","splunk_server":"dc2mgmtsplqa01","timeendpos":"114","timestartpos":"94"}}
Notice how the first event actually includes two. (Look for "\nQA4" in it.)
Why has Splunk combined the first two messages, but properly splits the third one into a separate event? Is there anything I can do to force a split on "\n"?
Thanks,
... View more