Got a response that handling multiple timestamp types can be done with a combination of the condition in the LINE_BREAKER and a TIME_FORMAT=\etc\system\local\custom_datetime.xml definition. ... View more

Turns out UDP syslog does not default to DATETIME_CONFIG=CURRENT and the multiline syslog input that contained a year in one of the lines that did not contain a timestamp was causing issues. Splunk detected this year and reset the index year to 2010 (from 2011) so all subsequent inputs were indexed a year in the past. Restarting splunkd reset the year back to CURRENT, until the multiline input was encountered again.... The solution was an \etc\system\local\props.conf entry for the host exhibiting the multiline syslog input: [host::4.3.2.1] LINE_BREAKER = ([\r\n]+(?=\w{3}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2})) This told splunk not to process the multiline input as multiple events but instead to merge them together until the next properly formatted date was detected. What we don't understand is why is started happening out of the blue... or why the issue did not resolve on subsequent multiline syslog inputs that contained 2011 in their non-timestamped lines. Sounds like we may never know and have decided to just fix the issue and move on. ... View more

The issue was not that Splunk stopped indexing. It turned out to be that the timestamp of the current time changed to a year in the past. I will close this post and open another to try to determine why the current timestamp changed and how to workaround with props.conf. ... View more