This next event and all subsequent events get logged a year in the past (ie: 2010/07/14 04:43:53... NOTE: 2010 INSTEAD OF 2011) until splunkd is restarted. This only started happening between the 13th and 14th of July and timestamps worked fine previous to that.
It was my understanding that UDP syslog inputs should default to DATETIME_CONFIG=CURRENT and the date should not be parsed from the event. I can't understand what is causing splunk to set the current index timestamp to a year in the past. What can be added to props.conf (or other) to work around this issue.
I am also interested in what needs to be added to transforms.conf to group the initial event into a single event instead of indexing as 3 separate events.
Please be gentle, I am fairly new to splunk and this will be my first foray into tweaking the props and transforms etc....
Turns out UDP syslog does not default to DATETIME_CONFIG=CURRENT and the multiline syslog input that contained a year in one of the lines that did not contain a timestamp was causing issues. Splunk detected this year and reset the index year to 2010 (from 2011) so all subsequent inputs were indexed a year in the past. Restarting splunkd reset the year back to CURRENT, until the multiline input was encountered again....
The solution was an \etc\system\local\props.conf entry for the host exhibiting the multiline syslog input:
This told splunk not to process the multiline input as multiple events but instead to merge them together until the next properly formatted date was detected.
What we don't understand is why is started happening out of the blue... or why the issue did not resolve on subsequent multiline syslog inputs that contained 2011 in their non-timestamped lines. Sounds like we may never know and have decided to just fix the issue and move on.