Here are some pieces of info that may be relevant:
Any ideas would be greatly appreciated. I've restarted the splunk server (full windows reboot) as well as restarting the splunk forwarder on the nodes. It seems like it's date related but none of the other sourcetypes seem to be affected.
My wild stab in the dark, based on the horrors of experience, is that the props.conf does not have timestamp format explicitly set, and maybe what we want to be Nov 1, 2016, is showing up in January 11th, 2016?
Can you share your props config for this sourcetype?
You're exactly right! The time format in the logs is dd-mm-yyyy and it would appear that splunk is expecting mm-dd-yyyy. I am a bit puzzled as to why it chose November to do this (as opposed to July or October or whatever). Maybe it's a different default in splunk 6.5.0 versus the 6.4.x and previous. When I look back in January I see interlaced data for 11-01-2016 and 01-11-2016.
[application_log] NO_BINARY_CHECK = 1 maxDist = 75 pulldown_type = 1 REPORT-myname=applicationlogmap [applicationlogmap] DELIMS="|" FIELDS="TimeStamp","ThreadID","LogLevel","ClassName","LogHash","Msg"
I'll read up on how to specify the time/date format and get this resolved. Thanks so much for taking a wild stab! I'll mark your answer as accepted as soon as I verify this works.
If you share a sample event I can assist with the config.
That would be awesome. Here's a sample line:
04-11-2016 09:15:29,482|http-apr-8080-exec-3| INFO|ConfigFilters||sessionManagement/template?name=portal/Login built in 0.0151612 seconds ().
Add Data Wizard FTW!
[ tbourne ] SHOULD_LINEMERGE=false NO_BINARY_CHECK=true TIME_FORMAT=%d-%m-%Y %H:%M:%S,%f TIME_PREFIX=^ TZ=UTC MAX_TIMESTAMP_LOOKAHEAD=30
This should clean up your timestamping and get you some perf gains. Best practice is to set linebreaker and time formatting params explicitly on all sourcetypes, as making splunk auto discover makes it work harder and ensures you avoid this pain in the future.