I'm currently troubleshooting some data inputs from a Universal Forwarder that I have forwarding to an intermediate Heavy Forwarder tier which forwards to my Indexer tier. I was under the understanding that Universal Forwarders should not do any parsing, however, when I look at the Universal forwarder splunkd.log files, I'm seeing quite a lot of "Failed to parse timestamp" and "The TIMEFORMAT specified is matching timestamps outside of the acceptable time window. If this timestamp is correct, consider adjusting MAXDAYSAGO and MAXDAYS_HENCE." on the Universal Forwarder.
If the UF is supposed to be sending streams of data and skipping any parsing operations, why am I see these errors at the UF?
Sample logs I'm seeing on the Universal Forwarder:
11-22-2016 01:37:15.717 +0000 WARN DateParserVerbose - The TIME_FORMAT specified is matching timestamps (ZERO_TIME) outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context: removed 11-22-2016 01:37:15.717 +0000 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Tue Nov 22 01:36:58 2016). Context: removed
Have you tried grabbing a sample of the data and using that to go through the "Add Data" wizard on another Splunk machine?
That might give you a heads up on the formatting needed on the time-stamp and also allow you to play with settings until its correct.
Did you install the UF version of Splunk (there are different packages)? Have you deployed any
INDEXED_EXTRACTIONS= configurations to the UF?