For a subset of hosts _time is the same for every event sent

[dozer]

I have a particular feed with 24 appliances that send their data via rest call over 8089 to a heavy forwarder which is then forwarded to the indexing cluster and indexed. 

For every event for every appliance, _time is correct with the exception of three appliances.

For those three appliances however, regardless of when the events are generated, _time is always 3:55:40.000 AM for appliance one, 3:25:00.000 AM for appliance two, and 3:58:00.000 AM for appliance three. And again, the other 21 appliances that send the exact same way are not having this issue.

My original thought was that it was a config issue with those three appliances. But the team that manages them confirmed they were all configured the same.

I have not been able to find any clues on the splunk side as to why this may be happening. Any help would be appreciated.

[richgalloway]

I agree it sounds like a configuration problem, but rather than look at the appliance configs, I'd look at the Splunk configs.

How are the three different from the other 21 in their Splunk configurations?  Do they use different sourcetypes?  If so, compare the props.conf settings.  I suspect some large number in the data is being mis-interpreted as a timestamp.

[dozer · Accepted solution]

Adding DATETIME_CONFIG = CURRENT to props ended up correcting it for those three. Still was not able to determine what config was actually responsible for causing the issue.