Log example: 11-23-2015 13:03:15.529 +0100 WARN TimeoutHeap - Either time adjusted forwards by, or event loop was descheduled for 3694994ms. 11-23-2015 13:03:15.607 +0100 WARN TimeoutHeap - Either time adjusted forwards by, or event loop was descheduled for 3694963ms. 11-23-2015 13:03:15.623 +0100 WARN TimeoutHeap - Either time adjusted forwards by, or event loop was descheduled for 3695009ms. 11-23-2015 13:03:15.733 +0100 WARN TimeoutHeap - Either time adjusted forwards by, or event loop was descheduled for 3694244ms. 11-23-2015 13:03:16.248 +0100 WARN TimeoutHeap - Either time adjusted forwards by, or event loop was descheduled for 3711931ms. 11-23-2015 13:03:18.154 +0100 WARN TimeoutHeap - Either time adjusted forwards by, or event loop was descheduled for 3703228ms. 11-23-2015 13:07:07.951 +0100 WARN TimeoutHeap - Either time adjusted forwards by, or event loop was descheduled for 4293229ms. 11-23-2015 13:00:19.506 +0100 WARN TimeoutHeap - Detected system time adjusted backwards by 3691155ms. 11-23-2015 13:00:19.506 +0100 WARN TimeoutHeap - Detected system time adjusted backwards by 3690904ms. 11-23-2015 13:00:19.756 +0100 WARN TimeoutHeap - Detected system time adjusted backwards by 3690920ms. 11-23-2015 13:00:19.772 +0100 WARN TimeoutHeap - Detected system time adjusted backwards by 3690920ms. 11-23-2015 13:00:20.553 +0100 WARN TimeoutHeap - Detected system time adjusted backwards by 3675326ms. 11-23-2015 13:00:26.240 +0100 WARN TimeoutHeap - Detected system time adjusted backwards by 3681920ms. 11-23-2015 13:05:36.024 +0100 WARN TimeoutHeap - Detected system time adjusted backwards by 3091934ms. As you see above typically I get 6-7 of the first log entry type, there is nothing in the log before that that's close in time and could result in a message of that kind, then you get a typically 7 of the second type of entry and then there is nothing before restarting the splunk forwarder service, notice the time difference between entry 7 and 8, the log time actually adjusted backwards. Although as it seems the number of milliseconds don't necissarily add up to the shift backwards in time as I have another box with roughly the same amount of milliseconds in the message but the log entry actually jumping ~45 minutes back in time. Interesting though that you mention the flapping in cpu because I have the impression that after the first "Restart-Service splunkforwarder" that it is actually flapping all over the place but that it's calming after issuing a second restart-service.... As this showed up in 6.3 I've gone the other way compared to you and upgraded to 6.3.1 yesterday and am keeping a close eye on this today. Funny thing though is that this showed up not directly after the upgrade to 6.3 but some weeks later.... (BTW when you say you've eliminated the wildcards what exactly are you meaning by that? ... View more

So I'm seeing this as well on some of my UFs. I have 4 servers on which this has occured 1-3 times durring the last week, there is nothing special about these 4 boxes, running same version, same config, as the rest of the bunch ~105 windows boxes. They're in the same Windows domain, but so are others, they are not on the same vmware host, it doesn't occur syncrounus on all 4, what is common though is that they complain about the time in their splunkd.log, but only one system has logged any other errors connected to time issues in eventlogs. ... View more