Monotonic Time Stuck and Search_Telemetry

[gearmstrong]

Good day,

We have been preriodically receiving the following message in our splunkd.log and I am having issues finding a way of making it subside. The error is as follows:
"01-09-2020 09:06:23.974 -0500 ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck?
event_message = Monotonic time source didn't increase; is it stuck?"
I turned on DEBUG Logging for PipelineComponent and looked at events both prior and immediately after and found references to Telemetry...?

"01-09-2020 09:06:14.293 -0500 DEBUG PipelineComponent - Choosing pipeline set with index=0 and number=0 with policy=round_robin and request_info:
request_type=tailing with input_path=E:\Program Files\Splunk\var\run\splunk\search_telemetry. event_message = Choosing pipeline set with index=0 and number=0 with policy=round_robin and request_info: request_type=tailing with input_path=E:\Program Files\Splunk\var\run\splunk\search_telemetry."

I have verified that we are not using a Directory Monitor or any other type of monitor that would 'look at' our search_telemetry files.
I am surmising that since files in this directory are transient, by the time that whatever looks at the files and starts to parse them, they are whisked away and we see these errors.

...All strings I have been pulling have broken... any thoughts?

Best regards,

Greg

[jfaldmomacu]

I just saw the message "ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck?" come in every 2-3 seconds or about 24 per minute. It did indicate a problem with my system which I am troubleshooting. 

[gearmstrong]

All,

Since no one has provided an answer or any feedback on this incident here, I thought I'd share the information that we received from one of our partners. The gist of the information is that this a generic and benign error.

"...*Splunk PS Slack channel. I was able to find this answer, which is apparently what Splunk support had previously sent to a customer.

"This is an error we have come across with some of our Windows customers, and seems more common of virtualized instances. The splunk process will periodically check the time of the OS system and will show this error if there is a difference (~15 ms) as an indication of the time progress internally. This is really an internal ERROR that should not be reported."

Can you confirm that that OS on that Splunk server does have the correct time? Another thought would be that since you are currently running a X.0.0 version of Splunk, to upgrade Splunk to the latest version and see if that will clear up the issue for you. *...

We have found no delta in time although we are operating in a virtuaized environment.

I hope this information helps you!

Best regards,

Greg