topic Re: Splunk 6.6.1 stops monitoring files in Monitoring Splunk

Splunk 6.6.1 stops monitoring files

craigkleen — Thu, 22 Jun 2017 17:27:01 GMT

Recently updating from 6.5.3 to 6.6.1, I started running into a situation where at least one of my Heavy Forwarders would intermittently stop sending data. The Heavy Forwarder is on a RedHat 6.x system, virtualized, with the latest patches.

After reading some answers, I found the command:

./bin/splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus

Which for at least one of the files output the following:

<s:key name="/var/log/aruba/controller1.log">
  <s:dict>
    <s:key name="file position">22440152</s:key>
    <s:key name="file size">20726882</s:key>
    <s:key name="parent">/var/log/aruba/*.log</s:key>
    <s:key name="percent">108.27</s:key>
    <s:key name="type">open file</s:key>
  </s:dict>
</s:key>

I figure it being over 100% read is odd. When it got into this state, I checked the metrics in _internal for this monitored file and they had stopped recording. With RedHat, I use 'logrotate', running on an hourly schedule. The change from the file being read to stopping seems to have occurred on the hour. It would stop working for 1-3 hours, then suddenly start reading again all by itself. It's worked before with 6.5.3, but not with 6.6.1. I downgraded last night to 6.5.3 on this one Heavy Forwarder, and it's back to working again, so I've eliminated logrotate and other OS patches.

Can anyone advise on some other things to try with 6.6.1 to see if there's some new setting I should be using?

Re: Splunk 6.6.1 stops monitoring files

woodcock — Thu, 22 Jun 2017 17:59:45 GMT

Definitely open a support case and add the bug tag to this question.

Re: Splunk 6.6.1 stops monitoring files

craigkleen — Thu, 22 Jun 2017 18:56:31 GMT

Will do. Thx

Re: Splunk 6.6.1 stops monitoring files

delink — Tue, 27 Jun 2017 20:48:44 GMT

Did a case get opened on this? I think I am seeing the same thing, and we can open one also.

Re: Splunk 6.6.1 stops monitoring files

craigkleen — Tue, 27 Jun 2017 23:21:01 GMT

I have, and submitted a diag. Waiting to hear back.

Re: Splunk 6.6.1 stops monitoring files

reswob4 — Thu, 14 Sep 2017 12:40:42 GMT

I think I have the same problem, or at least so similar it might as well be the same.

I have a CentOS 6.x HF that receives data via rsyslog and writes that data to different dated files based on filters. So logs from routers would be saved in a file called yyyy-mm-dd-rtrs.log. After I upgraded on Sept 9, all those feeds stopped (and more annoying, all my alerts for missing data didn't work... argh). All files labeled 2017-09-08-xxx.log are indexed. All files labeled 2017-09-09-xxx.log and later are not.

Please keep us updated on this issue.

Re: Splunk 6.6.1 stops monitoring files

delink — Thu, 14 Sep 2017 13:45:41 GMT

We do have a case open with Splunk on this, and it is a noted bug now. My customer was running the case with support, so I do not have the JIRA for it, but they did offer a hotfix, so I think this may be fixed in either 6.6.3 or whenever 6.6.4 comes out.

Re: Splunk 6.6.1 stops monitoring files

woodcock — Sun, 18 Nov 2018 04:13:44 GMT

What was the final resolution on this from support, @craigkleen?

Re: Splunk 6.6.1 stops monitoring files

craigkleen — Sat, 19 Jan 2019 00:24:11 GMT

Sorry for the delay. Splunk's response wasn't great. I basically had to write a "restart" into my log rotation script. It might be fixed with the 7.x series now, but I haven't had the time to go back and check.

Re: Splunk 6.6.1 stops monitoring files

davpx — Sat, 26 Jan 2019 03:14:03 GMT

You should upgrade. There are known bugs for tailing files in versions this old.