This issue occurs on two distincts linux splunk deployment using Splunk 6.6.4 and 7.0.1 and not on my 7.0.1 on mac os x
At some point the scheduler loops until splunk crashes:
1/11/18
8:54:45.680 PM
01-11-2018 20:54:45.680 +0000 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py
host = splunky1 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/11/18
8:54:45.512 PM
01-11-2018 20:54:45.512 +0000 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py
host = splunky1 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/11/18
8:54:45.461 PM
01-11-2018 20:54:45.461 +0000 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py
host = splunky1 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/11/18
8:54:45.309 PM
01-11-2018 20:54:45.309 +0000 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py
host = splunky1 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/11/18
8:54:44.626 PM
01-11-2018 20:54:44.626 +0000 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py
host = splunky1 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
Note 1:
The first anomaly before this behavior is that REST queries are sent with timestamps for which a REST call had already been issued and answered correctly instead of being increased by 30 minutes as configured in the handler
Note 2: a custom handler is used:
https://splunkbase.splunk.com/app/3850/
After doing some testing, this is due to the cookies that are dumped into the inputs.conf file. As soon as we have more than 2 inputs the scheduler crashes.
Commenting the cookies dump is the way to go to solve this issue.
I don't know why this is happening only on Windows and Linux. maybe something related to the modification time of the inputs.conf file.
Hi Nicolas,
Did you finally find an explanation regarding this issue? My splunkd consumes 110% of the CPU when I'm using this app with a custom responses handler.
Just checked your code and thought that you develop your own REST client for your purpose. Could you confirm?
Thanks.
That's funny my OSX doesn't complain as well but my production server on Windows is dying 🙂