New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py at very high frequency until crash

[nicolasjeanselm]

This issue occurs on two distincts linux splunk deployment using Splunk 6.6.4 and 7.0.1 and not on my 7.0.1 on mac os x

At some point the scheduler loops until splunk crashes:
1/11/18
8:54:45.680 PM

01-11-2018 20:54:45.680 +0000 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py
host = splunky1 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/11/18
8:54:45.512 PM

01-11-2018 20:54:45.512 +0000 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py
host = splunky1 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/11/18
8:54:45.461 PM

01-11-2018 20:54:45.461 +0000 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py
host = splunky1 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/11/18
8:54:45.309 PM

01-11-2018 20:54:45.309 +0000 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py
host = splunky1 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/11/18
8:54:44.626 PM

01-11-2018 20:54:44.626 +0000 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py
host = splunky1 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd

Note 1:
The first anomaly before this behavior is that REST queries are sent with timestamps for which a REST call had already been issued and answered correctly instead of being increased by 30 minutes as configured in the handler

Note 2: a custom handler is used:
https://splunkbase.splunk.com/app/3850/

[cyrillefranchet]

After doing some testing, this is due to the cookies that are dumped into the inputs.conf file. As soon as we have more than 2 inputs the scheduler crashes.

Commenting the cookies dump is the way to go to solve this issue.

I don't know why this is happening only on Windows and Linux. maybe something related to the modification time of the inputs.conf file.

[cyrillefranchet]

Hi Nicolas,

Did you finally find an explanation regarding this issue? My splunkd consumes 110% of the CPU when I'm using this app with a custom responses handler.

Just checked your code and thought that you develop your own REST client for your purpose. Could you confirm?

Thanks.

[cyrillefranchet]

That's funny my OSX doesn't complain as well but my production server on Windows is dying 🙂