All Apps and Add-ons

New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py at very high frequency until crash

nicolasjeanselm
Explorer

This issue occurs on two distincts linux splunk deployment using Splunk 6.6.4 and 7.0.1 and not on my 7.0.1 on mac os x

At some point the scheduler loops until splunk crashes:
1/11/18
8:54:45.680 PM

01-11-2018 20:54:45.680 +0000 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py
host = splunky1 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/11/18
8:54:45.512 PM

01-11-2018 20:54:45.512 +0000 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py
host = splunky1 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/11/18
8:54:45.461 PM

01-11-2018 20:54:45.461 +0000 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py
host = splunky1 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/11/18
8:54:45.309 PM

01-11-2018 20:54:45.309 +0000 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py
host = splunky1 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/11/18
8:54:44.626 PM

01-11-2018 20:54:44.626 +0000 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py
host = splunky1 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd

Note 1:
The first anomaly before this behavior is that REST queries are sent with timestamps for which a REST call had already been issued and answered correctly instead of being increased by 30 minutes as configured in the handler
alt text

Note 2: a custom handler is used:
https://splunkbase.splunk.com/app/3850/

cyrillefranchet
Explorer

After doing some testing, this is due to the cookies that are dumped into the inputs.conf file. As soon as we have more than 2 inputs the scheduler crashes.

Commenting the cookies dump is the way to go to solve this issue.

I don't know why this is happening only on Windows and Linux. maybe something related to the modification time of the inputs.conf file.

0 Karma

cyrillefranchet
Explorer

Hi Nicolas,

Did you finally find an explanation regarding this issue? My splunkd consumes 110% of the CPU when I'm using this app with a custom responses handler.

Just checked your code and thought that you develop your own REST client for your purpose. Could you confirm?

Thanks.

0 Karma

cyrillefranchet
Explorer

That's funny my OSX doesn't complain as well but my production server on Windows is dying 🙂

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...