All Apps and Add-ons

Multiple instances of rest.py start while there is only one stanza active (Twitter API)

vlaamseoverheid
Explorer

My REST API works fine, apart from the fact that new processes of rest.py get launched whithout any clear reason why this happens. This is causing Splunk to hammer twitter.com API and in no time I hit the rate limit (max 180 hits per 15 minutes) and I get blocked for a while.

The processes:



splunk 12085 1 0 13:46 00:00:00 python /opt/splunk/etc/apps/rest_ta/bin/rest.py

splunk 12095 1 0 13:46 00:00:00 python /opt/splunk/etc/apps/rest_ta/bin/rest.py

splunk 12100 1 0 13:46 00:00:00 python /opt/splunk/etc/apps/rest_ta/bin/rest.py

splunk 12107 1 1 13:46 00:00:00 python /opt/splunk/etc/apps/rest_ta/bin/rest.py

splunk 12113 1 1 13:46 00:00:00 python /opt/splunk/etc/apps/rest_ta/bin/rest.py

splunk 12118 1 1 13:46 00:00:00 python /opt/splunk/etc/apps/rest_ta/bin/rest.py

splunk 12126 1 2 13:46 00:00:00 python /opt/splunk/etc/apps/rest_ta/bin/rest.py

splunk 12137 1 3 13:46 00:00:00 python /opt/splunk/etc/apps/rest_ta/bin/rest.py



The debug log clearly shows the processes getting started:

05-25-2014 13:25:27.038 +0200 DEBUG ExecProcessor - cmd='python /opt/splunk/etc/apps/rest_ta/bin/rest.py' Added to run queue; unique_id=26

05-25-2014 13:25:27.038 +0200 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py

05-25-2014 13:25:27.038 +0200 INFO ExecProcessor - interval: run once

05-25-2014 13:25:29.525 +0200 DEBUG ExecProcessor - cmd='python /opt/splunk/etc/apps/rest_ta/bin/rest.py' Added to run queue; unique_
id=27

05-25-2014 13:25:29.525 +0200 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py

05-25-2014 13:25:29.525 +0200 INFO ExecProcessor - interval: run once

...as you can see there are only a few seconds between the two ExecProcessor instances and goes on and on until I disable the REST data input.

I would like to see only one active process that only talks to Twitter every polling_interval.

Any help is welcome.

Ah yes... Splunk version=6.1.1, REST API version: 1.3.4.

Best Regards,
Peter

Tags (1)

vlaamseoverheid
Explorer

Damien, I have installed another clean instance of 6.1.1 on yet another fresh Debian. Same issue.

0 Karma

45bpever
New Member

I see this on my Debian 16.04 and 18.04 installs with both Splunk 6.6.5 and Splunk 7.2.4.

Fresh installs, no other data inputs and I get piled up rest.py processes.

From what I can tell is that it appears that Splunk is scheduling rest.py to run via its scheduler, but the rest.py script assumes that it is a run-once instance, since it loops after starting.

This seems to be a workaround but it is modifying the logic of the baboonbones code.

https://answers.splunk.com/answers/389392/issue-with-restpy-not-closing-and-running-out-of-m.html

0 Karma

vlaamseoverheid
Explorer

Hi Damien, I have copied everything (api, REST stanza's)onto another Splunk server (running 6.0.3) and there it seems to run stable. Not sure what is wrong with that 6.1.1 instance but I guess we can leave it for now, though I don't dare to upgrade to 6.1.1 on that other server. Thanks for your time so far! What I will do at a later stage, is install a new server from scratch (format/OS Debian "wheezy"/Splunk 6.1.1/API 1.3.4/Stanzas) and comment on the thread again. Note that I have not changed the code and even reinstalled the REST API multiple times. To be continued..

0 Karma

Damien_Dallimor
Ultra Champion

I can't replicate this , and I have a very diverse set of inputs.conf rest stanzas setup , and I have never heard of this from any other users before. So this is firmly in the "scratching my head" category. Have you changed any of the python code perchance ?

0 Karma

45bpever
New Member

Damien,

Is this perhaps a Debian issue?

I am having this problem as well. I have been researching forums and boards for the answer and I haven't found anyone who has resolved it yet.

One common thread is that it seems that this seen on debian installs.

0 Karma

vlaamseoverheid
Explorer

Damien, still happens after reinstalling the app and configuration completely (rm -Rf style). Any ideas?

0 Karma

vlaamseoverheid
Explorer

Damien, I have done the exercise again and I must say that it does happen even with only 1 REST stanza configured and enabled.

By the way, the host is a fresh Debian with all updates installed.

0 Karma

vlaamseoverheid
Explorer

Thanks for your time Damien. This problem occurs even with only 1 single REST stanza configured and enabled. So yes I still see them, but for the record, I will remove all and start from scratch. Will get back to you soon.

0 Karma

jkat54
SplunkTrust
SplunkTrust

See if this shows the multiple rest.py stanzas:

/$SPLUNK_HOME$/bin/splunk btool inputs list|grep rest.py
0 Karma

vlaamseoverheid
Explorer

jkat54, this command does not show any output, as the script (rest.py) is not shown in the inputs.conf file (it is called somewhere else by the Splunk REST API).

0 Karma

vlaamseoverheid
Explorer

Thanks jkat54, will give that a try, will be back soon.

0 Karma

Damien_Dallimor
Ultra Champion

Stop Splunk. Ensure there are no rest.py processes running. Ensure that you have only 1 single REST stanza configured and enabled. Start Splunk. Do you still see multiple rest.py processes running ?

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!