topic Re: Multiple instances of rest.py start while there is only one stanza active (Twitter API) in All Apps and Add-ons

Multiple instances of rest.py start while there is only one stanza active (Twitter API)

vlaamseoverheid — Mon, 28 Sep 2020 16:43:12 GMT

My REST API works fine, apart from the fact that new processes of rest.py get launched whithout any clear reason why this happens. This is causing Splunk to hammer twitter.com API and in no time I hit the rate limit (max 180 hits per 15 minutes) and I get blocked for a while.

The processes:

splunk 12085 1 0 13:46 00:00:00 python /opt/splunk/etc/apps/rest_ta/bin/rest.py

splunk 12095 1 0 13:46 00:00:00 python /opt/splunk/etc/apps/rest_ta/bin/rest.py

splunk 12100 1 0 13:46 00:00:00 python /opt/splunk/etc/apps/rest_ta/bin/rest.py

splunk 12107 1 1 13:46 00:00:00 python /opt/splunk/etc/apps/rest_ta/bin/rest.py

splunk 12113 1 1 13:46 00:00:00 python /opt/splunk/etc/apps/rest_ta/bin/rest.py

splunk 12118 1 1 13:46 00:00:00 python /opt/splunk/etc/apps/rest_ta/bin/rest.py

splunk 12126 1 2 13:46 00:00:00 python /opt/splunk/etc/apps/rest_ta/bin/rest.py

splunk 12137 1 3 13:46 00:00:00 python /opt/splunk/etc/apps/rest_ta/bin/rest.py

The debug log clearly shows the processes getting started:

05-25-2014 13:25:27.038 +0200 DEBUG ExecProcessor - cmd='python /opt/splunk/etc/apps/rest_ta/bin/rest.py' Added to run queue; unique_id=26

05-25-2014 13:25:27.038 +0200 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py

05-25-2014 13:25:27.038 +0200 INFO ExecProcessor - interval: run once

05-25-2014 13:25:29.525 +0200 DEBUG ExecProcessor - cmd='python /opt/splunk/etc/apps/rest_ta/bin/rest.py' Added to run queue; unique_id=27

05-25-2014 13:25:29.525 +0200 INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py

05-25-2014 13:25:29.525 +0200 INFO ExecProcessor - interval: run once

...as you can see there are only a few seconds between the two ExecProcessor instances and goes on and on until I disable the REST data input.

I would like to see only one active process that only talks to Twitter every polling_interval.

Any help is welcome.

Ah yes... Splunk version=6.1.1, REST API version: 1.3.4.

Best Regards,
Peter

Re: Multiple instances of rest.py start while there is only one stanza active (Twitter API)

Damien_Dallimor — Tue, 27 May 2014 15:13:14 GMT

Stop Splunk. Ensure there are no rest.py processes running. Ensure that you have only 1 single REST stanza configured and enabled. Start Splunk. Do you still see multiple rest.py processes running ?

Re: Multiple instances of rest.py start while there is only one stanza active (Twitter API)

jkat54 — Tue, 27 May 2014 15:41:16 GMT

See if this shows the multiple rest.py stanzas:

/$SPLUNK_HOME$/bin/splunk btool inputs list|grep rest.py

Re: Multiple instances of rest.py start while there is only one stanza active (Twitter API)

vlaamseoverheid — Wed, 28 May 2014 06:30:58 GMT

Thanks for your time Damien. This problem occurs even with only 1 single REST stanza configured and enabled. So yes I still see them, but for the record, I will remove all and start from scratch. Will get back to you soon.

Re: Multiple instances of rest.py start while there is only one stanza active (Twitter API)

vlaamseoverheid — Wed, 28 May 2014 06:31:32 GMT

Thanks jkat54, will give that a try, will be back soon.

Re: Multiple instances of rest.py start while there is only one stanza active (Twitter API)

vlaamseoverheid — Wed, 28 May 2014 15:09:21 GMT

jkat54, this command does not show any output, as the script (rest.py) is not shown in the inputs.conf file (it is called somewhere else by the Splunk REST API).

Re: Multiple instances of rest.py start while there is only one stanza active (Twitter API)

vlaamseoverheid — Wed, 28 May 2014 15:17:01 GMT

Damien, I have done the exercise again and I must say that it does happen even with only 1 REST stanza configured and enabled.

By the way, the host is a fresh Debian with all updates installed.

Re: Multiple instances of rest.py start while there is only one stanza active (Twitter API)

vlaamseoverheid — Wed, 28 May 2014 15:23:42 GMT

Damien, still happens after reinstalling the app and configuration completely (rm -Rf style). Any ideas?

Re: Multiple instances of rest.py start while there is only one stanza active (Twitter API)

Damien_Dallimor — Wed, 28 May 2014 18:22:47 GMT

I can't replicate this , and I have a very diverse set of inputs.conf rest stanzas setup , and I have never heard of this from any other users before. So this is firmly in the "scratching my head" category. Have you changed any of the python code perchance ?

Re: Multiple instances of rest.py start while there is only one stanza active (Twitter API)

vlaamseoverheid — Fri, 30 May 2014 11:44:44 GMT

Hi Damien, I have copied everything (api, REST stanza's)onto another Splunk server (running 6.0.3) and there it seems to run stable. Not sure what is wrong with that 6.1.1 instance but I guess we can leave it for now, though I don't dare to upgrade to 6.1.1 on that other server. Thanks for your time so far! What I will do at a later stage, is install a new server from scratch (format/OS Debian "wheezy"/Splunk 6.1.1/API 1.3.4/Stanzas) and comment on the thread again. Note that I have not changed the code and even reinstalled the REST API multiple times. To be continued..

Re: Multiple instances of rest.py start while there is only one stanza active (Twitter API)

vlaamseoverheid — Mon, 02 Jun 2014 00:07:53 GMT

Damien, I have installed another clean instance of 6.1.1 on yet another fresh Debian. Same issue.

Re: Multiple instances of rest.py start while there is only one stanza active (Twitter API)

45bpever — Fri, 01 Mar 2019 17:33:50 GMT

Damien,

Is this perhaps a Debian issue?

I am having this problem as well. I have been researching forums and boards for the answer and I haven't found anyone who has resolved it yet.

One common thread is that it seems that this seen on debian installs.

Re: Multiple instances of rest.py start while there is only one stanza active (Twitter API)

45bpever — Fri, 01 Mar 2019 17:38:40 GMT

I see this on my Debian 16.04 and 18.04 installs with both Splunk 6.6.5 and Splunk 7.2.4.

Fresh installs, no other data inputs and I get piled up rest.py processes.

From what I can tell is that it appears that Splunk is scheduling rest.py to run via its scheduler, but the rest.py script assumes that it is a run-once instance, since it loops after starting.

This seems to be a workaround but it is modifying the logic of the baboonbones code.

https://answers.splunk.com/answers/389392/issue-with-restpy-not-closing-and-running-out-of-m.html