topic Re: Why does my modular input Run() command generate thousands of events (rather than dozens)? in Splunk Dev

Why does my modular input Run() command generate thousands of events (rather than dozens)?

michaelQumulo — Tue, 29 Sep 2020 09:35:19 GMT

I have a question about some odd behavior I'm seeing in my Modular Input app; specifically:

https://github.com/Qumulo/qumulo_splunk_app/blob/master/bin/qumulo.py#L296-L321

There is a bug here, but when I fix the bug, things don't work :).

The bug is: After I process the endpoint (can be one of iops, capacity or throughput) I call continue, which repeats the while True loop from the top. So time.sleep(polling_interval) never gets called, and I end up creating thousands of events and thrashing our REST API where I should be creating only dozens of events.

But when I remove the continue after endpoint handling, I get no events in Splunk at all, and I cannot understand why. What's more: if I place something simple like time.sleep(10) before I process any endpoints, I don't get any logged events either.

So there's something in my module input run() function that I'm not understanding -- how it is invoked or what is permissible inside of the run() function. I think it follows the python Twitter example structurally, but I'm not sure how run() is invoked, now often and so forth.

Any ideas?

Thanks in advance - Michael

Re: Why does my modular input Run() command generate thousands of events (rather than dozens)?

michaelQumulo — Mon, 02 May 2016 17:32:21 GMT

Clarification: I'm aware that the continue call in each conditional case prevents the sleep from happening, so my run() command just keeps executing, which creates thousands of events. The weird parts are as follows:

When I remove the continue commands I get no events at all.
In my log I am seeing two calls per interval for each endpoint type, rather than one.... and I can't see why.

Thanks again - Michael

Re: Why does my modular input Run() command generate thousands of events (rather than dozens)?

alacercogitatus — Wed, 11 May 2016 23:07:48 GMT

In Splunk 6.x you don't need to control the interval settings. This is native functionality. Remove all reference to sleeps, configs for interval including cron. In "More Settings" of any data input is the Interval setting. It executes exactly as you are trying to program but controlled by the master Splunk process. Then on the modular input, just execute the of statements and let them complete. You'll be much happier you did

Re: Why does my modular input Run() command generate thousands of events (rather than dozens)?

michaelQumulo — Mon, 23 May 2016 22:28:34 GMT

Hi - Thanks for answer (I just saw this now 5-23).

I did what you suggested but my app doesn't seem to be pulling any data as it stands right now (current version is here on GH). Perhaps I need to go and rebuild this with latest python SDK for a modular input first...

Re: Why does my modular input Run() command generate thousands of events (rather than dozens)?

gblock_splunk — Tue, 24 May 2016 20:06:28 GMT

If you can port it to our SDK that would be great and it should make your code cleaner / nicer, be easier for us to support you and remove a bunch of boilerplate code :-). I'll take a look at your code and see if I see anything funky.

Re: Why does my modular input Run() command generate thousands of events (rather than dozens)?

michaelQumulo — Wed, 25 May 2016 21:19:19 GMT

Thanks @gblock -- I will update my app to use the Python SDK and update....

Re: Why does my modular input Run() command generate thousands of events (rather than dozens)?

michaelQumulo — Mon, 06 Jun 2016 18:39:00 GMT

Hi gblock -- I've updated our app to use latest python SDK, you can find it here:

https://splunkbase.splunk.com/app/3082/

I now have only one other problem: The polling does not seem to work. That is: it works the first time and it pulls again if I change config (like changing the polling interval for each of my 3 endpoints to 60 seconds) but never again after that. Can you have a look?

thanks! Michael

Re: Why does my modular input Run() command generate thousands of events (rather than dozens)?

michaelQumulo — Mon, 06 Jun 2016 18:39:18 GMT

Hi gblock -- I've updated our app to use latest python SDK, you can find it here:

https://splunkbase.splunk.com/app/3082/

thanks! Michael