topic Modular Input Streaming with Python in Getting Data In

Modular Input Streaming with Python

phoenixdigital — Mon, 28 Sep 2020 17:50:59 GMT

So I have been reading the documentation on how to create modular inputs using the Python SDK here
http://dev.splunk.com/view/python-sdk/SP-CAAAER3

And I tested the random_number example from the SDK and it seems the random number is only generated when Splunk starts.

So as a learning exercise I tried to add a new parameter called 'interval' which will cause the number to generate every x seconds.

After a few failed attempts with indent errors in the script I finally got this to run without error
/opt/splunk/bin/splunk cmd /opt/splunk/bin/python /opt/splunk/etc/apps/random_numbers/bin/random_numbers.py

I restarted Splunk no data is coming in?

My new module input script is here
http://pastebin.com/TwVbHiUp

and my input.spec.conf is

[random_numbers://<name>]
*Generates events containing a random floating point number.

min = <value>
max = <value>
interval = <value>

Am I going about this the wrong way?

My final goal is to write something to listen to systemd journal. But I need to get past this hurdle of having something permanently running in the background either generating numbers in this case or listening to systemd down the track.
http://www.freedesktop.org/software/systemd/python-systemd/journal.html

Re: Modular Input Streaming with Python

phoenixdigital — Fri, 10 Oct 2014 04:43:31 GMT

Never mind resolved it with this change
http://pastebin.com/JfTNxQEv

My intervals were not defined when I was referencing them.

Going to have to change this a fair bit to spin off separate thread for each input in the loop

            for input_name, input_item in inputs.inputs.iteritems():

Would the best way to achieve this to have stream_events definition loop through all inputs and fire off a thread for each which has a while 1: in it?

What closes these threads when new inputs are added via the GUI?

Because if stream_events is called again there will be a second set of threads started. Do I have to put the smarts in to ensure two threads for the same input dont exist?

Re: Modular Input Streaming with Python

phoenixdigital — Fri, 10 Oct 2014 05:03:56 GMT

Adding threading doesnt seem to work.
http://pastebin.com/r5QyYb0T

Is there a better way I should be going about this?

Re: Modular Input Streaming with Python

phoenixdigital — Thu, 16 Oct 2014 01:07:26 GMT

Ok it appears that threading was the wrong track to go down.

The SDK supports single or multiple instances meaning that

single instance - one script handles all of the data inputs
multi instance - each input fires off its own version of the script

http://docs.splunk.com/Documentation/Splunk/6.1.4/AdvancedDev/ModInputsScripts#Single_or_multiple_instances_of_a_script

I tested it out again with the random numbers example

single instance - http://pastebin.com/77MMLqPz
multi instance example - http://pastebin.com/idDqG7XY

Now to build the systemd modular input based on this new found knowledge.