Getting Data In

Modular Input Streaming with Python

phoenixdigital
Builder

So I have been reading the documentation on how to create modular inputs using the Python SDK here
http://dev.splunk.com/view/python-sdk/SP-CAAAER3

And I tested the random_number example from the SDK and it seems the random number is only generated when Splunk starts.

So as a learning exercise I tried to add a new parameter called 'interval' which will cause the number to generate every x seconds.

After a few failed attempts with indent errors in the script I finally got this to run without error
/opt/splunk/bin/splunk cmd /opt/splunk/bin/python /opt/splunk/etc/apps/random_numbers/bin/random_numbers.py

I restarted Splunk no data is coming in?

My new module input script is here
http://pastebin.com/TwVbHiUp

and my input.spec.conf is

[random_numbers://<name>]
*Generates events containing a random floating point number.

min = <value>
max = <value>
interval = <value>

Am I going about this the wrong way?

My final goal is to write something to listen to systemd journal. But I need to get past this hurdle of having something permanently running in the background either generating numbers in this case or listening to systemd down the track.
http://www.freedesktop.org/software/systemd/python-systemd/journal.html

Tags (1)
0 Karma
1 Solution

phoenixdigital
Builder

Ok it appears that threading was the wrong track to go down.

The SDK supports single or multiple instances meaning that

single instance - one script handles all of the data inputs
multi instance - each input fires off its own version of the script

http://docs.splunk.com/Documentation/Splunk/6.1.4/AdvancedDev/ModInputsScripts#Single_or_multiple_in...

I tested it out again with the random numbers example

single instance - http://pastebin.com/77MMLqPz
multi instance example - http://pastebin.com/idDqG7XY

Now to build the systemd modular input based on this new found knowledge.

View solution in original post

phoenixdigital
Builder

Ok it appears that threading was the wrong track to go down.

The SDK supports single or multiple instances meaning that

single instance - one script handles all of the data inputs
multi instance - each input fires off its own version of the script

http://docs.splunk.com/Documentation/Splunk/6.1.4/AdvancedDev/ModInputsScripts#Single_or_multiple_in...

I tested it out again with the random numbers example

single instance - http://pastebin.com/77MMLqPz
multi instance example - http://pastebin.com/idDqG7XY

Now to build the systemd modular input based on this new found knowledge.

phoenixdigital
Builder

Adding threading doesnt seem to work.
http://pastebin.com/r5QyYb0T

Is there a better way I should be going about this?

0 Karma

phoenixdigital
Builder

Never mind resolved it with this change
http://pastebin.com/JfTNxQEv

My intervals were not defined when I was referencing them.

Going to have to change this a fair bit to spin off separate thread for each input in the loop

            for input_name, input_item in inputs.inputs.iteritems():

Would the best way to achieve this to have stream_events definition loop through all inputs and fire off a thread for each which has a while 1: in it?

What closes these threads when new inputs are added via the GUI?

Because if stream_events is called again there will be a second set of threads started. Do I have to put the smarts in to ensure two threads for the same input dont exist?

0 Karma
Get Updates on the Splunk Community!

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...

Platform Highlights | January 2023 Newsletter

 January 2023Peace on Earth and Peace of Mind With Business ResilienceAll organizations can start the new year ...