I have a slightly odd ball request, and I hope you guys can help me out.
Once of our main uses for Splunk is operational dashboards, we are able to show a lot of interesting information about how our network services are performing by analysing a variety of sources. However there are times where I need to drill down on the network an collect additional information about something very specific and in real time. As such I would like to trigger an additional "collection" when a form is submitted, this script would inject records into my Splunk index and my dashboard would have a realtime search for this datasource. Note that most of the time I would not collect this data for a number of reasons, mostly to do with capacity of management networks and device tolerance to intensive polling.
By way of example, my dashboard might find that I device failed its configuration, I would like to be able to have a form that does detailed analysis on the device, when that form is submitted a script may ping the device an log its output to splunk. The form has a realtime search for these ping results.
I can think of a few ways of doing this, most of them are ugly, im interested in your ideas?
Sounds like you need a custom Splunk command. Docs here.
I recently had a request to have Splunk interact with a file on the server, eliminating the need to edit the file via the CLI. As this would have to be on a potentially regular basis I did this with a Form so the user could select field values, and then pipe the populating search to the custom command with the scripts arguments being the fields selected by the user.
Hope this helps you on your way.
As a first step towards building a command to do this, I wrote a simple generating command that produces some random strings in a few fields which splunk would process. It's called "randomstrings". I just can't seem to get splunk to fire off the command though: I can see the command's config via the browser and I'm logging the invocations, it just doesn't seem to be getting started even with GETINFO (I have supports_getinfo set in the command.conf.
The command works when I run it from the command line. Where can I look to see why splunk won't give it some love too?