Send Fields from Search to External Python Script to Update Lookup Table

[wweiland]

I'm trying to send fields that I gather from a search command and send the results to a external python script. The script will then take those results and update a lookup table. I've gotten the python script working to take 3 arguments and update the csv file. I'm having problems getting the search command to output the information. I thought I was to use the script command, but I guess that is for something else. Has anyone else done this before? Any suggestions?

Thanks in advance.

[wweiland · Accepted solution]

I went ahead and got the savedsearch working. Now I can just do a ... | table blah1 blah2 blah3 | savedsearchcmd

I had to map the command in the local/commands.conf. The python script uses fileinput to find the text and replace it. Runs like a champ. Thanks again everyone.

Tricky part in python was trying to figure out how it delivered the information.

import splunk.Intersplunk

results = []

try:

results = splunk.Intersplunk.readResults(None, None, True)
for i in results:
    node = i.get('nodes')
    jobid = i.get('jobid')
    status = i.get('typeid')

except:

import traceback

stack = traceback.format_exc()

results = splunk.Intersplunk.generateErrorResults("Error : Traceback: " + str(stack))

[martin_mueller]

Feel free to post your command's definition here if you like.

[harry2007gsp]

@wweiland, where are you bringing 'Intersplunk' from?
I could not find it either inside splunk sdk or splunklib libraries.

[wweiland]

I'm not sure if this still works. This was from 2014 and I'm not in that environment anymore.

[martin_mueller]

You can do that with native Splunk.

Load new data, append+inputlookup the existing lookup table, run stats by node or whatever you need to merge the two, pipe to outputlookup.

[martin_mueller]

Take a look at the saved searches in the SoS app, they load stuff and store it in a lookup using outputlookup.

[wweiland]

martin_mueller, do you possibly have any pointers on what you suggested?

[wweiland]

New approach. I'll see if I can get that working as well. Thanks for your input.

[wweiland]

I need something that will update lines in the table. I have a list of compute nodes and when jobs start the job id will be assigned to that node. When the job ends the node will be marked idle. The logs wont produce the states of each node at search time. I also thougt about trying to do a mysql connector, but trying to stick to native splunk if possible.

[martin_mueller]

Have you considered updating the lookup table directly from the search using outputlookup?

http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/outputlookup

[wweiland]

I added it to the commands.conf, but when I do a search | script.py it fails. Do I have to do the outputResults if I don't plan to return anything?

[linu1988]

is the script working from search bar? then you need to add the script in commands.conf. Need to include

splunk.Intersplunk.outputResults(results)

for output.

http://docs.splunk.com/Documentation/Splunk/6.0.1/AdvancedDev/SearchScripts