How to call an external python script to populate a column in a table?

[craigkleen]

I need a little help putting all the pieces together on this. I'm trying to build a table in Splunk that would show a given IP and its "location" on my campus network. I'm getting the location by using a Python script to connect to the REST API on my Infoblox IPAM server and grabbing the DHCP Option 82 information.

I have the script working at the command line, but I haven't been able to figure out how to use it in a search or if it's even getting data back into Splunk yet. I want to create a little dashboard that our Helpdesk people can use, so I'm only going to give them a chance to input one IP address. I was expecting the search to go something like:

| myLocationScript 127.0.0.1

The script, placed in my app bin directory, with the right permissions, and added to commands.conf, resembles:

import os,sys
import splunk.Intersplunk
import subprocess

pyScript = "~splunk/etc/apps/myapp/bin/ipLocation-system.py "

p = subprocess.call(["~splunk/etc/apps/myapp/bin/ipLocation-system.py", sys.argv[1]], stdout=sys.stdout)

Which is obviously a wrapper script. The actual script, which uses the "requests" Python module that's not in Splunk, and does the work resembles:

#!/path/to/system/python
import requests
import json
import sys
import socket

url = "https://infoblox/api/"

myip = sys.argv[1]

if sane_ipv4_address(myip):
    response = requests.get(url + "?address=" + myip, verify=True, auth=('user', 'pass'))

    if response.status_code == 200:
        # if there's actual data, not an empty []
        if len(response.text) > 2:
            j = json.loads(response.content)
            [...further string processing...]
            print location # need to add the IP here too

I probably want to extend this in the near future to take multiple IP's and gather their locations. What am I missing? Do I need to write this out to a file and oneshot it into Splunk or something?

[martin_mueller · Accepted solution]

These are the droids you're looking for: http://docs.splunk.com/Documentation/Splunk/6.1.4/Knowledge/Addfieldsfromexternaldatasources#Set_up_...

In essence, this enhances regular old lookups that use a CSV file to use a script instead that retrieves the lookup's output fields (your location data) for the input fields (your IP) from somewhere else.

[martin_mueller]

Sounds good.

The count is zero because calling | stats count without any actual search before it does just that, counting zero events 🙂

[martin_mueller]

I'm not sure if the script is behaving like a scripted lookup should, so I'll tackle the search part first. Assuming 1.2.3.4 is a known IP, run this search:

| stats count | eval clientip = "1.2.3.4" | lookup iploc clientip OUTPUT location

That removes the dependency on any events and explicitly lists the output field. If that fails then we'll have to revisit the script and check if it works the lookup interface correctly.

[craigkleen]

Aha, I changed:

p = subprocess.call(["/opt/splunk/etc/apps/search/bin/ipLocation-system.py", result[clientip]] , stdout=outfile)

To:

p = subprocess.Popen(["/opt/splunk/etc/apps/search/bin/ipLocation-system.py", result[clientip]] , stdout=subprocess.PIPE,)
  print p.communicate()[0]

And that seems to have done the trick. Of course, the "count" column is still zero, but location is filled in. ANd my main search seems to work right. Thanks again!

[craigkleen]

That made a table of 0,1.2.3.4,"".

[craigkleen]

I'm sooo close, I'm sure. Thanks for the link. But, I can't seem to get the table to populate. Here's what I have now (borrowing bits from external_lookup.py):

import csv
import os,sys
import splunk.Intersplunk
import subprocess

clientip = sys.argv[1]
location = sys.argv[2]

infile = sys.stdin
outfile = sys.stdout

r = csv.DictReader(infile)
header = r.fieldnames

w = csv.DictWriter(outfile, fieldnames=r.fieldnames)
w.writeheader()

for result in r:
  p = subprocess.call(["~splunk/etc/apps/myapp/bin/ipLocation-system.py", result[clientip]] , stdout=outfile)

My transforms.conf contains:

[iploc]
external_cmd = ipLocation.py clientip location
external_type = python
fields_list = clientip,location

Restarted Splunk, and now trying to search with:

sourcetype=firewall srcip=a.b.c.d | eval clientip = srcip | table clientip dstip | lookup iploc clientip

The location column is there, but empty. The script runs at the command line and returns the CSV with the location information (the subprocess prints it). I seem to recall reading something about waiting for output...?

At the command line, I'm running it with:

 printf "clientip,location\n134.139.2.207\n" | ~/splunk/bin/python ipLocation.py clientip location

Which returns:

clientip,location
a.b.c.d,"001-180F-D1"

Thanks again