Why is my external lookup python script unable to create file table.csv when using the lookup command in a search?

[vincenteous]

Hi Everyone,

I have created a python script which will get data from a web service as an external lookup. Within my script, I create a csv file as a temporary table to prevent the lookup from getting data from the web service again for the same key. Below is the script which is located in $SPLUNK_HOME/etc/system/bin:

#!usr/bin/env python

import sys
import csv
import urllib
import os.path

def getmsisdn(IMSI):
        try:
                msisdnlist = urllib.urlopen("http://xx.xx.xx.xx:xxxx/?oprid=lstisdn&show=ISDN&imsi=" + IMSI + "&trx_id=SPLK############")
                lines = msisdnlist.readlines()
                temp = lines[0]
                msisdn = temp[3:]

                with open('/apps/splunk/etc/system/bin/table.csv', 'a') as csvfile:
                        fieldnames = ['IMSI', 'MSISDN']
                        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
                        writer.writerow({'IMSI': IMSI, 'MSISDN': msisdn})

                return msisdn
        except:
                return ''

def lookup(IMSI):
        if os.path.isfile('/apps/splunk/etc/system/bin/table.csv'):
                found = True
                with open('/apps/splunk/etc/system/bin/table.csv', 'r') as csvfile:
                        fieldnames = ['IMSI', 'MSISDN']
                        reader = csv.DictReader(csvfile, fieldnames=fieldnames)

                        for row in reader:
                                if (row['IMSI'] == IMSI) and (row['MSISDN'] != ''):
                                        return row['MSISDN']
                                else: found = False

                if not found: return getmsisdn(IMSI)
        else:
                return getmsisdn(IMSI)

def main():
        if len(sys.argv) != 3:
                print "Usage: python external_lookup.py [IMSI field] [MSISDN field]"
                sys.exit(0)

        IMSIf = sys.argv[1]
        MSISDNf = sys.argv[2]

        r = csv.reader(sys.stdin)

        w = ''
        header = []
        first = True


        for line in r:
                if first:
                        header = line
                        if MSISDNf not in header or IMSIf not in header:
                                print "IMSI and MSISDN fields must exist in CSV data"
                                sys.exit(0)

                        w = csv.DictWriter(sys.stdout,header)
                        first = False

                result = {}
                i = 0

                while i < len(header):
                        if i < len(line):
                                result[header[i]] = line[i]
                        else:
                                result[header[i]] = ''
                        i += 1

                if len(result[IMSIf]) and len(result[MSISDNf]):
                        w.writerow(result)
                elif len(result[IMSIf]):
                        result[MSISDNf] = lookup(result[IMSIf])
                        if result[MSISDNf]:
                                w.writerow(result)

main()

Here is the configuration I have put within transforms.conf:

[MSISDNlookup]
external_cmd = external_lookup3.py IMSIf MSISDNf
external_type = python
fields_list = IMSIf,MSISDNf

When I ran the script from the command line, it worked just fine. Tables.csv was properly created. Unfortunately, when I used this using the lookup command in a Splunk search, only the lookup to the web service succeeded, but the table.csv was not created at all.

Is this a bug? I am using Splunk 6.1.3 and it is a production environment, so if I were forced to upgrade, it would be hard.

Thanks in advance.

Best Regards,

Vincent

[vincenteous · Accepted solution]

After a long struggle, I've finally found that csv file. It's not that the file wasn't created, but it's created in an entirely different place with the same specified directory. I put the script inside my search head and table.csv was created in my indexer with the same directory as I've specified within my script.