Splunk Search

Why is my external lookup python script unable to create file table.csv when using the lookup command in a search?

vincenteous
Communicator

Hi Everyone,

I have created a python script which will get data from a web service as an external lookup. Within my script, I create a csv file as a temporary table to prevent the lookup from getting data from the web service again for the same key. Below is the script which is located in $SPLUNK_HOME/etc/system/bin:

#!usr/bin/env python

import sys
import csv
import urllib
import os.path

def getmsisdn(IMSI):
        try:
                msisdnlist = urllib.urlopen("http://xx.xx.xx.xx:xxxx/?oprid=lstisdn&show=ISDN&imsi=" + IMSI + "&trx_id=SPLK############")
                lines = msisdnlist.readlines()
                temp = lines[0]
                msisdn = temp[3:]

                with open('/apps/splunk/etc/system/bin/table.csv', 'a') as csvfile:
                        fieldnames = ['IMSI', 'MSISDN']
                        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
                        writer.writerow({'IMSI': IMSI, 'MSISDN': msisdn})

                return msisdn
        except:
                return ''

def lookup(IMSI):
        if os.path.isfile('/apps/splunk/etc/system/bin/table.csv'):
                found = True
                with open('/apps/splunk/etc/system/bin/table.csv', 'r') as csvfile:
                        fieldnames = ['IMSI', 'MSISDN']
                        reader = csv.DictReader(csvfile, fieldnames=fieldnames)

                        for row in reader:
                                if (row['IMSI'] == IMSI) and (row['MSISDN'] != ''):
                                        return row['MSISDN']
                                else: found = False

                if not found: return getmsisdn(IMSI)
        else:
                return getmsisdn(IMSI)

def main():
        if len(sys.argv) != 3:
                print "Usage: python external_lookup.py [IMSI field] [MSISDN field]"
                sys.exit(0)

        IMSIf = sys.argv[1]
        MSISDNf = sys.argv[2]

        r = csv.reader(sys.stdin)

        w = ''
        header = []
        first = True


        for line in r:
                if first:
                        header = line
                        if MSISDNf not in header or IMSIf not in header:
                                print "IMSI and MSISDN fields must exist in CSV data"
                                sys.exit(0)

                        w = csv.DictWriter(sys.stdout,header)
                        first = False

                result = {}
                i = 0

                while i < len(header):
                        if i < len(line):
                                result[header[i]] = line[i]
                        else:
                                result[header[i]] = ''
                        i += 1

                if len(result[IMSIf]) and len(result[MSISDNf]):
                        w.writerow(result)
                elif len(result[IMSIf]):
                        result[MSISDNf] = lookup(result[IMSIf])
                        if result[MSISDNf]:
                                w.writerow(result)

main()

Here is the configuration I have put within transforms.conf:

[MSISDNlookup]
external_cmd = external_lookup3.py IMSIf MSISDNf
external_type = python
fields_list = IMSIf,MSISDNf

When I ran the script from the command line, it worked just fine. Tables.csv was properly created. Unfortunately, when I used this using the lookup command in a Splunk search, only the lookup to the web service succeeded, but the table.csv was not created at all.

Is this a bug? I am using Splunk 6.1.3 and it is a production environment, so if I were forced to upgrade, it would be hard.

Thanks in advance.

Best Regards,

Vincent

0 Karma
1 Solution

vincenteous
Communicator

After a long struggle, I've finally found that csv file. It's not that the file wasn't created, but it's created in an entirely different place with the same specified directory. I put the script inside my search head and table.csv was created in my indexer with the same directory as I've specified within my script.

View solution in original post

vincenteous
Communicator

After a long struggle, I've finally found that csv file. It's not that the file wasn't created, but it's created in an entirely different place with the same specified directory. I put the script inside my search head and table.csv was created in my indexer with the same directory as I've specified within my script.

View solution in original post

.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!