topic Re: regular expression in my lookup table in Splunk Search

regular expression in my lookup table

fengl2 — Fri, 26 Apr 2013 07:26:35 GMT

hi,all,here is my problem:

here is my search:

mysearch | table fields1 fields2

and I got:

fields1 fields2

foofoo abcccd

barbar asdddf

the lookup table I define in lookups is as below,the keywords is regular expression which I want match the fields2

keyword fields3

abccc\w+ 10

asddd\w+ 20

what I want is

fields1 fields2 fields3

foofoo abcccd 10

barbar asdddf 20

so how can I get this done?

Re: regular expression in my lookup table

Ayn — Fri, 26 Apr 2013 07:58:42 GMT

There is no regex support in static lookup tables unfortunately. You could achieve this by writing a dynamic lookup script that does this, the obvious drawback obviously being that it's a bit more hassle to roll up your sleeves and start coding.

I've written this kind of dynamic lookup for this exact purpose and have it lying around somewhere, but don't know where right now - let me know if you want it and I'll have another look.

EDIT: So, looked around and found it. DISCLAIMER, I'm by no means a real Python coder 🙂

#!/usr/bin/python

# A dynamic lookup that takes CSV as input, performs a regex match against another CSV, then returns the CSV results                                                          
import csv
import sys
import re
import os
import glob

def inlookup(inf, inval, outf):
    try:
        # The app makes the assumption that a directory in the form <customer>_indexer_config exists. If multiple
        # directories matching this template exist for some weird reason, only the first one is used.
        config_app_path = os.path.join(os.environ['SPLUNK_HOME'],'etc','apps','yourapp')
        csvname = "yourlookup.csv"
        csvpath = os.path.join(config_app_path,'lookups',csvname)
    except Exception as e:
        sys.stderr.write("No %s file found." % csvname)
        sys.exit(0)

    try:
        c = open(csvpath, 'rb')
        f = csv.DictReader(c)

        for row in f:
            if re.search(row[inf], inval):
                return row[outf]

    except Exception as e:
        sys.stderr.write(e)
        sys.exit(1)
        return []


def main():
    if len(sys.argv) != 3:
        print "Usage: %s <in field> <out field>" % (sys.argv[0])
        sys.exit(0)

    inf = sys.argv[1]
    outf = sys.argv[2]
    r = csv.DictReader(sys.stdin)
    w = csv.DictWriter(sys.stdout, r.fieldnames)
    w.writeheader()

    for result in r:
        # If all fields are already present, there's no need
        # to look anything up
        if len(result[inf]) and len(result[outf]):
            w.writerow(result)

        elif len(result[inf]):
            outvalue = inlookup(inf, result[inf], outf)
            result[outf] = outvalue
            w.writerow(result)


main()

As you can see in the start of the inlookup function you need to specify your path and lookup filename explicitly. As far as I know there's unfortunately no way of providing an argument for a lookup to consume it that way, so it needs to be hardcoded.

Re: regular expression in my lookup table

fengl2 — Fri, 26 Apr 2013 13:43:23 GMT

yeah,I really appreciate it if you could have another look,the problem I mentioned is a real case in my work and I stuck here.By the way I write some python script in my daily work,thanks in advance if you could provide the answer!

Re: regular expression in my lookup table

Ayn — Fri, 26 Apr 2013 14:15:45 GMT

Amended my answer with the code I found lying around... 😉

Re: regular expression in my lookup table

fengl2 — Sat, 27 Apr 2013 02:34:53 GMT

thanks,it is very helpful!

Re: regular expression in my lookup table

rakesh_498115 — Mon, 04 Nov 2013 14:37:14 GMT

Hi Ayn,

Can you pls give me the steps in exucting this ?

I have done the following , but this seems not working

I have copied above in my app's bin directory i.e /opt/splunk/etc/apps/MY_APP/bin/regexpython.py with the filename regexpython.py
I have my lookup file name lookup_UniqueId.csv , which has fields Id, Name

Id is the value that comes in the logs, and correspondingly it matches the Name that are present in the lookup file

Now with ur code of regex . i have added this line in my lookup Id,Name ^2\d+6$,"UserDefinedCategory"

ie. if my Id is starting with 2 and ends with 6

Re: regular expression in my lookup table

rakesh_498115 — Mon, 04 Nov 2013 14:37:58 GMT

I want the Name UserDefinedCategory should be displayed in the category...but this is not workin ?? am i missin something ??

Re: regular expression in my lookup table

rakesh_498115 — Mon, 28 Sep 2020 15:12:19 GMT

transforms.conf

[UniqueID_Lookup]
external_cmd = regexpython.py Id,Name
external_type = python
fields_list = Id,Name

props.conf

LOOKUP-UniqueID_Lookup = UniqueID_Lookup Id AS Id OUTPUTNEW Name AS Name

Re: regular expression in my lookup table

tony_alibelli — Tue, 13 Jan 2015 09:18:13 GMT

Hi all we have some trouble with this python script
Splunk error code
"returned error code 1"

Please Help

Re: regular expression in my lookup table

Lowell — Fri, 10 Feb 2017 17:47:25 GMT

I just thought it may be worth pointing out that the mvrex command which is implemented by the SA-cim_validator app may be something worth taking a look at. While the command itself doesn't deal with lookups, values pulled back from lookups are send through this command on at least one of the dashboards:

https://github.com/hire-vladimir/SA-cim_validator/blob/master/bin/mvrex.py

Anyways, the combo of regex within lookups is pretty rare. Thought this may give some future readers some ideas to think about.