Splunk Search

regular expression in my lookup table

Explorer

hi,all,here is my problem:

here is my search:

mysearch | table fields1 fields2

and I got:


fields1 fields2

foofoo abcccd

barbar asdddf


the lookup table I define in lookups is as below,the keywords is regular expression which I want match the fields2


keyword fields3

abccc\w+ 10

asddd\w+ 20


what I want is

fields1 fields2 fields3

foofoo abcccd 10

barbar asdddf 20

so how can I get this done?

Super Champion

I just thought it may be worth pointing out that the mvrex command which is implemented by the SA-cim_validator app may be something worth taking a look at. While the command itself doesn't deal with lookups, values pulled back from lookups are send through this command on at least one of the dashboards:

https://github.com/hire-vladimir/SA-cim_validator/blob/master/bin/mvrex.py

Anyways, the combo of regex within lookups is pretty rare. Thought this may give some future readers some ideas to think about.

0 Karma

New Member

Hi all we have some trouble with this python script
Splunk error code
"returned error code 1"

Please Help

0 Karma

Legend

There is no regex support in static lookup tables unfortunately. You could achieve this by writing a dynamic lookup script that does this, the obvious drawback obviously being that it's a bit more hassle to roll up your sleeves and start coding.

I've written this kind of dynamic lookup for this exact purpose and have it lying around somewhere, but don't know where right now - let me know if you want it and I'll have another look.

EDIT: So, looked around and found it. DISCLAIMER, I'm by no means a real Python coder 🙂

#!/usr/bin/python

# A dynamic lookup that takes CSV as input, performs a regex match against another CSV, then returns the CSV results                                                          
import csv
import sys
import re
import os
import glob

def inlookup(inf, inval, outf):
    try:
        # The app makes the assumption that a directory in the form <customer>_indexer_config exists. If multiple
        # directories matching this template exist for some weird reason, only the first one is used.
        config_app_path = os.path.join(os.environ['SPLUNK_HOME'],'etc','apps','yourapp')
        csvname = "yourlookup.csv"
        csvpath = os.path.join(config_app_path,'lookups',csvname)
    except Exception as e:
        sys.stderr.write("No %s file found." % csvname)
        sys.exit(0)

    try:
        c = open(csvpath, 'rb')
        f = csv.DictReader(c)

        for row in f:
            if re.search(row[inf], inval):
                return row[outf]

    except Exception as e:
        sys.stderr.write(e)
        sys.exit(1)
        return []


def main():
    if len(sys.argv) != 3:
        print "Usage: %s <in field> <out field>" % (sys.argv[0])
        sys.exit(0)

    inf = sys.argv[1]
    outf = sys.argv[2]
    r = csv.DictReader(sys.stdin)
    w = csv.DictWriter(sys.stdout, r.fieldnames)
    w.writeheader()

    for result in r:
        # If all fields are already present, there's no need
        # to look anything up
        if len(result[inf]) and len(result[outf]):
            w.writerow(result)

        elif len(result[inf]):
            outvalue = inlookup(inf, result[inf], outf)
            result[outf] = outvalue
            w.writerow(result)


main()

As you can see in the start of the inlookup function you need to specify your path and lookup filename explicitly. As far as I know there's unfortunately no way of providing an argument for a lookup to consume it that way, so it needs to be hardcoded.

Motivator

transforms.conf

[UniqueID_Lookup]
external_cmd = regexpython.py Id,Name
external_type = python
fields_list = Id,Name

props.conf

LOOKUP-UniqueID_Lookup = UniqueID_Lookup Id AS Id OUTPUTNEW Name AS Name

0 Karma

Motivator

I want the Name UserDefinedCategory should be displayed in the category...but this is not workin ?? am i missin something ??

0 Karma

Motivator

Hi Ayn,

Can you pls give me the steps in exucting this ?

I have done the following , but this seems not working

  1. I have copied above in my app's bin directory i.e /opt/splunk/etc/apps/MY_APP/bin/regexpython.py with the filename regexpython.py
  2. I have my lookup file name lookup_UniqueId.csv , which has fields Id, Name

Id is the value that comes in the logs, and correspondingly it matches the Name that are present in the lookup file

  1. Now with ur code of regex . i have added this line in my lookup Id,Name ^2\d+6$,"UserDefinedCategory"

ie. if my Id is starting with 2 and ends with 6

0 Karma

Explorer

thanks,it is very helpful!

0 Karma

Legend

Amended my answer with the code I found lying around... 😉

0 Karma

Explorer

yeah,I really appreciate it if you could have another look,the problem I mentioned is a real case in my work and I stuck here.By the way I write some python script in my daily work,thanks in advance if you could provide the answer!

0 Karma