topic Re: How do you lookup match field names by wildcard or regex? in Splunk Search

How do you lookup match field names by wildcard or regex?

xshen_anji — Fri, 29 Mar 2019 14:08:38 GMT

I have some customer provided CSV lookup files. These lookup files have some "similar" field names, which means they contain some common keywords. I would like do a keyword match in lookup command to these similar fields. Is there a way to do this ?

An example is:

lookup file1's title is like: population, average income, location
lookup file2's title is like: population, income, location

I would like to use the field that contains "income" as the lookup condition, how can I do this with one common lookup statement?

Re: How do you lookup match field names by wildcard or regex?

somesoni2 — Fri, 29 Mar 2019 15:42:15 GMT

How are you planning to run lookup on both lookup table files? What all have you tried so far? Any specific reason to have common lookup statement?

Re: How do you lookup match field names by wildcard or regex?

xshen_anji — Sat, 30 Mar 2019 14:51:37 GMT

Basically, I am running an app which processes a lookup file with some data files. The lookup csv file, which is generated from reporting systems of different vendors, varied a little bit in title fields, but the keywords are basically the same . Since there are a lot of customers, it would be easy to manage if I have one lookup statement to support all these lookup files. Now I am trying to make it a rule to all the customers, asking for manually editing the titles to make them the same, but it would still be desirable to tolerate some kind of fault or difference.

Re: How do you lookup match field names by wildcard or regex?

ddrillic — Sat, 30 Mar 2019 23:08:22 GMT

According to How to use wildcard in lookup-based searches and alerts?

You can specify -

 match_type = WILDCARD(income)

In the transforms.conf definition of your lookup.

I just tested it, I have -

$SPLUNK_HOME/etc/apps/search/local

$ cat transforms.conf 

[hosts_reporting]
batch_index_query = 0
case_sensitive_match = 1
filename = hosts_reporting.csv
match_type = WILDCARD(host)

One of the host names in hosts_reporting.csv is the beginning of a host name and it comes up via -

index=_internal [ | inputlookup hosts_reporting.csv | eval host=host + "*" ]

But it doesn't when running -

index=_internal [ | inputlookup hosts_reporting ]

Isn't it weird?

Re: How do you lookup match field names by wildcard or regex?

ddrillic — Mon, 01 Apr 2019 01:32:45 GMT

Any thoughts on this one, by any chance? @woodcock?

Re: How do you lookup match field names by wildcard or regex?

woodcock — Mon, 01 Apr 2019 02:03:55 GMT

See here for how to lookup with RegEx:
https://answers.splunk.com/answers/386488/regex-in-lookuptable.html

Re: How do you lookup match field names by wildcard or regex?

woodcock — Mon, 01 Apr 2019 02:09:45 GMT

You are misunderstanding the 2 different ways to use lookup files. One way is with the ... | lookup command syntax, which uses the WILDCARD() syntax (among other settings) within the Lookup definitions, the other is with the |inputlookup command syntax which DOES NOT interact with the Lookup definitions. In the latter case, just do something like this:

index=_internal [ |inputlookup hosts_reporting.csv | eval host=host + "*" ]

Re: How do you lookup match field names by wildcard or regex?

ddrillic — Mon, 01 Apr 2019 10:54:26 GMT

Very interesting @woodcock - thank you for the clarification.

Re: How do you lookup match field names by wildcard or regex?

woodcock — Mon, 01 Apr 2019 20:24:25 GMT

If you have your answer, pick one and click Accept to close the question.

Re: How do you lookup match field names by wildcard or regex?

ddrillic — Mon, 01 Apr 2019 21:39:52 GMT

Hi @woodcock - it's not mine - I just joined the ride ; -)