How do you lookup match field names by wildcard or regex?

[xshen_anji]

I have some customer provided CSV lookup files. These lookup files have some "similar" field names, which means they contain some common keywords. I would like do a keyword match in lookup command to these similar fields. Is there a way to do this ?

An example is:

lookup file1's title is like: population, average income, location
lookup file2's title is like: population, income, location

I would like to use the field that contains "income" as the lookup condition, how can I do this with one common lookup statement?

[woodcock]

You are misunderstanding the 2 different ways to use lookup files. One way is with the ... | lookup command syntax, which uses the WILDCARD() syntax (among other settings) within the Lookup definitions, the other is with the |inputlookup command syntax which DOES NOT interact with the Lookup definitions. In the latter case, just do something like this:

index=_internal [ |inputlookup hosts_reporting.csv | eval host=host + "*" ]

[ddrillic]

Very interesting @woodcock - thank you for the clarification.

[woodcock]

If you have your answer, pick one and click Accept to close the question.

[ddrillic]

Hi @woodcock - it's not mine - I just joined the ride ; -)

[woodcock]

See here for how to lookup with RegEx:
https://answers.splunk.com/answers/386488/regex-in-lookuptable.html

[ddrillic]

According to How to use wildcard in lookup-based searches and alerts?

You can specify -

 match_type = WILDCARD(income)

In the transforms.conf definition of your lookup.

I just tested it, I have -

$SPLUNK_HOME/etc/apps/search/local

$ cat transforms.conf 

[hosts_reporting]
batch_index_query = 0
case_sensitive_match = 1
filename = hosts_reporting.csv
match_type = WILDCARD(host)

One of the host names in hosts_reporting.csv is the beginning of a host name and it comes up via -

index=_internal [ | inputlookup hosts_reporting.csv | eval host=host + "*" ]

But it doesn't when running -

index=_internal [ | inputlookup hosts_reporting ]

Isn't it weird?

[ddrillic]

Any thoughts on this one, by any chance? @woodcock?

[somesoni2]

How are you planning to run lookup on both lookup table files? What all have you tried so far? Any specific reason to have common lookup statement?

[xshen_anji]

Basically, I am running an app which processes a lookup file with some data files. The lookup csv file, which is generated from reporting systems of different vendors, varied a little bit in title fields, but the keywords are basically the same . Since there are a lot of customers, it would be easy to manage if I have one lookup statement to support all these lookup files. Now I am trying to make it a rule to all the customers, asking for manually editing the titles to make them the same, but it would still be desirable to tolerate some kind of fault or difference.