looping through all rows of lookup file and match

[ankitarath2011]

Hi,

I have a lookup file with following structure.

pattern,output_value
 "aaa\s+:\d\d:", 2
 "aaa\s+:\d:", 1

For my index I need match each pattern from this file for each event and then take the corresponding output_value. Something like mentioned below.

For each_event in all_events_for_my_idx:
     for line in lookup_patterns:
           if match(line[0], each_event):
                my_val = line[1]
                break

I am new to Splunk, so wrote the logic like other program .

[maciep]

how many patterns do you have? if it's no many, you can just use eval command with case/match functions.

... | eval my_val = case(match(_raw,"aaa\s+:\d\d:"),2,match(_raw,"aaa\s+:\d:"),1,true(),"n/a")

If you have a lot that might be a bit more difficult. Splunk does not have a regex match built into the lookup functionality - only wildcard and cidr.

And being new to Splunk, you'll have to rework your brain a bit from programming to puzzle solving. In general (but not exactly always true), when you pipe your search results to a new command, splunk loops through all of the events - that's your outer loop. And so you need to munge your results through one or commands/pipes until it's the way you want it - in this case we're using eval to create a new field on each iteration

[ankitarath2011]

Hey Thanks. There are many pattern actually and requirement is such that we can not keep it in query. We need to keep it in some configurable file (e.g. lookup) where more can be added.

[maciep]

maybe in that case you can put it in props.conf as an EVAL .... and then my_val would just automatically be created for that sourcetype?

or if it's needed in a dashboard, the a lookup could be used to ultimately create a token (dashboard var) with the case functionality and then that token used in the search.

or maybe (but unlikely) something could be done to add each pattern as a new field to the data and then foreach used to loop through and test your field against that pattern - could work maybe, but pretty ugly.

the problem is that the way you want to use the lookup, isn't what lookups were designed for, with respect to the regex matching part. that looping/matching is inherently done with the lookup command and that command knows nothing of regex i believe.

[ankitarath2011]

Yes I understand, I am ok with any other file if lookup file will not work in such case. But, I need to take it from file. Any suggestion on that??

For props, can we keep case statement in props.??

[maciep]

Correct. In the Web, it's considered a Calculated Field (Settings -> Fields -> Calculated Fields) as is typically applied to a sourcetype. And on the backend that is just written to props.conf

[your_sourcetype]
EVAL-my_val = case(match(_raw,"aaa\s+:\d\d:"),2,match(_raw,"aaa\s+:\d:"),1,true(),"n/a")

Not sure what your environment looks like, but just note that this is considered a knowledge object and as such has permissions (who can read/write it) and scope (private/app/global) associated with it.

[mayurr98]

hey @ankitarath2011,

The pattern value that you have in the lookup table has to be in the Splunk events as well. I am assuming that it's not in the events and by default Splunk doesn't have a pattern field, however, it has punct field. If you could tweak your lookup table in the punct format then you could use that lookup table to give corresponding output_value to each event.

index=<your_index> | lookup lookup_table punct OUTPUT output_value

Refer doc for punct field
https://docs.splunk.com/Documentation/Splunk/7.3.1/Knowledge/Usedefaultfields#punct

[ankitarath2011]

No, this is not working. The pattern column will have reg ex patterns that I want to match with one of the fields (Id) in the events. The Id field will have values like "abc:22:", "aaa:9:"

index=<your_index> | lookup lookup_table punct OUTPUT output_value

The above one is not working as it needs exact match not reg ex pattern. And moreover, for some of the patterns will not have punct in it.