How to Apply Dynamic Regex Matching in a Multisearch Using Values from a Lookup Table?

[varma364]

Hello Splunk experts,

I’m currently trying to create a search using a multisearch command where I need to dynamically apply regex patterns from a lookup file to the Web.url field in a tstats search.

When I use my current approach, it directly adds the regex value as a literal search condition instead of applying it as a regex filter. For example, instead of dynamically matching URLs with the regex, it ends up as if it’s searching for the literal pattern.

I have a lookup that contains fields like url_regex and other filter parameters, and I need to:

1. Dynamically use these regex patterns in the search, so that only URLs matching the regex from the lookup get processed further.

2. Ensure that the logic integrates correctly within a multisearch, where the base search is filtered dynamically based on these values from the lookup.

I’ve shared some screenshots showing the query and the resulting issue, where the regex appears to be used incorrectly. How can I properly use these regex values to match URLs instead of treating them as literal strings?

Search :- 

| inputlookup my_lookup_file
| search Justification="Lookup Instructions"
| fields url_regex, description
| fillnull value="*"
| eval url_regex="Web.url=\"" . url_regex . "\""
| eval filter="source=\"my_sourcetype\" " . "filter_field=" . " \""
| eval search="| tstats `summariesonly` prestats=true count from datamodel=Web where sourcetype=\"" . filter . " by Web.url Web.user"
| stats values(search) as search
| eval search=multisearch [
mvjoin(search, " | ")
] . "

| stats count by search"

As highlighted in the yellow from above I wanted to have the regex matching string instead of the direct regex search from events?

Also, lastly, once the multisearch query generates another search as output, how can I automatically execute that resulting search within my main query?

Any guidance would be greatly appreciated!

[yuanliu]

Congratulations for heeding @PickleRick's advice and repost your search in text.  Now, let me try to understand this use case.  You are trying to use a lookup file to generate SPL code for some other purpose. For that generated code, you wish to use multisearch.  But that multisearch has nothing to do with the question itself.  Is this accurate?

Then, you want use the returned values from inputlookup as regex to match an indexed field named Web.url in a tstats command.  Is this correct?

Documentation on tstats will tell you that the where clause of this command can only accept filters applicable in search command; in fact, only a fraction of these filters.  In other words, you cannot use those regex directly in tstats command.

This is not to say that your search goal cannot be achieved.  You just need to restructure the subsearches so you can use the where command instead of where clause in tstats.  But let me first point out that your text illustration of the search not only does not match your screenshot, but also is wrong because url_regex is no longer used in the field filter, therefore no longer used in formulation of the search field.  You cannot possibly get the output as your screenshot show.  There is another "transcription" error in the last eval command as well because the syntax is incorrect.

Correcting for those errors and simplifying the commands, here is something you can adapt:

| inputlookup my_lookup_file where Justification="Lookup Instructions"
| eval search = "[| tstats `summariesonly` prestats=true count from datamodel=Web where sourcetype=\"mysourcetype\" by Web.url Web.user | where match(Web.url, \"" . url_regex . "\")]"
| stats values(search) as search
| eval search = "| multisearch " . mvjoin(search, "
")

Suppose your my_lookup_file contains the following entries (ignoring description field as it is not being used; also ignore fillnull because "*" is not a useful regex to match any URL.)

url_regex

regex

[re]gex

^regex

regex$

the above search will give you

search

| multisearch [| tstats `summariesonly` prestats=true count from datamodel=Web where sourcetype="mysourcetype" by Web.url Web.user | where match(Web.url, "[re]gex")] [| tstats `summariesonly` prestats=true count from datamodel=Web where sourcetype="mysourcetype" by Web.url Web.user | where match(Web.url, "^regex")] [| tstats `summariesonly` prestats=true count from datamodel=Web where sourcetype="mysourcetype" by Web.url Web.user | where match(Web.url, "regex")] [| tstats `summariesonly` prestats=true count from datamodel=Web where sourcetype="mysourcetype" by Web.url Web.user | where match(Web.url, "regex$")]

Is this what you are looking for?

Here is full emulation to get the above input and output:

| makeresults format=csv data="url_regex
regex
[re]gex
^regex
regex$"
``` the above emulates
| inputlookup my_lookup_file where Justification="Lookup Instructions"
```
| eval search = "[| tstats `summariesonly` prestats=true count from datamodel=Web where sourcetype=\"mysourcetype\" by Web.url Web.user | where match(Web.url, \"" . url_regex . "\")]"
| stats values(search) as search
| eval search = "| multisearch " . mvjoin(search, "
")

Play with it and compare with your real lookup.

[PickleRick]

1. Please don't post screenshots - copy-paste your code and results into code blocks or preformatted paragraphs. It makes it easier for everyone and is searchable.

2. You're trying to do something that is generally not supported - you can generate conditions for a search dynamically by means of subsearch, not whole searches. To some extent you could use the map command but it is relatively limited.

3. You can't use multisearch with non-streaming commands (like tstats).

[varma364]

thank you for the response and I’ve updated the query now. 

[PickleRick]

OK. Now let's back up a little.

Explain in your own words, without using SPL what business problem you're trying to solve here. What are you trying to achieve?

You're clearly trying to "implement non-SPL thing in SPL" which is usually not a very good idea. Or at least not a very efficient one. And same things can often be achieved in a different way.