topic Re: A search that makes searches and executes them in Splunk Search

A search that makes searches and executes them

rkoster — Fri, 08 Apr 2016 22:46:27 GMT

So I have this search that I believe makes other searches from a list of regexs that I have stored in a csv.

[ | inputlookup regex_test.csv
| eval search_this = "[search sourcetype=proxy | eval Comment=\""+Comment+"\" | regex uri=\""+RegEx+"\"]"
| stats values(search_this) AS all_searches
| eval search="| multisearch "
| eval search=search+mvjoin(all_searches, " ")
| fields search ]

The above search fails:

Search process did not exit cleanly, exit_code=0, description="killed by signal 6: Aborted". Please look in search.log for this peer in the Job Inspector for more info.
 This search has encountered a fatal error and has been marked as zombied.

However the search that is created from the query above does not fail when you input into the search bar (yes it returns results on match):

 | multisearch [search sourcetype=proxy| eval Comment="EITest v1" | regex uri="^http:\/\/(?:(?!www|authorize|sdranch|azbar|finra|porsche|money|blog|forums?|it)[^\x2e]+\.[^\x2e]+\.com[^\x2f]*|[^\x2e]+\.tk)\/[ik](?=[^\n]{0,500}[a-z\x2d]{17,100})(?=[^\n]{0,500}[a-z\/\x2d]{60,200})(?:[a-z\x2d]+\/){4,32}(?:index\.html)?$"] [search sourcetype=proxy| eval Comment="EITest v2" | regex uri="^http:\/\/[^\x2f]+\.[a-z]{2}\/(?=[^\x3f]+[a-z0-9]{17,}\/)(?:[a-z0-9]+\/){2,}(?:index\.php)?$"]

Apart from additional caffeine, what am I missing to make this great success?

Have I chosen the wrong command or the wrong way to go about creating this search of searches? The goal is to have a list of RegEx,Comment and search for matches against each RegEx in the list.

Thanks for taking some time to think about this one!

Re: A search that makes searches and executes them

martin_mueller — Sun, 10 Apr 2016 16:10:12 GMT

Your search string assembly cannot work like that. After the stats you have only one field, all_searches. Then you do something with "multisearch", but immediately overwrite that in the final eval.

Additionally, the multisearch approach is inherently slow if all you want is to test a set of regular expressions. Instead, you could assemble one large regular expression made up of your individual expressions separated with an or'ing pipe.

Lastly, you mention Comment - how does that fit into your search? I can't see that referenced anywhere.

Re: A search that makes searches and executes them

rkoster — Tue, 12 Apr 2016 17:30:34 GMT

Hi Martin,

Thanks for taking some time to review and provide feedback.

I will edit the search in the question to include the appropriately ordered syntax; I re-ordered the query a bit while I was posting in an attempt to make it a bit more readable and thought through.

Assembling one large regex, while it would return results, will not reach my end goal which would include the "Comment" field from the lookup. I assumed that multisearch would allow for transforming commands such as adding a:

| stats values(Comment) AS Comment values(uri) AS uri by domain

However multisearch will not do that: Error in 'multisearch' command: Multisearch subsearches may only contain purely streaming operations (subsearch 1 contains a non-streaming command.)

So I am left with a scenario where I have two columns in a CSV with multiple rows: RegEx, Comment

I want to search my proxy logs for matches on each regex in order to return the Comment, domain, and uri by domain.

Re: A search that makes searches and executes them

martin_mueller — Tue, 12 Apr 2016 19:15:04 GMT

I see... this is slightly hacky, but should work:

sourcetype=proxy [inputlookup regex_test.csv | eval search = "| eval Comment = if(match(uri, \"".RegEx."\"), \"".Comment."\", Comment) " | fields search | mvcombine search | nomv search] | stats values(Comment) AS Comment values(uri) AS uri by domain

Depending on how frequently you update your lookup, you might want to take a look at my Regex IDS app at https://splunkbase.splunk.com/app/2805/ - I'm sure that approach can be modified to fit your case in a less hacky way.

Re: A search that makes searches and executes them

rkoster — Tue, 12 Apr 2016 22:42:37 GMT

I think that will work! Thank you!

It is important that the Comment field within the CSV has any quotes escaped otherwise the eval command that is created terminates too early.

To remove the non-matching regexs (to set this up for an alarm) I added this to your query:

| where Comment!=""

Re: A search that makes searches and executes them

woodcock — Fri, 29 Apr 2016 13:45:06 GMT

Take a look at this Q&A for a solution that you should be able to modify to fit:

https://answers.splunk.com/answers/386488/regex-in-lookuptable.html#answer-387536