Splunk Search

conditional execution of search

damucka
Builder

Hello,

I have a parts of the search, which I would like to execute conditionally. In the below example I am trying to trigger a database dump based on the decision variable set before.
It all works fine, but even when the decision variable is empty, the map also gets executed and it takes really long time, like 40 seconds. I am running it in the alert on minute base, so I would like to avoid executing map at all, if not necessary.
How would I just skip the map execution in case the decision variable is empty?
Is it possible?

 |rename comment AS " *********************************** Start rtedump triggering  *********************************************  "
 | eval host_to_trigger=case(decision=1, host_to_trigger)

 |rename comment AS "In case host_to_trigger / decision are not set, assign the empty value to it, otherwise there will be an error from the map command"
 | fillnull value="" host_to_trigger
 | fillnull value="" decision
 | eval triggertime = strftime(now(),"%H:%M")
 | table host_to_trigger decision triggertime   

 | map maxsearches=20 search="dbxquery query=\"call SYS.MANAGEMENT_CONSOLE_PROC('runtimedump dump -f /usr/sap/BWP/HDB02/$host_to_trigger$/trace/DB_BWP/iAlerting_rtedump_ANOMALY_$triggertime$.trc','$host_to_trigger$:30240',?)\" connection=\"HANA_MLBSO_BWP\" | eval decision=\"$decision$\"| fields - decision"

 |rename comment AS " *********************************** End rtedump triggering  ************************************************  "
Tags (1)
0 Karma

skoelpin
SplunkTrust
SplunkTrust

You could use the "splunk search" alert action to nest the conditional search inside. Assuming the first search sets the trigger to true, it then fires the Splunk alert action which then runs your conditional search

https://splunkbase.splunk.com/app/3837/

0 Karma

damucka
Builder

Thank you.
That would be an option, but I do not find the "splunk search" under the alert actions available. Do I have to install anything specific for that?
I have the Splunk Version 7.0.0.

Kind Regards,
Kamil

0 Karma

skoelpin
SplunkTrust
SplunkTrust

I should've included the Splunkbase link in my original answer, see the edited answer.

You will need to install this on your SH(C) and once you do, this will add the additional alert action to your alerts. This allows you to paste in SPL which will fire. You can also pass results from the first search that makes up the alert to the "splunk search" alert action. This will allow you to conditionally fire SPL, I use it all the time

0 Karma

damucka
Builder

Hi,

My second search is quite a long one. I defined it as a savedsearch and trying to fire it from the "splunk search", no success. I was trying also to paste the SPL text directly into the "splunk search" entry line there, but it does not work either.

Could you perhaps help, how would I trigger the below, the best as a savedsearch?

Kind Regards,
Kamil

| makeresults

 |rename comment AS " *********************************** Set the list of hosts ***********************************************  "
 | eval master1 ="ls5920", 
        slave11 ="ls5921", 
        slave12 ="ls5922",
        slave13 ="ls5923", 
        slave14 ="ls5924",      
        master2 ="ls5925", 
        slave21 ="ls5926", 
        slave22 ="ls5927",
        slave23 ="ls5928", 
        slave24 ="ls5929"        
 | table [|makeresults |  eval search ="master1 slave11 slave12 slave13 slave14 master2 slave21 slave22 slave23 slave24" | table search ] 
 | transpose 
 | rename "row 1" AS host_to_trigger 
 | table host_to_trigger



 |rename comment AS " *********************************** Start rtedump triggering  *********************************************  "

 | eval triggertime = strftime(now(),"%H:%M")
 | table host_to_trigger triggertime   

 | map maxsearches=20 search="dbxquery query=\"call SYS.MANAGEMENT_CONSOLE_PROC('runtimedump dump -f /usr/sap/BWP/HDB02/$host_to_trigger$/trace/DB_BWP/iAlerting_rtedump_ANOMALY_$triggertime$.trc','$host_to_trigger$:30240',?)\" connection=\"HANA_MLBSO_BWP\" "

 |rename comment AS " *********************************** End rtedump triggering  ************************************************  "
0 Karma

skoelpin
SplunkTrust
SplunkTrust

You should try doing something simpler to prove it works.. I'd recommend passing a value from search one to search two to verify its working as expected then build on top of that. If you are not using a generating command, you should prefix your second query with search

0 Karma

damucka
Builder

I understand the general concept. I do not have to pass any value, the action should always get executed when the alert is triggered and this I control in the custom trigger conditions of the alert (decision = 1).
With the above I tried already everything ...
| savedsearch mysearch
search | savedsearch mysearch
search | makersesults ....
|makeresults ....
search [|makeresults]

Nothing works. I can see in the "scheduler" index that the action splunk_search was invoked but it did not do the required things. When I kick it off from the search / savedsearch, it works ...

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Yeah, thats another approach. If the first search fires because it met the triggering condition, then it fires off your second search. Once again, you should try something simple to verify it works as expected.. Like this

| search index=abc | sendemail ...

0 Karma

damucka
Builder

Additionally, I would be really interested how you solve the conditional calling of parts of SPL code, also for other cases. For the above, the recommended solution will do, but I have some other cases, where there is one alert, but depending on the anomaly reason (variable set in the alert search), different actions should be triggered, like memory clearance, dump creation, etc. With the above solution, I would have to create separate alerts per anomaly reason, which would multiply them very much.
So, what would be the best approach to implement kind of if ... else conditional calls in the search?
I know, SPL is not python, but perhaps it is possible somehow and do not see it ...

Thank you,
Regards,
Kamil

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Yes, I do this all the time. You will need to have a base search which has a trigger, this base search will represent your alert. You then add some conditional logic on there via eval to get a value. You then can pass that value to the "splunk search" alert action which will run a second search. Something like this

| eval mem_clearence=if(field>90,'mem_value',0)
| eval cpu_high=if(field>99,'cpu_value',0)

^^This would be the SPL appended to the first search that represents the alert. You then set the trigger inside the alert. If that trigger fires, it then kicks off the "splunk search" alert action which fires its open SPL. So if you want to pass the values from search one to search two, you simply add this to your second search

$result.mem_clearence$
0 Karma

damucka
Builder

Thank you, I have just installed the App and will be looking at it.

However, please forgive my ignorance, but I am still not able to see the conditionality in the example above.
Let us say, that depending on the some values from the first base search / alert I would like to either trigger mem_clearance or cpu_high actions present in the "splunk search" alert action.
Let us make the follwing example:
- the base search sets the rca_reason either to mem_clearance or cpu_high depending on some values. Now, I would like to execute either the memory or the cpu action present in the "splunk search" action. Not both of them. How would I control it?

Will this app allow me creation of multiple "splunk search" actions, which depending on that if e.g. the value of mem_clearance is mem_value or 0 will execute the corresponding memory action?
And another action for CPU?
... this would be great.

Would the output of these "splunk search" actions be also accessible in the "send e-mail" action? Like if I would like to take out some fields/variables from the memory action and send e-mail with it? Or would the e-mail action only see the variables from the main alert search?

Kind Regards,
Kamil

0 Karma

skoelpin
SplunkTrust
SplunkTrust

You would pass the value from the first search to the second search and have conditional logic in your second search which would either fire or do nothing based off the conditional logic. I think I see what you're wanting to do, you want to make your alert OO and fit multiple KPI's in a single alert rather than doing an alert for each one. amiright?

Yes, you can append |sendemail to the end of your search to email results of the second search

0 Karma

damucka
Builder

"have conditional logic in your second search which would either fire or do nothing based off the conditional logic"

This is exactly what I need help with. How would I build it? Could you please give code example? I am really struggling with executing anything conditionally (if condition, the run the subsearch for example) in Splunk.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

The conditional logic from search one sets a flag (i.e. 1 or 0 if its true or false). You pass this flagged value to the second search and it will only execute if the flag from search 1 is true. Or you use something like this

 |eval type1=if(type="Group", [search index=[INDEX] host=[HOST] |table host | head 1 | return $host], "") 
0 Karma

damucka
Builder

Unfortunately the search gets always executed, just the type1 does or does not get assigned depending on the value of type. This means there is no way to avoid the search with the above, which is not good enough for me unfortunately, because the the search is long I would like to skip it. I am still searching for something like CASE known from the programming languages where depending on the setting of the variable, different cases would get executed ...

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Yeah, Splunk supports case statements.. Like this

| eval FIELD=case(field=1,1,field=2,2) 

You need to assign a value based on conditional logic, then pass that value to search 2. You can use something like a | where field=2 in search 2. This will prevent the search from firing unless the trigger from search 1 is true

0 Karma

Richfez
SplunkTrust
SplunkTrust

I was looking for an answer to the same sort of question, if one exists, and as I read this exchange I wanted to comment.  Partly as a "better but still very unsatisfactory answer" for folks, and maybe also for a bit of "let's see if anyone else comes up with a solution".

I think the ask is really for -

Is there a way to have a code path that either is or is not *executed* depending on some condition; a variable being set or not, or being of a particular value or whatever?

In every example thus far, the code path executes regardless of the status of the variable.  You can test this yourself with a search/expression there that fails and fails *hard*. 

Here's an example. On Splunk <any edition but free> this returns either your srchJobsQuota value, or the word "skipped" depending on if you flip the true to false.  To be clear, this works fine:

| makeresults 
| eval results = if(false(), [| rest /services/authorization/roles/admin splunk_server=local | return $srchJobsQuota], "")

But if I change my test box from my dev license to the Splunk Free license, since Splunk Free doesn't support that rest call *at all* (LOL - "Payment Required"), then no matter which way false/true is flipped, this fails.

In both cases in Splunk Free, true and false, you get this same error:

Error in 'eval' command: The expression is malformed. An unexpected character is reached at ', "skipped")'.
[subsearch]: Unexpected status for to fetch REST endpoint uri=https://127.0.0.1:8089/services/authorization/roles/admin?count=0 from server=https://127.0.0.1:8089 - Payment Required 

So it's executing that code without regard to whether it's in the active path or not.  Because it's a subsearch, they run first.

I thought maybe I could get around this using map, but all attempts so far that I've tried have failed because I can't find a way to run an arbitrary string (which can easily be created from an if like above) from map.  It looks almost like you should be able to, but either it just won't work or I haven't found the magic combination.  And I've tried quite a bit...

I'll think on it some more, maybe there will be more suggestions from the peanut gallery too.  🙂

Happy Splunking,

Rich

 

0 Karma

Richfez
SplunkTrust
SplunkTrust

And of course, the very next link I hadn't yet followed yet, was a answer by the inimitable Martin, https://community.splunk.com/t5/Archive/How-do-you-interpret-string-variable-as-SPL-in-Map-function/...

which if you can create a macro do magic on your string, then this can be done.

It's totally a hack, but at least it's something.

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...