Splunk Search

help with eval if needed

Contributor

I have the following search:

|makeresults | eval trigger=0|eval decision=if(trigger==1,
[
| makeresults

 |rename comment AS " *********************************** Set the list of hosts ***********************************************  "
 | eval master1 ="ls5920", 
        slave11 ="ls5921", 
        slave12 ="ls5922",
        slave13 ="ls5923", 
        slave14 ="ls5924",      
        master2 ="ls5925", 
        slave21 ="ls5926", 
        slave22 ="ls5927",
        slave23 ="ls5928", 
        slave24 ="ls5929"        
 | table [|makeresults |  eval search ="master1 slave11 slave12 slave13 slave14 master2 slave21 slave22 slave23 slave24" | table search ] 
 | transpose 
 | rename "row 1" AS host_to_trigger 
 | table host_to_trigger


 |rename comment AS " *********************************** Start rtedump triggering  *********************************************  "

 | eval triggertime = strftime(now(),"%H:%M")
 | table host_to_trigger triggertime   

 | map maxsearches=20 search="dbxquery query=\"call SYS.MANAGEMENT_CONSOLE_PROC('runtimedump dump -f /usr/sap/BWP/HDB02/$$host_to_trigger$$/trace/DB_BWP/iAlerting_rtedump_ANOMALY_$$triggertime$$.trc','$$host_to_trigger$$:30240',?)\" connection=\"HANA_MLBSO_BWP\" "

 |rename comment AS " *********************************** End rtedump triggering  ************************************************  "

|eval t=1|return $t
]

,"0")

Now, there are following problems with the above:
- the search inside if statement gets triggered, although the trigger is clearly set to 0 before, why?
- I get the following error at the end:
Error in 'eval' command: The expression is malformed. An unexpected character is reached at ',"0")'.
This I do not understand, because when I replace the search inside the if with the simple substitution to 1, all works fine:

|makeresults | eval trigger=1|eval decision=if(trigger==1,1,"0")

Could you please advise?
Am I demanding too much from the if statement? And if yes, how else would I do the logic above?

Kind Regards,
Kamil

0 Karma
1 Solution

SplunkTrust
SplunkTrust

Hi @damucka,

Instead of making a trigger using if you can use tokens to build your trigger and leverage the depends function on the panel to allow control over the trigger.

That would make the logic you are trying to build a lot easier as you will only have to write the query you wish to trigger in the event where trigger==1.

More details on using depends can be found here :
https://docs.splunk.com/Documentation/Splunk/7.3.0/Viz/PanelreferenceforSimplifiedXML#Shared_attribu...

Let me know if you need more help with that.

Cheers,
David

View solution in original post

0 Karma

Contributor

Hello,

I am trying to conditionally call the map command at the end of the following alert code below. At the moment it works this way that if the condtions are not met, then the map command gets executed anyway with empty variables (hosttotrigger), which causes load on the DB anyway, each minute.
This is just one of the actions (map) that I would like to call conditionally. So the question is a bit generic:
- how would I call the parts of the code, given some variables being set before?

Kind Regards,
Kamil

| makeresults

 |rename comment AS " *********************************** Set the list of hosts ***********************************************  "
 | eval master1 ="ls5920", 
        slave11 ="ls5921", 
        slave12 ="ls5922",
        slave13 ="ls5923", 
        slave14 ="ls5924",      
        master2 ="ls5925", 
        slave21 ="ls5926", 
        slave22 ="ls5927",
        slave23 ="ls5928", 
        slave24 ="ls5929"        
 | table [|makeresults |  eval search ="master1 slave11 slave12 slave13 slave14 master2 slave21 slave22 slave23 slave24" | table search ] 
 | transpose 
 | rename "row 1" AS host_to_trigger 
 | table host_to_trigger

|rename comment AS " *********************************** Evaluate if there was an Anomaly Alert *******************************  "
 |eval rtetrigger = [search index=_internal sourcetype=scheduler 
    savedsearch_name="Anomaly Detection BWP" 
    alert_actions="*email*"
    earliest=-60m latest=now
    | stats count as resultcount
    | eval rtetrigger=case(resultcount>0, "1",1<2,"0") | return $rtetrigger]

|rename comment AS " ****** Check how many rtedumps have been triggered already from iAlerting *********************************  "    
  |eval rtecount = [search index=_internal sourcetype=scheduler 
    savedsearch_name="Anomaly Detection BWP - rtedump" 
    alert_actions="*email*"
    earliest=-60m latest=now
    | stats count as resultcount
    | eval rtecount = resultcount | return $rtecount]

|rename comment AS " ****** Evaluate if there were dumps from hanasitter triggered in the last 3 min ***************************  "   
  | eval hanasitter = 
  [|dbxquery query="select * from m_service_thread_Samples where thread_detail = 'runtimedump dump' and user_name <> 'BWP_PSP_PERF_DB' 
    and seconds_between(timestamp,current_timestamp) < 180 order by timestamp desc" connection="HANA_MLBSO_BWP" 
   | stats count as resultcount | eval hanasitter = case(resultcount>0, "1",1<2,"0")  
  | return $hanasitter]

 | eval decision = case(hanasitter=0 AND rtetrigger=1 AND rtecount < 3,"1",1<2,"")

 |rename comment AS " *********************************** Start rtedump triggering  *********************************************  "
 | eval host_to_trigger=case(decision=1, host_to_trigger)

 |rename comment AS "In case host_to_trigger / decision are not set, assign the empty value to it, otherwise there will be an error from the map command"
 | fillnull value="" host_to_trigger
 | fillnull value="" decision
 | eval triggertime = strftime(now(),"%H:%M")
 | table host_to_trigger decision triggertime   

 | map maxsearches=20 search="dbxquery query=\"call SYS.MANAGEMENT_CONSOLE_PROC('runtimedump dump -f /usr/sap/BWP/HDB02/$host_to_trigger$/trace/DB_BWP/iAlerting_rtedump_ANOMALY_$triggertime$.trc','$host_to_trigger$:30240',?)\" connection=\"HANA_MLBSO_BWP\" | eval decision=\"$decision$\"| fields - decision"

 |rename comment AS " *********************************** End rtedump triggering  ************************************************  "
0 Karma

Contributor

Hello,

In this specific example the solution was easy, applying "where" helped.
Before that I am dumping the variables to the file, in order to restore them afterwards because the map command wipes them out.
See below.

Kind Regards,
Kamil

 | eval triggertime = strftime(now(),"%H:%M")
 | table host_to_trigger decision triggertime   

 | outputtext usexml=false | fields decision host_to_trigger triggertime| fields - _raw  | outputcsv rtetriggering_BWP.txt
 | where isnotnull(host_to_trigger) and isnotnull(decision)

 | map maxsearches=20 search="dbxquery query=\"call SYS.MANAGEMENT_CONSOLE_PROC('runtimedump dump -f /usr/sap/BWP/HDB02/$host_to_trigger$/trace/DB_BWP/iAlerting_rtedump_ANOMALY_$triggertime$.trc','$host_to_trigger$:30240',?)\" connection=\"HANA_MLBSO_BWP\" | eval decision=\"$decision$\"| fields - decision"
0 Karma

SplunkTrust
SplunkTrust

Hi @damucka,

Instead of making a trigger using if you can use tokens to build your trigger and leverage the depends function on the panel to allow control over the trigger.

That would make the logic you are trying to build a lot easier as you will only have to write the query you wish to trigger in the event where trigger==1.

More details on using depends can be found here :
https://docs.splunk.com/Documentation/Splunk/7.3.0/Viz/PanelreferenceforSimplifiedXML#Shared_attribu...

Let me know if you need more help with that.

Cheers,
David

View solution in original post

0 Karma

Contributor

Hello David,

Thank you.
The truth is I am badly in need of some Splunk function that would allow me execute parts of my code conditionally. The tokens and depends sound interesting, but from the doku it looks like it is related to dashboards, which I do not have.

Let me shortly describe what I want to achieve. I have quite complicated alert, which:
- selects the db with dbxquery and fetches the table of anomalous hosts and the corresponding root causes
- now, for all anomalous hosts the database rtedump generation should be triggered, this is the part above. But only if the anomaly is there, not always - and this is the first condition I am having problems with. Here I need some "if" functionality, that depending on anomaly decision would trigger the map queries. At the moment they get executed no matter what and it takes time and causes issues.
- But this is not all, depending on the root cause of the anomaly, which I get at the very beginning from my dbxquery, I would like to trigger some dependant actions, being at the end another dbxqueries to the database.

... and I am having massive problems with this, I would not have expected the conditional logic to be at Splunk side.

What I tried so far was:
- eval if statements as above, but as soon as the content of if becomes a bit more complicated, the if statement seems to give up
- I shifted the "to be triggered" logic into the separate savedsearch naively thinking that if statement would have less issues with it, but the result was the same
- I tried to call the "to be triggered" logic from the "splunk search" alert action, either directly or again as a savedseach. This does not work either.

I would really need help with this.
What I am thinking now is to call python out of the alert action, but then I am not sure how much I can come back from the python call back to the alert flow, like passing back the result of the python calling my savedsearch back to alert in order to send an e-mail with it ...

So, surely because of my misunderstanding of the if concept of Splunk SPL, the thing becomes somehow complicated. I can try to workaround it for some parts, but at the end it boils down to the same question:
- is it possible to conditionally execute parts of the SPL or not. This issue hits me more and more last days.
If needed I can paste the full example of course.

Kind Regards,
Kamil

0 Karma

SplunkTrust
SplunkTrust

Yes, you expect too much from if. Like @diogofgm says, what exactly are you trying to accomplish?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

Motivator

What are you trying to accomplish here? There might be other way to do it.

------------
Hope I was able to help you. If so, an upvote would be appreciated.
0 Karma