Splunk Search

Execution of multiple macros in a search take too much time

emipintus
Explorer

Hi, I have an alert which executes a very simple search. The search consists of a macro invoked 40 times, each time with different input parameters. This is the search inside the alert:

`macro_level_alarm(tag="C245", volmax="1520", name="245")`
| append
[search `macro_level_alarm(tag="C246", volmax="1520", name="246")`]
...
...
...
| append
[search `macro_level_alarm(tag="C518", volmax="600", name="518")`]
| table Name ValueLatestEvent TimeLatestEvent

The macro's query is again very simple. It simple executes a plain search over the latest 15 minutes and gets the most recent value. Here is the query:

    source="***" index="***" (Tag="$tag$") 
    | streamstats latest(_time) as latest_time by Tag| where _time=latest_time
    | eval ValueLatestEvent=round(((Value*100)/$volmax$),1) 

    | eval Name= "$name$" 
    | convert timeformat="%Y-%m-%d %H:%M" ctime(_time) AS TimeLatestEvent
    | table Name ValueLatestEvent TimeLatestEvent

The alert is scheduled to execute every 4 hours.
The problem I'm facing is that, when I execute this alert invoking the macro only 10 times, everything is fine and I get the result in few seconds. If I try to invoke the macro more than 10 times (eg: 11 or 40 as in this case) Splunk gets stucked in parsing, then times out and returns no result.
I guess there are some parameters to change inside limits.conf. I've already modified

[search]
max_rt_search_multiplier

but I'm not sure this is the right one as nothing changed and my alert still gets stucked.
Do you have ever experienced anything like this and have any idea on how to solve it?

Thank you so much

0 Karma
1 Solution

dmarling
Builder

Would you be open to rewriting your query and to stop using a macro? It would be faster and have less parsing time if you used a single query with 40 OR statements between your tags. It would be more complicated on your eval's for Name and ValueLatestEvent as you would need to have a large case statement to account for the combination of factors, but it should be doable. If you have this combination of tag, volmax, and name list into a lookup file you could pass the values from that file into your search a couple of different ways. I'd be willing to help if you could provide. Here's an example that removes the streamstats to find the latest event as that is unnecessary when you use the latest command with a stats.

source="***" index="***" (Tag="C245" OR Tag="C246" OR Tag="C518") 
| stats latest(_time) as _time latest(Value) as Value by Tag 
| eval ValueLatestEvent=round((Value*100)/case(Tag="C245" OR Tag="C246", 1520, Tag="C518", 600),1)  
| rex field=Tag "C(?<Name>\d+)"
| convert timeformat="%Y-%m-%d %H:%M" ctime(_time) AS TimeLatestEvent 
| table Name ValueLatestEvent TimeLatestEvent

This is just using the three examples you provided, but the only tricky part here is the eval for ValueLatestEvent as you will need to modify that case statement so each of your examples are accounted for.

If this comment/answer was helpful, please up vote it. Thank you.

View solution in original post

dmarling
Builder

Would you be open to rewriting your query and to stop using a macro? It would be faster and have less parsing time if you used a single query with 40 OR statements between your tags. It would be more complicated on your eval's for Name and ValueLatestEvent as you would need to have a large case statement to account for the combination of factors, but it should be doable. If you have this combination of tag, volmax, and name list into a lookup file you could pass the values from that file into your search a couple of different ways. I'd be willing to help if you could provide. Here's an example that removes the streamstats to find the latest event as that is unnecessary when you use the latest command with a stats.

source="***" index="***" (Tag="C245" OR Tag="C246" OR Tag="C518") 
| stats latest(_time) as _time latest(Value) as Value by Tag 
| eval ValueLatestEvent=round((Value*100)/case(Tag="C245" OR Tag="C246", 1520, Tag="C518", 600),1)  
| rex field=Tag "C(?<Name>\d+)"
| convert timeformat="%Y-%m-%d %H:%M" ctime(_time) AS TimeLatestEvent 
| table Name ValueLatestEvent TimeLatestEvent

This is just using the three examples you provided, but the only tricky part here is the eval for ValueLatestEvent as you will need to modify that case statement so each of your examples are accounted for.

If this comment/answer was helpful, please up vote it. Thank you.

emipintus
Explorer

Actually, the idea of using a macro was mine and not given by the customer. I thought it would've been easier and faster to write the alert (and indeed it was), but I didn't think to execution time. I tried your proposal and it worked perfectly. The execution time is now very low and the alert is working properly. Thank you very much for your kind help

0 Karma

dmarling
Builder

Glad it worked! I converted it to an answer. If you could please accept it when you get a free moment, I would appreciate it!

If this comment/answer was helpful, please up vote it. Thank you.
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...