Execution of multiple macros in a search take too much time

[emipintus]

Hi, I have an alert which executes a very simple search. The search consists of a macro invoked 40 times, each time with different input parameters. This is the search inside the alert:

`macro_level_alarm(tag="C245", volmax="1520", name="245")`
| append
[search `macro_level_alarm(tag="C246", volmax="1520", name="246")`]
...
...
...
| append
[search `macro_level_alarm(tag="C518", volmax="600", name="518")`]
| table Name ValueLatestEvent TimeLatestEvent

The macro's query is again very simple. It simple executes a plain search over the latest 15 minutes and gets the most recent value. Here is the query:

    source="***" index="***" (Tag="$tag$") 
    | streamstats latest(_time) as latest_time by Tag| where _time=latest_time
    | eval ValueLatestEvent=round(((Value*100)/$volmax$),1) 

    | eval Name= "$name$" 
    | convert timeformat="%Y-%m-%d %H:%M" ctime(_time) AS TimeLatestEvent
    | table Name ValueLatestEvent TimeLatestEvent

The alert is scheduled to execute every 4 hours.
The problem I'm facing is that, when I execute this alert invoking the macro only 10 times, everything is fine and I get the result in few seconds. If I try to invoke the macro more than 10 times (eg: 11 or 40 as in this case) Splunk gets stucked in parsing, then times out and returns no result.
I guess there are some parameters to change inside limits.conf. I've already modified

[search]
max_rt_search_multiplier

but I'm not sure this is the right one as nothing changed and my alert still gets stucked.
Do you have ever experienced anything like this and have any idea on how to solve it?

Thank you so much

[dmarling · Accepted solution]

Would you be open to rewriting your query and to stop using a macro? It would be faster and have less parsing time if you used a single query with 40 OR statements between your tags. It would be more complicated on your eval's for Name and ValueLatestEvent as you would need to have a large case statement to account for the combination of factors, but it should be doable. If you have this combination of tag, volmax, and name list into a lookup file you could pass the values from that file into your search a couple of different ways. I'd be willing to help if you could provide. Here's an example that removes the streamstats to find the latest event as that is unnecessary when you use the latest command with a stats.

source="***" index="***" (Tag="C245" OR Tag="C246" OR Tag="C518") 
| stats latest(_time) as _time latest(Value) as Value by Tag 
| eval ValueLatestEvent=round((Value*100)/case(Tag="C245" OR Tag="C246", 1520, Tag="C518", 600),1)  
| rex field=Tag "C(?<Name>\d+)"
| convert timeformat="%Y-%m-%d %H:%M" ctime(_time) AS TimeLatestEvent 
| table Name ValueLatestEvent TimeLatestEvent

This is just using the three examples you provided, but the only tricky part here is the eval for ValueLatestEvent as you will need to modify that case statement so each of your examples are accounted for.

[emipintus]

Actually, the idea of using a macro was mine and not given by the customer. I thought it would've been easier and faster to write the alert (and indeed it was), but I didn't think to execution time. I tried your proposal and it worked perfectly. The execution time is now very low and the alert is working properly. Thank you very much for your kind help

[dmarling]

Glad it worked! I converted it to an answer. If you could please accept it when you get a free moment, I would appreciate it!