Splunk Search
Highlighted

How do I optimize a search with too many eval commands?

Explorer

The search used looks like this:

index=my_sanitized_index_name sourcetype=web_access_logs | timechart count(eval(x_Status < 400)) as Success count(eval(x_Status>=400)) as Failure | addtotal row=f col=t labelfield=_time | eval SuccessRate = 100 * Success / (Success + Failure)

Where x_Status is the return code from a web server farm's collective access logs

In its current form, I did a search for the "last 15 minutes". It found 859,984 events and took 25.384 seconds to execute. I need to run this thing to give me stats for the last week. It usually stops working around the third hour.

I am completely new to Splunk, I am certain there is a better way to do this. I just don't know what that is. Help?

0 Karma
Highlighted

Re: How do I optimize a search with too many eval commands?

SplunkTrust
SplunkTrust

IMHO, your problem is not with the number of eval commands (which isn't that many), but with the sheer number of events you're trying to process. Those hundreds of millions of events should be distributed among many indexers for best performance.

But that doesn't help you now. Consider running your search across small intervals like you're doing, but save the results in a summary index. Then run another search to collect the data from the summary search into your weekly report. That will be much faster.

---
If this reply helps you, an upvote would be appreciated.
Highlighted

Re: How do I optimize a search with too many eval commands?

SplunkTrust
SplunkTrust
0 Karma
Highlighted

Re: How do I optimize a search with too many eval commands?

Explorer

That sounds promising. Can you link me to any documentation you would recommend as a good resource for a beginning Splunk user?

0 Karma
Highlighted

Re: How do I optimize a search with too many eval commands?

SplunkTrust
SplunkTrust

Well, The one above is good. A video is available here

http://www.splunk.com/view/SP-CAAACZW

0 Karma
Highlighted

Re: How do I optimize a search with too many eval commands?

SplunkTrust
SplunkTrust

See if this make some improvements.

index=my_sanitized_index_name sourcetype=web_access_logs | eval Type=if(x_Status<400,"Success","Failure") | timechart count by Type | addtotal row=f col=t labelfield=_time | eval SuccessRate = 100 * Success / (Success + Failure)
0 Karma
Highlighted

Re: How do I optimize a search with too many eval commands?

SplunkTrust
SplunkTrust

The only field you need from those events is the x_status and the _time, so explicitly get rid of everything else the first moment that you can, using the fields command. Try this and see what happens to your search time.

 index=my_sanitized_index_name sourcetype=web_access_logs | fields _time x_status 
| timechart count(eval(x_Status < 400)) as Success count(eval(x_Status>=400)) as Failure 
| addtotal row=f col=t labelfield=_time 
| eval SuccessRate = 100 * Success / (Success + Failure)
Highlighted

Re: How do I optimize a search with too many eval commands?

Explorer

This cut down execution time by 75%. I still should probably look into setting up summary indices, but this was very helpful. Thank you.

0 Karma
Highlighted

Re: How do I optimize a search with too many eval commands?

SplunkTrust
SplunkTrust

You're welcome. richgalloway's answer is probably your best long-term solution, but for efficiency in splunk you should also always get rid of all the data you don't need at the earliest opportunity.

Highlighted

Re: How do I optimize a search with too many eval commands?

SplunkTrust
SplunkTrust

Agree. In fact, would suggest to implement @DalJeanis's optimization method to your summary index search (and any other search) solution of @richgalloway.