How to measure Execution Latency of Ad-hoc Searches?

[PowerPacked]

Hi Everyone

How to determine and measure if any Ad hoc Searches are getting queued and by what time on total?
Basically, I would like to have a panel showing the execution latency of Ad-Hoc searches.

Thanks in advance

[adonio · Accepted solution]

hello there,

thank you for the question, got me thinking a lot about it.
i am not 100% sure of the solution, but here is what i got:
looks like, splunk generates the values for search_id field based on the time you pressed Enter or hit the Search icon
values are in epoch with milliseconds, with single quotes around example: '1522959775.983'
also, Splunk captures the execution time, in the field exec_time - epoch without milliseconds
i think my search might need some more filtering but hopefully its a good start for you:

index="_audit" action=search  NOT search_id=*scheduler* NOT saved_search=*
| rex field=search_id "'(?<s_id>\d+\.\d{3})'"
| table _time s_id exec_time total_run_time
| eval ad_hoc_latency = exec_time - s_id

see screenshot below:

hope it helps

[splunk_arz]

Thank you very much!
I was looking for such a solution 🙂

I endet up with this one:

index="_audit" action=search  NOT search_id=*scheduler* NOT saved_search=*
| rex field=search_id "'.*_(?<s_id>\d+\.\d+)'"
| eval ad_hoc_latency = round(exec_time - s_id, 3)
| eval ad_hoc_latency = max(ad_hoc_latency,0)
| table _time s_id exec_time total_run_time, ad_hoc_latency
| where ad_hoc_latency>0
| eval Description=case(ad_hoc_latency>0 AND ad_hoc_latency<=0.5,"0-0.5", ad_hoc_latency>0.5 AND ad_hoc_latency<=2,"0.5-2", ad_hoc_latency>2 AND ad_hoc_latency<=5,"2-5", ad_hoc_latency>5 AND ad_hoc_latency<=15,"5-15",ad_hoc_latency>15,">15")
| timechart span=10m count by Description

[deepashri_123]

Hey@PowerPacked,

Have a look at this accepted answer:
https://answers.splunk.com/answers/583285/how-to-list-ad-hocscheduled-searches-in-order-of-c.html
Let me know if this helps!!

[PowerPacked]

Thanks for reply @deepashri_123

I know there are couple of searches in DMC about most memory consuming & long running, But i am specifically interested in Execution Latency of Ad - Hoc Searches as i want to monitor the latency.

Even there is a search in DMC about execution latency of Scheduled Searches, but not about ad-hoc.
& am not sure if splunk is writing fields like Search Dispatch time & Search Start time about Ad -hoc Searches.