Getting Data In

How to measure Execution Latency of Ad-hoc Searches?

Builder

Hi Everyone

How to determine and measure if any Ad hoc Searches are getting queued and by what time on total?
Basically, I would like to have a panel showing the execution latency of Ad-Hoc searches.

Thanks in advance

0 Karma
1 Solution

SplunkTrust
SplunkTrust

hello there,

thank you for the question, got me thinking a lot about it.
i am not 100% sure of the solution, but here is what i got:
looks like, splunk generates the values for search_id field based on the time you pressed Enter or hit the Search icon
values are in epoch with milliseconds, with single quotes around example: '1522959775.983'
also, Splunk captures the execution time, in the field exec_time - epoch without milliseconds
i think my search might need some more filtering but hopefully its a good start for you:

index="_audit" action=search  NOT search_id=*scheduler* NOT saved_search=*
| rex field=search_id "'(?<s_id>\d+\.\d{3})'"
| table _time s_id exec_time total_run_time
| eval ad_hoc_latency = exec_time - s_id

see screenshot below:
alt text

hope it helps

View solution in original post

SplunkTrust
SplunkTrust

hello there,

thank you for the question, got me thinking a lot about it.
i am not 100% sure of the solution, but here is what i got:
looks like, splunk generates the values for search_id field based on the time you pressed Enter or hit the Search icon
values are in epoch with milliseconds, with single quotes around example: '1522959775.983'
also, Splunk captures the execution time, in the field exec_time - epoch without milliseconds
i think my search might need some more filtering but hopefully its a good start for you:

index="_audit" action=search  NOT search_id=*scheduler* NOT saved_search=*
| rex field=search_id "'(?<s_id>\d+\.\d{3})'"
| table _time s_id exec_time total_run_time
| eval ad_hoc_latency = exec_time - s_id

see screenshot below:
alt text

hope it helps

View solution in original post

Explorer

Thank you very much!
I was looking for such a solution 🙂

I endet up with this one:

index="_audit" action=search  NOT search_id=*scheduler* NOT saved_search=*
| rex field=search_id "'.*_(?<s_id>\d+\.\d+)'"
| eval ad_hoc_latency = round(exec_time - s_id, 3)
| eval ad_hoc_latency = max(ad_hoc_latency,0)
| table _time s_id exec_time total_run_time, ad_hoc_latency
| where ad_hoc_latency>0
| eval Description=case(ad_hoc_latency>0 AND ad_hoc_latency<=0.5,"0-0.5", ad_hoc_latency>0.5 AND ad_hoc_latency<=2,"0.5-2", ad_hoc_latency>2 AND ad_hoc_latency<=5,"2-5", ad_hoc_latency>5 AND ad_hoc_latency<=15,"5-15",ad_hoc_latency>15,">15")
| timechart span=10m count by Description
0 Karma

Motivator

Hey@PowerPacked,

Have a look at this accepted answer:
https://answers.splunk.com/answers/583285/how-to-list-ad-hocscheduled-searches-in-order-of-c.html
Let me know if this helps!!

0 Karma

Builder

Thanks for reply @deepashri_123

I know there are couple of searches in DMC about most memory consuming & long running, But i am specifically interested in Execution Latency of Ad - Hoc Searches as i want to monitor the latency.

Even there is a search in DMC about execution latency of Scheduled Searches, but not about ad-hoc.
& am not sure if splunk is writing fields like Search Dispatch time & Search Start time about Ad -hoc Searches.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!