Hi Folks
May I know what is this search_startup_time field in this event from splunk _audit index & also would like to understand relation between exec_time & search_startup_time
I am looking to find if any ad hoc search got queued and run lately after actual search started time.
Thanks in advance
search_starup_time
is the time that parsing is complete and is ready to wait for responses from indexers.
exec_time
is the epoch time when exactly the search was executed by the user
total_run_time
is the time in seconds that has been taken for the job to complete
Also below are some interesting fields,
scanCount
- The number of events that are scanned or read off disk
eventCount
- The number of events returned by the search.
resultCount
- The total number of results returned by the search.
eventAvailableCount
- The number of events that are available for export.
dropCount
- In real-time searches only, the number of possible events dropped due to queue size.
search_starup_time
is the time that parsing is complete and is ready to wait for responses from indexers.
exec_time
is the epoch time when exactly the search was executed by the user
total_run_time
is the time in seconds that has been taken for the job to complete
Also below are some interesting fields,
scanCount
- The number of events that are scanned or read off disk
eventCount
- The number of events returned by the search.
resultCount
- The total number of results returned by the search.
eventAvailableCount
- The number of events that are available for export.
dropCount
- In real-time searches only, the number of possible events dropped due to queue size.
Thanks for Reply @splunker12er
Can i also know if Splunk is writing any way to find, if Ad Hoc searches were Queued and run after exec_time.
I am looking for (Ad Hoc Searches Latency Time)
Thanks
there are 4 status options for the info field...1. completed. 2. cancelled. 3. granted 4. failed
"granted
" means that the scheduler or the user was allowed to run the search. The search will run when possible.
"Completed
" - once the job is done you will see this status
a job can be delayed or queued depending of the prioritization, or execution windows or concurrent search limits, etc. like (NOT "search_id='scheduler" NOT "search='|history" NOT "search='typeahead" NOT "search='| metadata type=*
)
You can below query to see the searches run by users, with mainly the query , search_id, total_run_time, info, etc.. also you can modify the filter to exclude searches you are not interested
index=_audit NOT(user="splunk-system-user" OR user="admin") action=search info!="granted"|table search_id,search,scan_count,event_count,result_count,available_count,drop_count,is_realtime,exec_time,search_et,search_lt,api_et,api_lt,searched_buckets,total_run_time,info,user|eval Run_Time=toString(total_run_time,"duration")|eval exec_time=strftime(exec_time,"%d/%b/%y %H:%M:%S"),search_et=strftime(search_et,"%d/%b/%y %H:%M:%S"),search_lt=strftime(search_lt,"%d/%b/%y %H:%M:%S")|RENAME Run_Time as "Search Run Time",exec_time as "Search Exec.Time",search_et as "Search Data From", search_lt as "Search Data To"|fields - total_run_time,api_et,api_lt,available_count,drop_count,is_realtime|sort 0 -"Search Run Time"|join search_id [search index=_audit NOT(user="splunk-system-user" OR user="admin") action=search info=granted search=* NOT "search_id='scheduler" NOT "search='|history" NOT "search='typeahead" NOT "search='| metadata type=* | search totalCount>0"|fields search_id, search]
I sort results by total search run time - from there you can analyse which search by user takes how much time to get completed -
If this comments/answers help , please upvote / mark as answered
Thanks Again @splunker12er
I am not looking at how much time it took for a search to complete.
I am looking at if any searches got queued and whats the que time
for example: this search shows what is the execution latency (Que Time) of Scheduled Searches
index=_internal sourcetype=scheduler (status="completed" OR status="skipped" OR status="deferred")
| eval window_time = if(isnotnull(window_time), window_time, 0)
| eval execution_latency = max(dispatch_time - (scheduled_time + window_time), 0)
| timechart span=1h partial=f avg(execution_latency) AS avg_exec_latency, count(eval(status=="completed" OR status=="skipped")) AS total_exec, count(eval(status=="skipped")) AS skipped_exec
| eval skip_ratio = round(skipped_exec / total_exec * 100, 2)
| eval avg_exec_latency = round(avg_exec_latency, 2) | fields _time, avg_exec_latency
But am not able to find one similar for Ad Hoc searches