What is this (search_startup_time) field in _audit index ?

[PowerPacked]

Hi Folks

May I know what is this search_startup_time field in this event from splunk _audit index & also would like to understand relation between exec_time & search_startup_time

I am looking to find if any ad hoc search got queued and run lately after actual search started time.

Thanks in advance

[splunker12er · Accepted solution]

search_starup_time is the time that parsing is complete and is ready to wait for responses from indexers.
exec_time is the epoch time when exactly the search was executed by the user
total_run_time is the time in seconds that has been taken for the job to complete

Also below are some interesting fields,

scanCount - The number of events that are scanned or read off disk
eventCount - The number of events returned by the search.
resultCount - The total number of results returned by the search.
eventAvailableCount - The number of events that are available for export.
dropCount - In real-time searches only, the number of possible events dropped due to queue size.

[PowerPacked]

Thanks for Reply @splunker12er

Can i also know if Splunk is writing any way to find, if Ad Hoc searches were Queued and run after exec_time.

I am looking for (Ad Hoc Searches Latency Time)

Thanks

[splunker12er]

there are 4 status options for the info field...1. completed. 2. cancelled. 3. granted 4. failed

"granted" means that the scheduler or the user was allowed to run the search. The search will run when possible.
"Completed" - once the job is done you will see this status

a job can be delayed or queued depending of the prioritization, or execution windows or concurrent search limits, etc. like (NOT "search_id='scheduler" NOT "search='|history" NOT "search='typeahead" NOT "search='| metadata type=* )

You can below query to see the searches run by users, with mainly the query , search_id, total_run_time, info, etc.. also you can modify the filter to exclude searches you are not interested

index=_audit NOT(user="splunk-system-user" OR user="admin") action=search info!="granted"|table search_id,search,scan_count,event_count,result_count,available_count,drop_count,is_realtime,exec_time,search_et,search_lt,api_et,api_lt,searched_buckets,total_run_time,info,user|eval Run_Time=toString(total_run_time,"duration")|eval exec_time=strftime(exec_time,"%d/%b/%y %H:%M:%S"),search_et=strftime(search_et,"%d/%b/%y %H:%M:%S"),search_lt=strftime(search_lt,"%d/%b/%y %H:%M:%S")|RENAME Run_Time as "Search Run Time",exec_time as "Search Exec.Time",search_et as "Search Data From", search_lt as "Search Data To"|fields - total_run_time,api_et,api_lt,available_count,drop_count,is_realtime|sort 0 -"Search Run Time"|join search_id [search index=_audit NOT(user="splunk-system-user" OR user="admin") action=search info=granted search=* NOT "search_id='scheduler" NOT "search='|history" NOT "search='typeahead" NOT "search='| metadata type=* | search totalCount>0"|fields search_id, search]

[splunker12er]

I sort results by total search run time - from there you can analyse which search by user takes how much time to get completed -

If this comments/answers help , please upvote / mark as answered

[PowerPacked]

Thanks Again @splunker12er

I am not looking at how much time it took for a search to complete.

I am looking at if any searches got queued and whats the que time

for example: this search shows what is the execution latency (Que Time) of Scheduled Searches

index=_internal sourcetype=scheduler (status="completed" OR status="skipped" OR status="deferred")
| eval window_time = if(isnotnull(window_time), window_time, 0)
| eval execution_latency = max(dispatch_time - (scheduled_time + window_time), 0)
| timechart span=1h partial=f avg(execution_latency) AS avg_exec_latency, count(eval(status=="completed" OR status=="skipped")) AS total_exec, count(eval(status=="skipped")) AS skipped_exec
| eval skip_ratio = round(skipped_exec / total_exec * 100, 2)
| eval avg_exec_latency = round(avg_exec_latency, 2) | fields _time, avg_exec_latency

But am not able to find one similar for Ad Hoc searches