Monitoring Splunk

whats does index=_audit contain?


Im trying to understand what does - all the field value pairs under _audit index refer to , but not able to find the right doc in splunk.

I want to understand what apiStartTime, apiEndTime , total_run_time , search_run_time , runtime , scan_count etc mean??

I'm trying to find the execution time of searches and set up an alert when a search is running for a long duration and found many articles on this.

But I want to understand the basic first i.e the various contents of _audit index and its meaning. Could someone kindly guide me and provide the splunk doc related to this?

My aim is to find the execution time of a search and set up an alert for a long running search.

Tags (2)
0 Karma

Super Champion

I couldn't find documentation of _audit to that level of information.
But if your aim is to find execution time and alerting, the best is to use DMC

DMC -> Search -> Activity -> Search Usage Statistics: Deployment -> Long running Queries

A sample query which you can run in instance having DMC is like below and you can easily put alerts on them

`dmc_audit_get_searches_for_groups(*)`       | stats min(_time) as _time, values(user) as user, max(total_run_time) as total_run_time, first(search) as search, first(search_type) as search_type, first(apiStartTime) as apiStartTime, first(apiEndTime) as apiEndTime by search_id, host       | where isnotnull(search) AND search_type="ad hoc" |             search user="*" search="*"             | eval earliest = case(like(apiStartTime, "%ZERO_TIME%") AND like(apiEndTime, "%ZERO_TIME%"), "all time", like(apiStartTime, "%ZERO_TIME%"), "-", 1 == 1, apiStartTime )             | eval latest = case(like(apiStartTime, "%ZERO_TIME%") AND like(apiEndTime, "%ZERO_TIME%"), "all time", like(apiEndTime, "%ZERO_TIME%"), "-", 1 == 1, apiEndTime )              | `dmc_time_format(_time)`             | stats max(total_run_time) as total_run_time by search, _time, earliest, latest, search_type, user, host, search_id             | sort - total_run_time              | eval total_run_time = `dmc_convert_runtime(total_run_time)`             | eval total_run_time = if(isnotnull(total_run_time), total_run_time, "-")             | fields search, total_run_time, _time, earliest, latest, search_type, user, host, search_id             | rename search as "Report Name/Search String", total_run_time as "Search Runtime", _time as "Search Start", earliest as "Earliest Time", latest as "Latest Time", search_type as Type, user as "User", host as "Host", search_id as SID     
0 Karma


@koshyk Thankyou for your response . I did notice this under DMC and we have this DMC on our license server and I dont find anyresults for this.

I would like to understand _audit and content and provide a dashboard on the Search Head cluster.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!