Monitoring Splunk

whats does index=_audit contain?

Communicator

Im trying to understand what does - all the field value pairs under _audit index refer to , but not able to find the right doc in splunk.

I want to understand what apiStartTime, apiEndTime , totalruntime , searchruntime , runtime , scan_count etc mean??

I'm trying to find the execution time of searches and set up an alert when a search is running for a long duration and found many articles on this.

But I want to understand the basic first i.e the various contents of _audit index and its meaning. Could someone kindly guide me and provide the splunk doc related to this?

My aim is to find the execution time of a search and set up an alert for a long running search.

Tags (2)
0 Karma

Super Champion

I couldn't find documentation of _audit to that level of information.
But if your aim is to find execution time and alerting, the best is to use DMC

DMC -> Search -> Activity -> Search Usage Statistics: Deployment -> Long running Queries

A sample query which you can run in instance having DMC is like below and you can easily put alerts on them

`dmc_audit_get_searches_for_groups(*)`       | stats min(_time) as _time, values(user) as user, max(total_run_time) as total_run_time, first(search) as search, first(search_type) as search_type, first(apiStartTime) as apiStartTime, first(apiEndTime) as apiEndTime by search_id, host       | where isnotnull(search) AND search_type="ad hoc" |             search user="*" search="*"             | eval earliest = case(like(apiStartTime, "%ZERO_TIME%") AND like(apiEndTime, "%ZERO_TIME%"), "all time", like(apiStartTime, "%ZERO_TIME%"), "-", 1 == 1, apiStartTime )             | eval latest = case(like(apiStartTime, "%ZERO_TIME%") AND like(apiEndTime, "%ZERO_TIME%"), "all time", like(apiEndTime, "%ZERO_TIME%"), "-", 1 == 1, apiEndTime )              | `dmc_time_format(_time)`             | stats max(total_run_time) as total_run_time by search, _time, earliest, latest, search_type, user, host, search_id             | sort - total_run_time              | eval total_run_time = `dmc_convert_runtime(total_run_time)`             | eval total_run_time = if(isnotnull(total_run_time), total_run_time, "-")             | fields search, total_run_time, _time, earliest, latest, search_type, user, host, search_id             | rename search as "Report Name/Search String", total_run_time as "Search Runtime", _time as "Search Start", earliest as "Earliest Time", latest as "Latest Time", search_type as Type, user as "User", host as "Host", search_id as SID     
0 Karma

Communicator

@koshyk Thankyou for your response . I did notice this under DMC and we have this DMC on our license server and I dont find anyresults for this.

I would like to understand _audit and content and provide a dashboard on the Search Head cluster.

0 Karma