Hello,
When I ran
index=_audit NOT user="splunk-system-user"
|stats count by action
I find that accelerate_search and search is fairy high.
So I was wandering which is panel search and which is adhoc search, which is adhoc search, which is alert (schedule_search). We need to create a report about performance.
The DMC comes with most reports you will want to start with. From here you can start to understand some behavior and then when ready deep dive the individual panels to understand what indexes have which values and what those values mean.
DMC -> Monitoring Console -> Search
Here you can look at "Search usage Statistics: Deployment" OR "Search Activity: Deployment", you can try instance but that is a single server view and not the entire picture if you have search head clusters.
Here is a basic interpretation of the start of a DMC search - there is more to it but it is a good place to start.
index=_audit sourcetype=audittrail action=search search_id!="rsa_*"
| eval user=if((user == "n/a"),null(),user), search_type=case(match(search_id,"^SummaryDirector_"),"summarization",match(search_id,"^((rt_)?scheduler__|alertsmanager_)"),"scheduled",match(search_id,"\\d{10}\\.\\d+(_[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12})?$"),"ad hoc",true(),"other"), search=if((isnull(savedsearch_name) OR (savedsearch_name == "")),search,savedsearch_name)
| stats min(_time) as _time, values(user) as user, max(total_run_time) as total_run_time, first(search) as search, first(search_type) as search_type, first(apiStartTime) as apiStartTime, first(apiEndTime) as apiEndTime by search_id, host