Hello, I am really struggling with trying to find a data source to get all the following pieces of information about the searches which are run on our splunk enterprise cluster. These are the fields I am interesting in finding.
total run time
search type (real time, scheduled, ad hoc, remote etc)
status (running, paused, finalized, completed etc)
messages (any error or warn messages)
The scheduler logs are in the _internal index. If you use the Monitoring Console, you'll also find some good stuff that will point you to data sources that will be helpful for looking at scheduler behavior under Search -> Scheduler Activity.
By tweaking one of the Monitoring Console searches a little, I get something like this which covers most of the fields you are looking for:
index=_internal host= sourcetype=scheduler
| eval alert_actions = if(isnull(alert_actions) OR alert_actions == "", "none", alert_actions)
| eval window_time = if(isnotnull(window_time), window_time, 0)
| eval execution_latency = max(dispatch_time - (scheduled_time + window_time), 0)
| dedup sid
| table sid, user, app,scheduled_time, dispatch_time, run_time, result_count, status
However, that will only give you info for the scheduled searches and not the real time, ad hoc, or remote searches. For those, you'll want to take a look at the audit index. If you do a transaction on the searchid field, you can look at other fields like search for the search string, is_realtime to determine if it's realtime. The _audit index logs will also tell you the start and end time ranges of the search and how many events were scanned while it was running.
+1 for starting with search section in the monitoring console! Excellent place to open panels in search and see where it gets it's goodies! It also has a toggle for ad-hoc searches. Otherwise Activity > Jobs in the top right corner of your screen can be a good place if you are digging into particular jobs...
Also shout out to the search activity app https://splunkbase.splunk.com/app/2632/#/details