I have been tasked to find a way to report on the overall query load to our Splunk system by customers that we have using it. The information I need shows up in the job inspector. Is that information stored in the _internal index anywhere, or is there another data source which I can query to set up dashboards for this purpose?
It might be here:
| rest /services/search/jobs
Is this data logged? Is there a config param that can be enabled to write this info to logs?
Would be nice to analyze when looking at long running / expensive searches.
Anything can be sent to a summary index with a scheduled search.
This data is not logged in this level of detail anywhere but you can save it yourself in a csv/lookup or in a summary index.
What information are you looking for out of the Inspector? That information is actually coming from the search artifact in the dispatch directory, and goes away when the search expires.
However, you also get metrics in the introspection index. (index=introspection sourcetype=splunkresourceusage component=PerProcess) and there are built-in dashboards that might be helpful. (Activity > System Activity, or Settings > Distributed Management Console)
The Splunk on Splunk app also gives insight into search metrics, although there is overlap between it and the built-in dashboards mentioned above.