Can _internal and/or _audit solve my problem? Or do I need to use the Monitoring Console?

[swangertyler]

I was tasked with getting some "metrics" for our Splunk instance, as well as creating a dashboard with some "customer-facing metrics". I would prefer to try and not use the Monitoring Console, as using that will introduce new complications/problems to solve given the infrastructure (We currently have it set up, but not behind out Identity Management solution. I would need to jump through a bunch of hoops to get it behind there and don't want to if I don't have to).

My question, can I use the indexes _internal or _audit to get me stuff like:

• Average query response time
• "Splunk availability".
• The rate at which we are indexing

I place availability in quotes because I assume the desired information is, whether or not splunkd was running AND the search head cluster was up and available on the network and I frankly have no idea what is or is not in _internal or _audit. I could not find anything in the Docs that goes over what any of the fields in the events are.

Any help is greatly appreciated.

[amitm05 · Accepted solution]

@swangertyler

Query Response Time - Internal Index
Splunk Availability - Internal Index OR Splunk Rest API (| rest /services/server/info)
Indexing rate - Internal Index (component metrics) OR Splunk Rest API

These 3 that you mentioned can definitely be captured from internal index OR Rest API Commands.
Additionally Running of Splunkd and SH cluster availability are also available through REST API commands.

If there is anything more specifically you want to know off, you can mention. But yes it wouldnt be wrong to say that you'll be able to get your Splunk platform monitoring covered quite well with _internal, _introspection, _audit and Rest APIs

[niketn]

@swangertyler even if you run Monitoring Console on your local machine using combination of Splunk's _internal indexex like _internal, _introspection and _audit index and also Splunk's REST API calls you should be able to build something of your own. However, you should first define your use case and see whether you need all of Monitoring Console or partial or something beyond Monitoring console.

[alonsocaio]

You should also take a look at splunk rest api. Maybe It will be useful for you to get some information about your environment.