Monitoring Splunk

Can _internal and/or _audit solve my problem? Or do I need to use the Monitoring Console?

swangertyler
Path Finder

I was tasked with getting some "metrics" for our Splunk instance, as well as creating a dashboard with some "customer-facing metrics". I would prefer to try and not use the Monitoring Console, as using that will introduce new complications/problems to solve given the infrastructure (We currently have it set up, but not behind out Identity Management solution. I would need to jump through a bunch of hoops to get it behind there and don't want to if I don't have to).

My question, can I use the indexes _internal or _audit to get me stuff like:

  • Average query response time
  • "Splunk availability".
  • The rate at which we are indexing

I place availability in quotes because I assume the desired information is, whether or not splunkd was running AND the search head cluster was up and available on the network and I frankly have no idea what is or is not in _internal or _audit. I could not find anything in the Docs that goes over what any of the fields in the events are.

Any help is greatly appreciated.

Tags (1)
0 Karma
1 Solution

amitm05
Builder

@swangertyler

Query Response Time - Internal Index
Splunk Availability - Internal Index OR Splunk Rest API (| rest /services/server/info)
Indexing rate - Internal Index (component metrics) OR Splunk Rest API

These 3 that you mentioned can definitely be captured from internal index OR Rest API Commands.
Additionally Running of Splunkd and SH cluster availability are also available through REST API commands.

If there is anything more specifically you want to know off, you can mention. But yes it wouldnt be wrong to say that you'll be able to get your Splunk platform monitoring covered quite well with _internal, _introspection, _audit and Rest APIs

View solution in original post

amitm05
Builder

@swangertyler

Query Response Time - Internal Index
Splunk Availability - Internal Index OR Splunk Rest API (| rest /services/server/info)
Indexing rate - Internal Index (component metrics) OR Splunk Rest API

These 3 that you mentioned can definitely be captured from internal index OR Rest API Commands.
Additionally Running of Splunkd and SH cluster availability are also available through REST API commands.

If there is anything more specifically you want to know off, you can mention. But yes it wouldnt be wrong to say that you'll be able to get your Splunk platform monitoring covered quite well with _internal, _introspection, _audit and Rest APIs

niketn
Legend

@swangertyler even if you run Monitoring Console on your local machine using combination of Splunk's _internal indexex like _internal, _introspection and _audit index and also Splunk's REST API calls you should be able to build something of your own. However, you should first define your use case and see whether you need all of Monitoring Console or partial or something beyond Monitoring console.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

alonsocaio
Contributor

You should also take a look at splunk rest api. Maybe It will be useful for you to get some information about your environment.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...