We just got Splunk Enterprise up and running, and I'd like some tips on how to tell if it's healthy. Can you get me started, and point me to some resources?
The Splunk Enterprise Monitoring Console is an app included with every Splunk installation. It consists of dashboards, platform alerts, and health checks. It enables Splunk administrators to gain insight into the system health of Splunk Enterprise, including indexing and search performance, OS resource usage, and license usage. But it's not just a stethoscope on system health, the information in the monitoring console provides insight about how your searches are working, and where you can tune them to make them better!
The monitoring console goes beyond just showing if your indexer or search heads are up or down. The monitoring console has a series of dashboards that help you find answers to common problems, for example, why users are getting "peer unresponsive" errors, or why search performance is slow. These diagnostics can also indicate where you may have inefficient searches set up, or if you have too many automated reports running that are affecting system performance.
Metrics in the Monitoring Console can also help you know when to scale. If you notice your system performance consistently running at near-peak levels even after optimizing searches, it may be time to add an indexer.
There are a plethora of community created apps that take monitoring of Splunk to the next level. Take a peek at the comments of this post to learn more.
The Splunk Enterprise Monitoring Console is an app included with every Splunk installation. It consists of dashboards, platform alerts, and health checks. It enables Splunk administrators to gain insight into the system health of Splunk Enterprise, including indexing and search performance, OS resource usage, and license usage. But it's not just a stethoscope on system health, the information in the monitoring console provides insight about how your searches are working, and where you can tune them to make them better!
The monitoring console goes beyond just showing if your indexer or search heads are up or down. The monitoring console has a series of dashboards that help you find answers to common problems, for example, why users are getting "peer unresponsive" errors, or why search performance is slow. These diagnostics can also indicate where you may have inefficient searches set up, or if you have too many automated reports running that are affecting system performance.
Metrics in the Monitoring Console can also help you know when to scale. If you notice your system performance consistently running at near-peak levels even after optimizing searches, it may be time to add an indexer.
There are a plethora of community created apps that take monitoring of Splunk to the next level. Take a peek at the comments of this post to learn more.
Added related video.
I just updated the post to include proper links to the Monitoring Console and a pointer to the discussion here about other community contributed apps the y'all recommend. Karma coming your way @jacobevans
Do not miss these apps:
Alerts for Admins: https://splunkbase.splunk.com/app/3796/
Meta Woot!: https://splunkbase.splunk.com/app/2949/
Data Curator: https://splunkbase.splunk.com/app/1848/
Howdy @jmulcaster_splunk,
This is what you're looking for: https://docs.splunk.com/Documentation/Splunk/latest/DMC/Monitoringoverview
This is also a really nice app that works as an addition to the built-in monitoring console: https://splunkbase.splunk.com/app/3796/ (Click the "details" tab for configuration instructions).
Note that neither of these will work out-of-the-box in a distributed environment. You need to go through all of the configuration items to have either working properly.
The github link for Alerts for Splunk Admins is here, as per the README:
"The overall idea behind this application is to provide a variety of alerts that detect issues or potential issues within the splunk log files and then advise via an alert that this has occurred This application was built as there were a variety of messages in the Splunk console and logs in Splunk that if acted upon could have prevented an issue within the environment.
There are also a few dashboards for investigating indexer performance, heavy forwarder queue usage and data model acceleration issues"
The app has expanded over the years and I would like to continue to add more to it, contributions are always welcome!
One of the apps original goals was to have most of the functionality appear in the monitoring console, over time a small number of the alerts have been replaced by monitoring console functionality!
Cheers @gjanders, I really like the app. Keep up the good work! I have it somewhere on my todo list to go through the full app and add in my own things some day.
No problem, if you find something that would benefit many users feel free to contribute