The Splunk Product Best Practices team provided this response. Read more about How Crowdsourcing is Shaping the Future of Splunk Best Practices.
The Splunk Cloud monitoring console is an app, like the Search app. It consists of dashboards, platform alerts, and health checks. It enables Splunk administrators to gain insight into Splunk's system health, including indexing and search performance, OS resource usage, and license usage. But it's not just a stethoscope on system health, the information in the monitoring console provides insight about how your searches are working, and where you can tune them to make them better!
How the Cloud Monitoring Console helps promote a healthy Splunk deployment
The monitoring console goes beyond just showing if your indexer or search heads are up or down. The monitoring console has a series of dashboards that help you monitor Splunk Cloud deployment health, for example, why users are getting "peer unresponsive" errors, or why search performance is slow. These diagnostics can also indicate where you may have inefficient searches set up, or if you have too many automated reports running that are affecting system performance.
- Search Usage Statistics: The Search Usage Statistics dashboards help you understand your search performance on details such as Aggregate Search Runtime, Top 10 Memory-Consuming Searches, and Long-Running Searches. Warning: users of these dashboards have been known to favorite the documentation on how and how to Write better searches....don't say we didn't warn you!
- Scheduler activity: The Scheduler Activity dashboards monitor the activity and success rate of the search scheduler. This can help you configure the priority of scheduled reports, run traffic control on your scheduled searches, and ensure they are efficient and make good use of system resources.
- License usage: The License Usage dashboard provides insight about your data ingestion and daily license usage, license warnings, and the last 30 days of your license usage directly from the Splunk Web.
- Platform alerts: A platform alert is a saved search in the monitoring console that notifies administrators of conditions that might compromise their Splunk software environment.
How to get started using the Splunk Cloud monitoring console
- Set up the Splunk Cloud monitoring console. Review your data retention capacity and configure Splunk Cloud to generate an alert when the value exceeds your usage license.
- Check your system health. Locate the Splunk Cloud monitoring console and get familiar with the dashboards and the information they show. From the Overview dashboard, check the CPU usage of your indexer(s). Is it in the green (0-59%), orange (60-79%), or red (80% or more) status range? Are there any triggered alerts? From the Topology view under Indexers, toggle to show the indexing rate per second.
- Set up a regular schedule of health maintenance checks. As a best practice, incorporate the monitoring console dashboards into a regular schedule of health maintenance checks. For example, you can monitor search efficiency on a weekly interval, and monitor overall deployment health every month. You can also configure the priority of the scheduled reports.
- Optimize search performance. Ensure you have healthy searches for optimal performance of your entire Splunk Cloud environment. Check for skipped searches, review searches by user, and review long-running searches. Check for and resolve data quality issues, such as line or event breaking issues.
- Check things out with other Splunk users. Search Splunk Answers for answers, or ask a question of your own. If you're still not sure, contact Splunk support by submitting a case on the Splunk Support and Services portal! Don't forget to generate a diagnostic file to give Support insight into your configuration and performance history.
The Splunk Product Best Practices team provided this response. Read more about How Crowdsourcing is Shaping the Future of Splunk Best Practices.
The Splunk Cloud monitoring console is an app, like the Search app. It consists of dashboards, platform alerts, and health checks. It enables Splunk administrators to gain insight into Splunk's system health, including indexing and search performance, OS resource usage, and license usage. But it's not just a stethoscope on system health, the information in the monitoring console provides insight about how your searches are working, and where you can tune them to make them better!
How the Cloud Monitoring Console helps promote a healthy Splunk deployment
The monitoring console goes beyond just showing if your indexer or search heads are up or down. The monitoring console has a series of dashboards that help you monitor Splunk Cloud deployment health, for example, why users are getting "peer unresponsive" errors, or why search performance is slow. These diagnostics can also indicate where you may have inefficient searches set up, or if you have too many automated reports running that are affecting system performance.
- Search Usage Statistics: The Search Usage Statistics dashboards help you understand your search performance on details such as Aggregate Search Runtime, Top 10 Memory-Consuming Searches, and Long-Running Searches. Warning: users of these dashboards have been known to favorite the documentation on how and how to Write better searches....don't say we didn't warn you!
- Scheduler activity: The Scheduler Activity dashboards monitor the activity and success rate of the search scheduler. This can help you configure the priority of scheduled reports, run traffic control on your scheduled searches, and ensure they are efficient and make good use of system resources.
- License usage: The License Usage dashboard provides insight about your data ingestion and daily license usage, license warnings, and the last 30 days of your license usage directly from the Splunk Web.
- Platform alerts: A platform alert is a saved search in the monitoring console that notifies administrators of conditions that might compromise their Splunk software environment.
How to get started using the Splunk Cloud monitoring console
- Set up the Splunk Cloud monitoring console. Review your data retention capacity and configure Splunk Cloud to generate an alert when the value exceeds your usage license.
- Check your system health. Locate the Splunk Cloud monitoring console and get familiar with the dashboards and the information they show. From the Overview dashboard, check the CPU usage of your indexer(s). Is it in the green (0-59%), orange (60-79%), or red (80% or more) status range? Are there any triggered alerts? From the Topology view under Indexers, toggle to show the indexing rate per second.
- Set up a regular schedule of health maintenance checks. As a best practice, incorporate the monitoring console dashboards into a regular schedule of health maintenance checks. For example, you can monitor search efficiency on a weekly interval, and monitor overall deployment health every month. You can also configure the priority of the scheduled reports.
- Optimize search performance. Ensure you have healthy searches for optimal performance of your entire Splunk Cloud environment. Check for skipped searches, review searches by user, and review long-running searches. Check for and resolve data quality issues, such as line or event breaking issues.
- Check things out with other Splunk users. Search Splunk Answers for answers, or ask a question of your own. If you're still not sure, contact Splunk support by submitting a case on the Splunk Support and Services portal! Don't forget to generate a diagnostic file to give Support insight into your configuration and performance history.
