Monitoring Splunk

Please share a short Splunk preventative tasks list a Splunk Admin. would do Daily / weekly to defend the turf. Thank u

SamHTexas
Builder

Please share a Splunk preventative tasks list a Splunk Admin. would do Daily / weekly to defend the turf. Thank u in advance. Please share SPLs if you would.

Labels (1)
Tags (1)
0 Karma

SamHTexas
Builder

Thank u very much. If you think of similar measures for defensive purposes please share. Happy Memorial day 2021.

0 Karma

tscroggins
Influencer

@SamHTexas 

EDIT: This list has more to do with platform stability than "defending the turf," but it's much easier to identify problems in an otherwise healthy environment than a sick one.

I generally do the following:

1. Configure the monitoring console and enable alerts. If you're using forwarders, configure forwarder monitoring. This should cover basic availability monitoring.

2. Create a report or dashboard quantifying _internal (or app specific) ERROR and WARN* events by source, component, or whichever category works best for you conceptually. Manage these as defects using quality control tools, e.g. Pareto charts.

3. Identify hosts and sources present today that were not present yesterday, i.e. new sources.

4. Identify hosts and sources present yesterday that are not present today, i.e. missing sources.

5. Identify anomalous changes in event counts across critical hosts and sources.

6. Work with your infrastructure or capacity team (if they're separate functions) to baseline Splunk performance and identify anomalous variances in principal components: CPU, memory, I/O, and storage.

Beyond the basics, you're getting into service quality and quantifying/qualifying user behavior: search performance, search coverage, data retention relative to storage pools, etc.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...