Monitoring Splunk

How to find problematic searches (scheduled, real-time, etc) that are affecting performance in our shared Splunk environment?

Ricapar
Communicator

As our shared Splunk environment matures, we're trying to build in some checks to make sure everyone is being a good citizen and not running searches that can create large impacts to others.

On my checklist are the following:

  • Realtime saved searches
  • Saved searches with short schedules (every minute)
  • Saved searches over very large time ranges
  • Saved searches searches that take a very long time to execute
  • (anything else that should be here?)

I've been able to identify most of these by doing a recursive grep through the etc directory on the search head, looking for specific entries in savedsearches.conf. However, the process is somewhat clunky and I have a feeling there data is already somewhere in Splunk, I just don't know about it.

I don't want to start imposing restrictions on the roles level (yet), but at the very least I'd like to be able to set up an alert to myself and the other Splunk admins notifying us of when a user saves or schedules a possibly problematic search.

jeremiahc4
Builder

I have been using the Splunk On Splunk app for finding problematic searches, etc.. and it's been working great.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...