topic Re: How to find when splunk is not indexing? in Alerting

How to find when splunk is not indexing?

irakeshraut — Wed, 11 Feb 2015 06:34:00 GMT

Is there any way to find out if splunk stopped indexing? I would like to send a notification when splunk stops indexing.

For notification i will use scoutapp or i may just send an email.

OS --> production ubuntu / Development Mac
We are using distributed system and get log from multiple servers.

Re: How to find when splunk is not indexing?

lmyrefelt — Wed, 11 Feb 2015 19:17:06 GMT

There are plenty. Can you provide us with more information ? How is your architecture ? OS ?
Do you want to use Splunk to monitor it self ?

Have you read the docs ?

http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Whatsinhere

There is also plenty of good apps now-a-days for your day-to-day monitoring and troubleshooting of splunk at apps.splunk.com

Re: How to find when splunk is not indexing?

irakeshraut — Wed, 11 Feb 2015 23:25:54 GMT

I will use scoutapp to monitor splunk. But only difficulty is to find when splunk is not indexing so i can send the alert.

I was thinking about doing search using splunk api and search for index=* for last 5 minute( I am not sure if its possible). If there is no result from our search then send the alert. I am pretty sure there must be smarter way of doing this.

Can i use eventcount for this ? or any other better way?

Re: How to find when splunk is not indexing?

ChrisG — Wed, 11 Feb 2015 23:36:53 GMT

How is this different from your previous posting? http://answers.splunk.com/answers/214857/how-to-find-out-if-splunk-is-indexing-data.html

Re: How to find when splunk is not indexing?

irakeshraut — Thu, 12 Feb 2015 00:50:00 GMT

Ya i have posted 3 question so far. All related to same think because I haven't got working solution so far.

Re: How to find when splunk is not indexing?

irakeshraut — Thu, 12 Feb 2015 00:51:16 GMT

How about answering to actual question rather than digging what I am asking?

Re: How to find when splunk is not indexing?

ppablo — Thu, 12 Feb 2015 01:53:26 GMT

Hi @irakeshraut

@ChrisG is just trying to help with decluttering this forum. If you are going to post questions with duplicate content, please delete your previous posts so there aren't 3 of the same topic here on Splunk Answers. Each time you've posted a brand new (same) question, you haven't been providing enough details for users to work with and they have to keep asking you for extra information. When someone has attempted to answered your questions, that's when you've been providing extra information on how you're trying to achieve your goal, but at that point, they’re probably on to their own priorities.

This is a free space to find answers, so it's not like users are going to be sitting watching a post the entire day and can't wait for a back and forth thread to get all the details they need to give you a full and complete answer. If @ChrisG and I were REST API experts, then we’d be more than happy and willing to answer your questions, but unfortunately we’re not. What we’re trying to do is optimize your posts so you can find the right experts here on Splunk Answers to help you find your solutions. We’re all trying to help each other out here in this space, all we ask is that users be respectful to each other.

Re: How to find when splunk is not indexing?

ppablo — Thu, 12 Feb 2015 01:53:45 GMT

Pulling information from your other 2 posts and this one, you should consider rephrasing your question and content (if you choose to post a new question, I would post it in 15 hours when traffic on the site is high AND delete all your previous posts with the same content):

Question:
How to search the count of indexed events in the last 5 minutes via REST API and send an alert if the count is 0?

Information you should provide in your content:
-What version of Splunk are you running?
-On what operating system?
-What is your architecture? (single instance, distributed search, etc)
-Be as detailed as possible with what you’re trying to achieve, your desired outcome, and HOW you want to get there. (ex: you haven’t been putting in your content that you’re trying to do this via the REST API, so users aren’t going to just assume that by default)

Other information to consider:
Are you trying to find out if Splunk has stopped indexing, or if there just aren't any events indexed in the last 5 minutes? These are two different things, but you’ve been asking both interchangeably. Checking if Splunk has stopped indexing implies something has broken. How frequently are you expecting data to be indexed? Just because there were no events in the last 5 minutes doesn’t mean Splunk has stopped indexing, so you could be getting false positives.

Re: How to find when splunk is not indexing?

irakeshraut — Thu, 12 Feb 2015 02:02:48 GMT

Thanks makes sense 🙂

Re: How to find when splunk is not indexing?

irakeshraut — Thu, 12 Feb 2015 02:05:53 GMT

Thanks for your reply. I am actually trying to find if splunk has stopped indexing.

Re: How to find when splunk is not indexing?

ppablo — Thu, 12 Feb 2015 02:30:42 GMT

Cool. I'd recommend looking at the troubleshooting documentation that @lmyrefelt provided in their answer above. Splunk logs a lot of useful data on itself to help you pinpoint if there are any problems in your environment (ex: if all indexing has stopped entirely). There's a list of the different internal logs in that documentation and a description of what each one contains. So if something has stopped working out of the blue, you would usually look through or run searches against these internal logs for errors on the instances where the problem is happening.

You can very well still use the approach of searching the count of events in your environment via the REST API and triggering an alert if you are getting zero results for a certain amount of time. Since you just started using Splunk yesterday and might not be familiar with how frequently you'll be indexing events, it might be hard to assume what window of time you should be alerting on. 5 minutes might be too strict of a number. Again, I'm not an expert on this so I can't give you a concrete answer, but hopefully this will be a good starting point with things to think about moving forward.

Re: How to find when splunk is not indexing?

lmyrefelt — Thu, 12 Feb 2015 08:50:52 GMT

Well, Splunk can have "stopped" indexing based on a "Number" of factors ... including a lot of them that might have nothing to do with splunk itself ... (i.e.. network, application, etc, etc) You are probably better of monitoring the Splunkd process and or its open ports 8089 ? At least in addition to a search to be able to determine if it really is Splunk that stopped indexing data or if it is Something blocking data from reaching splunk.

And your scout-app seems to be able to do it all.

Re: How to find when splunk is not indexing?

irakeshraut — Thu, 12 Feb 2015 09:55:37 GMT

Thanks for your reply. We are already monitoring splunkd using scoutapp.

Re: How to find when splunk is not indexing?

ChrisG — Thu, 12 Feb 2015 16:56:38 GMT

You can also use the distributed management console. It has an indexing performance dashboard and related platform alerts. The docs for it are in the Admin Manual, starting with the topic Configure the distributed management console.