topic Re: Locating missing events in Splunk Search

Locating missing events

justinfranks — Wed, 25 Sep 2013 23:25:05 GMT

Hey all,

I've searched for an answer to this but cannot see one, so apologies if this has been answered before.

Some background: We have a number of FileMaker DBs and I am attempting setup some automation so confirm that the DBs are all open. To do this, a FileMaker Server-Side script runs every 15 mins and confirms that they are all open and output it's results to a table.

I am using the JDBC connector in the Splunk DB Connect app to index this table. This all working fine and I am able to search. However, if the FileMaker database is unreachable the events are not indexed and this is expected behaviour but I was wondering if there was a way to detect a lack of events for a specific time period to be able to report on them?

The events are VERY simple, a timestamp and one KVP. Example: "26/09/2013 09:15:00 AM" FilesOpen=13

Would "| overlap | collect" work in this instance? If so, is there any decent documentation on this method? The built-in documentation is limited at best.

Thanks in advance,

Justin.

Re: Locating missing events

Lucas_K — Thu, 26 Sep 2013 01:35:10 GMT

You could just check your _internal index for the indexing rate for events going to that particular index.

If the rate drops below a threshold then send an alert.

index="_internal" sourcetype="splunkd" group="per_index_thruput" NOT (series="_*" OR series="/*" OR series="summary*") | rename series as index  | stats avg(kbps) by index

Re: Locating missing events

justinfranks — Thu, 26 Sep 2013 02:37:43 GMT

Thanks for the response!

This would be good if I wanted to just alert. However, I would like to report on the missing events. ie count them.

Re: Locating missing events

gkanapathy — Thu, 26 Sep 2013 04:27:39 GMT

... | timechart span=15m count

And alert if the value is less than the expected number (or it's zero).

Re: Locating missing events

gkanapathy — Thu, 26 Sep 2013 04:29:59 GMT

Or ... | timechart span=15m sum(FilesOpen)