topic How to check if Splunk is getting data from the server? in Alerting

How to check if Splunk is getting data from the server?

bsaujla131984 — Sat, 02 Nov 2019 04:01:32 GMT

I have created various alerts, however, sometimes alerts get generated even though there is no issue at all.

Upon investigation, we have found out this happens when Splunk is not getting from servers.

Is there a way we can put a condition in alert to verify if data is coming from the server?

Re: How to check if Splunk is getting data from the server?

to4kawa — Sat, 02 Nov 2019 09:03:08 GMT

| metadata type=hosts where index=your_index
| foreach *Time 
    [eval <<FIELD>>_c = strftime(<<FIELD>>,"%c")]
    | where recentTime =< relative_time(now(),"-10m@m") AND host="your_host"

Hi,

If the host specified in this query has not come 10 minutes ago, the log will be lost.

Try changing the relative_time argument.
I think you should alert when the result over 1.

When sending the host name by alert mail
Specify the fields to be displayed using the fields command.( fields host )

Re: How to check if Splunk is getting data from the server?

gcusello — Sat, 02 Nov 2019 10:33:47 GMT

Hi bsaujla131984,
you should create a lookup with all the server to monitor (called e.g. perimeter.csv with a column called host) and schedule an alert (e.g. every five minutes) like this:

| metasearch index=_internal
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval count=0, host=lower(host) | fields host count ]
| stats sum(count) AS total BY host.
| where total=0

In this way if you have results, it means that one or more servers of your perimeter didn't send internal Splunk logs in the last period (e.g. 5 minutes).

If you like, deleting last row, you can display the status of your servers:

total=0 => missing
total>0 => running

You can also display this dashboard in graphic mode.

Ciao.
Giuseppe