How to check if Splunk is getting data from the server?

[bsaujla131984]

I have created various alerts, however, sometimes alerts get generated even though there is no issue at all.

Upon investigation, we have found out this happens when Splunk is not getting from servers.

Is there a way we can put a condition in alert to verify if data is coming from the server?

[gcusello]

Hi bsaujla131984,
you should create a lookup with all the server to monitor (called e.g. perimeter.csv with a column called host) and schedule an alert (e.g. every five minutes) like this:

| metasearch index=_internal
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval count=0, host=lower(host) | fields host count ]
| stats sum(count) AS total BY host.
| where total=0

In this way if you have results, it means that one or more servers of your perimeter didn't send internal Splunk logs in the last period (e.g. 5 minutes).

If you like, deleting last row, you can display the status of your servers:

• total=0 => missing
• total>0 => running

You can also display this dashboard in graphic mode.

Ciao.
Giuseppe

[to4kawa]

| metadata type=hosts where index=your_index
| foreach *Time 
    [eval <<FIELD>>_c = strftime(<<FIELD>>,"%c")]
    | where recentTime =< relative_time(now(),"-10m@m") AND host="your_host"

Hi,

If the host specified in this query has not come 10 minutes ago, the log will be lost.

Try changing the relative_time argument.
I think you should alert when the result over 1.

When sending the host name by alert mail
Specify the fields to be displayed using the fields command.( fields host )