Set up an alert whenever a host stops sending logs to Splunk

[Roy_9]

Hello,

I have a lookup where all the hostnames are available under the field called "title" with respect to teams.I would like to set up an alert for team "abc" if any of the host stops reporting for more than 15 mins, I tried the below search but unable to get the results.Can anyone please help me with the search, it is highly helpful.

Search which i am using:

| inputlookup 123.csv
| search team="abc"
| table title
| rename title as host
| appendpipe [ | stats count as islookupcount ]
| eval current_time =now()
| eval islookupcount = coalesce(islookupcount, 0)
| search islookupcount = 0
| eventstats latest(_time) as last_event_time by host
| where current_time - last_event_time > 900
| eval stopped_sending_time=strftime(current_time,"%Y-%m-%d %H:%M:%S")
| table unit_id, host, stopped_sending_time

please help me with the better search for my usecase may be i am not using the right one.

Thanks

[Roy_9]

@gcusello i found the similar solution of yours but i am unable to achieve for my usecase. Can you please help me in tweaking search for mine.

https://community.splunk.com/t5/Alerting/How-do-you-detect-when-a-host-stops-sending-logs-to-Splunk/...

[gcusello]

Hi @Roy_9 ,

could you better descibe your requirement?

I understood that you want to trigger an alert if an host didn't send events in tha last 15 minutes, is this correct?

I don't understand why you take events from a lookup, you should take events from an index.

anyway, see my approach and adapt it to your requirement:

| tstats count latest(_time) AS latest WHERE index=_internal BY host
| where now()-latest>900
| eval latest=strftime(latest,"%Y-%m-%d %H:%M:%S")
| table host latest

in this way you take all the hosts that didn't send events in the last 15 minutes but sent events in the time period.

If an host never sent events in the time range you don't detect it.

to complete your Use case, you should create a lookup containing all the hosts to monitor (called e.g. perimeter.csv) containing at least one field (host) and run something like this:

| tstats count latest(_time) AS latest WHERE index=_internal BY host
| append [ | inputlookup perimeter.csv | count=0 | fields host count ]
| stats sum(count) AS total value(latest) AS latest BY host
| where now()-latest>900 OR total=0
| eval 
   latest=strftime(latest,"%Y-%m-%d %H:%M:%S"), 
   status=if(total=0,"Never sent","Last event: ".latest)
| table host status

Ciao.

Giuseppe

[gcusello]

Hi @Roy_9,

you have to rename the title field in host and add the Team field to display it, something like this:

| tstats count latest(_time) AS latest WHERE index=_internal BY host
| append [ | inputlookup 123.csv | rename title AS host | count=0 | fields host Team count ]
| stats 
   sum(count) AS total 
   value(latest) AS latest 
   values(Team) AS Team 
   BY host
| where now()-latest>900 OR total=0
| eval 
   latest=strftime(latest,"%Y-%m-%d %H:%M:%S"), 
   status=if(total=0,"Never sent","Last event: ".latest)
| table host Team status

Ciao.

Giuseppe

[Roy_9]

@gcusello  I am getting the error Unknown search command 'count' and also if i want to obtain the values for only particular team, let' s say "Network team", where can i add this?

[gcusello]

Hi @Roy_9 ,

count is at the end of the previous row, not in a new row.

About Team, what's ne field name that you have in the lookup?

add it instead Team in the search.

Ciao.

Giuseppe

[Roy_9]

@gcusello field name is service team and value is Network. I didn't get this part.

count is at the end of the previous row, not in a new row.

Can you please send me the updated search, that would be highly helpful.

Thanks in advance.

[gcusello · Accepted solution]

Hi @Roy_9,

if the field name in the lookup is "service team", try this:

| tstats count latest(_time) AS latest WHERE index=_internal BY host
| append [ 
   | inputlookup 123.csv 
   | rename title AS host 
   | eval count=0 
   | fields host "service team" count 
   ]
| stats 
   sum(count) AS total 
   value(latest) AS latest 
   values("service team") AS "service team"
   BY host
| where now()-latest>900 OR total=0
| eval 
   latest=strftime(latest,"%Y-%m-%d %H:%M:%S"), 
   status=if(total=0,"Never sent","Last event: ".latest)
| table host "service team" status

If the field name in the lookup is different, correct my search.

Ciao.

Giuseppe

[gcusello]

Hi @Roy_9 ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

[Roy_9]

Thanks @gcusello, yes i already have a lookup in the name of 123.csv where the host's are listed under column called "title" and different team names are listed under column Teams.

Can you please advise how do i need to append these two fields Title and Teams to the above search?