topic Re: How to display a message when a specific host is not ingesting? in Splunk Search

How to display a message when a specific host is not ingesting?

auzark — Wed, 23 Feb 2022 04:04:05 GMT

My dilemma.

index=prod_s3 sourcetype=My_Sourcetype earliest=-30m
(host=2016) OR (host=2018) OR
(host=2015) OR (host=2017)
|stats count as value by host

The above query will return a count for each host that is ingesting, however If one of the above hosts is not ingesting, I wish to alert on that host, displaying the host name as output with a message.

Any help is appreciate.

Re: How to display a message when a specific host is not ingesting?

SanjayReddy — Wed, 23 Feb 2022 04:59:06 GMT

Hi @auzark

There are couple ways you can acheive it

you can use lookup way as suggested by @gcusello and @adonio in another post , you can refer to following link

https://community.splunk.com/t5/Getting-Data-In/How-to-create-an-alert-for-the-host-not-sending-data-for-an-hour/m-p/407769

Re: How to display a message when a specific host is not ingesting?

gcusello — Wed, 23 Feb 2022 07:20:50 GMT

Hi @auzark,

there are many answers in Community aboput this problem.

In few words, the best approach is to create a lookup (called e.g. perimeter.csv with one column called host) containing all the servers to check, and run a search like this:

If you have only few hosts you could also run something like this:

but I hint to use the first soilution because it's more flexible.

Ciao.

Giuseppe

Re: How to display a message when a specific host is not ingesting?

auzark — Wed, 23 Feb 2022 08:01:06 GMT

Thanks guys, I'll try tomorrow.

Can the .csv be named something other than perimeter? and can I name my column in the lookup something other than host?

Re: How to display a message when a specific host is not ingesting?

gcusello — Wed, 23 Feb 2022 08:06:04 GMT

hi @auzark,

you can use the name you like both for the lookupname and the lookup column.

The only attention, if you use a name different than "host", is that you have to modify a little the search:

Ciao.

Giuseppe

Re: How to display a message when a specific host is not ingesting?

SanjayReddy — Wed, 23 Feb 2022 08:09:13 GMT

Hi @auzark

1.Yes, you can use any custom name that you can refer it , you need to use same in lookup

2.cloumn name in lookup can be anyname but you need to rename them in qurey , name need to march with host name in index

| inputlookup <lookupname>.csv | rename <colnameinlookup> as host | eval host=lower(host)

Re: How to display a message when a specific host is not ingesting?

auzark — Fri, 25 Feb 2022 06:35:48 GMT

Hi Sanjay and Giuseppe,

I really appreciate your help and have set up a loookup.csv with the host values I'm concerned about, however, I still can not get it to work . Below are my results, looking for =0 and >0.
I'm running over a set 3 minute period so that I can capture a count=0.

------NonStop_JumpHosts.csv

hosts
vpw00002015
vpw00002016
vpw00002017
vpw00002018

-----------------Test run #1, where > 0.

25 events (24/02/2022 13:57:00.000 to 24/02/2022 14:00:00.000)

host,total
vpw00002015,24
vpw00002016,1

-----------------Test run #2, where = 0.

25 events (24/02/2022 13:57:00.000 to 24/02/2022 14:00:00.000)

No results found.

-----------------Test run #3, where > 0, over a 15 minute period to show all hosts in lookup.

37 events (24/02/2022 13:45:00.000 to 24/02/2022 14:00:00.000)

host,total
vpw00002015,24
vpw00002016,2
vpw00002017,4
vpw00002018,7

---------------------------------------------------------------------

Re: How to display a message when a specific host is not ingesting?

gcusello — Fri, 25 Feb 2022 06:49:47 GMT

Hi @auzark,

when you created the NonStop_JumpHosts.csv lookup, did you remembered to create also lookup definition?

Anyway, I think that the problem is in the host column of your lookup: in the lookup the field is called "hosts", instead in your search is called "host", so you don't find anything!

So, you have two choices:

change the column name of the lookup or modfy the search in this way:

To better see results, you could also run the search without the last row, so in one search you have both the situations (presence and absense).

Ciao.

Giuseppe

Re: How to display a message when a specific host is not ingesting?

auzark — Fri, 25 Feb 2022 07:04:06 GMT

Thanks, I'll give that a try.

Re: How to display a message when a specific host is not ingesting?

auzark — Fri, 25 Feb 2022 07:30:00 GMT

Hi Giuseppe and Sanjay,

Thanks for your help.... that works..... Doh!!! hosts vs host.

-------------------------------------------------------------------------

I was also trying to use a join, unsuccessfully.

inputlookup NonStop_JumpHosts.csv
| table host

| join type=inner host
[ | search index=prod_s3 sourcetype=WinEventLog:Security (host=vpw00002016) OR (host=vpw00002018) OR (host=vpw00002015) OR (host=vpw00002017)
| stats count as NumberOfEvents by host ]
| where ISNOTNULL(NumberOfEvents)
| eval Title=host." No data received for the last 30 min."
| eval Description=Title." Splunk has found no events occurring for the ".host." server for the last 30 min."

I was wondering could I have done this with a join command as another methodology?

Re: How to display a message when a specific host is not ingesting?

gcusello — Fri, 25 Feb 2022 07:47:50 GMT

Hi @auzark,

I don't like the join solution: I use join only if haven't any other solution and if I have few events, otherwise the search will be very slow!

Anyway, your solution has another problem: it isn't a good approach to use the search in a subsearch because there's the limit of 50,000 results in the subsearch, so you could have a incomplete results from it.

Ciao and happy splunking.

Giuseppe

Re: How to display a message when a specific host is not ingesting?

auzark — Wed, 02 Mar 2022 05:59:16 GMT

Hi gcusello,

Another question, if you do not mind.

I have added the below 2 lines to the query you helped me with. I have been trying to get a field other than the "host" field from my lookup table to output as part of the Title or Description?

| eval Title=host." No data received for the last xx min."
| eval Description=Title." Splunk has found no events occurring for the ".host." server for the last xx min."

Also can you describe what host=lower(host) is doing?

Thanks..

Re: How to display a message when a specific host is not ingesting?

gcusello — Wed, 02 Mar 2022 08:12:23 GMT

Hi @auzark,

host=lower(host) transforms all the hostnames in lowercase to avoid duplicates related to the upper o lower case.

Obviously you can add all the information you like to the search, e.g. you could also add more fields (e.g. IP or Department) to the lookup and display them in the final stats command, in this way:

index=prod_s3 sourcetype=WinEventLog:Security (host=vpw00002016) OR (host=vpw00002018) OR (host=vpw00002015) OR (host=vpw00002017) | eval host=lower(host) | stats count by host | append [ | inputlookup NonStop_JumpHosts.csv | eval host=lower(hosts), count=0 | fields host count IP Department] | stats values(IP) AS IP values(Department) AS Department sum(count) AS total BY host | where total=0

Ciao.

Giuseppe