I have a list of hosts that submit logs periodically. I need Splunk to generate an alert if the last time it received a log from a host on this list is older than a configurable value per host.
The list of hosts was created Excel, saved as a CSV, uploaded successfully into the Lookup Editor and is called criticalhosts.csv. Below is the contents (I've used example data)
IP Hostname FQDN MaxTimeoutInSeconds Function Description
10.0.0.1 host1 host1.corpdomain.com 900 webserver Old Webserver
10.0.0.2 host2 3600 webserver New Webserver
10.0.0.3 host3.corpdomain.com 86400 mailserver Mailserver
10.0.0.4 300 appserver Appserver
The data represents real hosts. Within my domain, DNS is not necessarily well-maintained and does not always have entries for existing hosts. As such, the "host" value in the logs is sometimes IP, sometimes hostname or sometimes FQDN. While I agree this should be standardized, that's not the world I currently inhabit.
The MaxTimeoutInSeconds value is the maximum amount of time in seconds allowable for the lastTime entry for a particular host.
So, if I wanted to keep it simple and use a fixed time rather than the MaxTimeoutInSeconds of the lookup table, say at least 15 minutes old(900 seconds) but younger than 30 days old (2592000 seconds) I'd use the following.
| metadata type=hosts
| eval lastEventAgeInSeconds = (now() - lastTime)
| search lastEventAgeInSeconds > 900 lastEventAgeInSeconds < 2592000
So far, so good. Now, to use that data and find all log entries matching an IP in my lookup table and display them in a human format I'd use the following.
| metadata type=hosts
| eval lastEventAgeInSeconds = (now() - lastTime)
| search lastEventAgeInSeconds > 900 lastEventAgeInSeconds < 2592000
| join [|inputlookup criticalhosts.csv | eval host=IP]
| convert ctime(lastTime)
| table host Hostname FQND lastEventAgeInSeconds lastTime Function Description
That gives me a very usable subset of what I want to know. Now, I could create another search to find all log entries matching a hostname in my lookup table my changing the inputlookup statement like so.
| metadata type=hosts
| eval lastEventAgeInSeconds = (now() - lastTime)
| search lastEventAgeInSeconds > 900 lastEventAgeInSeconds < 2592000
| join [|inputlookup criticalhosts.csv | eval host=Hostname]
| convert ctime(lastTime)
| table host Hostname FQND lastEventAgeInSeconds lastTime Function Description
But that requires a separate search for matching IP, hostname or FQDN. And that's not mentioning the use of the MaxTimeoutInSeconds value in the lookup table.
==== HERE'S THE QUESTION ===
How do I have a single search that matches against IP, hostname or FQDN in addition to using the MaxTimeoutInSeconds values in my lookup table instead of fixed timeouts?
... View more