automating the missing forwarder records query

umiotoko — Mon, 01 Oct 2012 21:39:12 GMT

Newbie to splunk, hello everyone...

I use the UniversalForwarder on a pool of windows IIS servers. Each server has 4 unique sites (IP's). I've had some problems with the universal forwarder ignoring some IIS logfiles, eventually this was fixed by adding both "crcSalt = " and "alwaysOpenFile = 1" to the inputs.conf on the IIS servers.

In order to validate that this fix is working, I want to automate a check.

The query "#Fields: | stats values(source) by host" for the last 7 days should return results like:

c:\inetpub\logs\site1\ex20120924.log
c:\inetpub\logs\site1\ex20120925.log
c:\inetpub\logs\site1\ex20120926.log
c:\inetpub\logs\site1\ex20120927.log
c:\inetpub\logs\site1\ex20120928.log
c:\inetpub\logs\site1\ex20120929.log
c:\inetpub\logs\site1\ex20120930.log
c:\inetpub\logs\site2\ex20120924.log
c:\inetpub\logs\site2\ex20120925.log
c:\inetpub\logs\site2\ex20120926.log
c:\inetpub\logs\site2\ex20120927.log
c:\inetpub\logs\site2\ex20120928.log
c:\inetpub\logs\site2\ex20120929.log
c:\inetpub\logs\site2\ex20120930.log

Above are good results.

Below are bad results:

c:\inetpub\logs\site1\ex20120924.log
c:\inetpub\logs\site1\ex20120925.log
<no entry for ex20120926.log>
c:\inetpub\logs\site1\ex20120927.log
c:\inetpub\logs\site1\ex20120928.log
c:\inetpub\logs\site1\ex20120929.log
c:\inetpub\logs\site1\ex20120930.log
c:\inetpub\logs\site2\ex20120924.log
c:\inetpub\logs\site2\ex20120925.log
c:\inetpub\logs\site2\ex20120926.log
c:\inetpub\logs\site2\ex20120927.log
c:\inetpub\logs\site2\ex20120928.log
c:\inetpub\logs\site2\ex20120929.log
<no entry for ex20120930.log>

Above has 2 entries missing.

In pseudo logic, I would run the query:
"#Fields: | stats values(source) by host" for the last 7 days

through a loop counter that returns an error if each site (site1/site2) fails to return 7 records.
If this is too complex, I suppose I could look for "at least XX rows" or such. Any suggestions would be appreciated.

Re: automating the missing forwarder records query

Ayn — Mon, 01 Oct 2012 21:59:24 GMT

Use the stats function dc to get a distinct count of the field values, then check if the count is 7. Then create an alert that triggers if 0 results are returned. You could also reverse that logic - check if count is NOT 7 and trigger alert if more than 0 results are returned.

#Fields: | stats dc(source) as uniquecount by host | where uniquecount!=7

topic automating the missing forwarder records query in Getting Data In

automating the missing forwarder records query

Re: automating the missing forwarder records query