Getting Data In

automating the missing forwarder records query

umiotoko
New Member

Newbie to splunk, hello everyone...

I use the UniversalForwarder on a pool of windows IIS servers. Each server has 4 unique sites (IP's). I've had some problems with the universal forwarder ignoring some IIS logfiles, eventually this was fixed by adding both "crcSalt = " and "alwaysOpenFile = 1" to the inputs.conf on the IIS servers.

In order to validate that this fix is working, I want to automate a check.

The query "#Fields: | stats values(source) by host" for the last 7 days should return results like:

c:\inetpub\logs\site1\ex20120924.log
c:\inetpub\logs\site1\ex20120925.log
c:\inetpub\logs\site1\ex20120926.log
c:\inetpub\logs\site1\ex20120927.log
c:\inetpub\logs\site1\ex20120928.log
c:\inetpub\logs\site1\ex20120929.log
c:\inetpub\logs\site1\ex20120930.log
c:\inetpub\logs\site2\ex20120924.log
c:\inetpub\logs\site2\ex20120925.log
c:\inetpub\logs\site2\ex20120926.log
c:\inetpub\logs\site2\ex20120927.log
c:\inetpub\logs\site2\ex20120928.log
c:\inetpub\logs\site2\ex20120929.log
c:\inetpub\logs\site2\ex20120930.log

Above are good results.

Below are bad results:

c:\inetpub\logs\site1\ex20120924.log
c:\inetpub\logs\site1\ex20120925.log
<no entry for ex20120926.log>
c:\inetpub\logs\site1\ex20120927.log
c:\inetpub\logs\site1\ex20120928.log
c:\inetpub\logs\site1\ex20120929.log
c:\inetpub\logs\site1\ex20120930.log
c:\inetpub\logs\site2\ex20120924.log
c:\inetpub\logs\site2\ex20120925.log
c:\inetpub\logs\site2\ex20120926.log
c:\inetpub\logs\site2\ex20120927.log
c:\inetpub\logs\site2\ex20120928.log
c:\inetpub\logs\site2\ex20120929.log
<no entry for ex20120930.log>

Above has 2 entries missing.

In pseudo logic, I would run the query:
"#Fields: | stats values(source) by host" for the last 7 days

through a loop counter that returns an error if each site (site1/site2) fails to return 7 records.
If this is too complex, I suppose I could look for "at least XX rows" or such. Any suggestions would be appreciated.

Tags (2)
0 Karma

Ayn
Legend

Use the stats function dc to get a distinct count of the field values, then check if the count is 7. Then create an alert that triggers if 0 results are returned. You could also reverse that logic - check if count is NOT 7 and trigger alert if more than 0 results are returned.

#Fields: | stats dc(source) as uniquecount by host | where uniquecount!=7
Get Updates on the Splunk Community!

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...

Dynamic Links from Alerts to IM Navigators - New in Observability Cloud

Splunk continues to improve the troubleshooting experience in Observability Cloud with this latest enhancement ...