How do I exclude a certain set of servers in search results based on a particular set of values

[Jarohnimo]

I know my question is gray so allow me to explain.

I have a splunk dashboard that monitors the Current Application Pool state of all our web servers. The basic concept is, if the sites are running they will have a Value of 3, if they are stopped they will have a value of 0. The problem we have is that not all sites exist on all web servers so we need to be able to exclude down application pools on certain servers.

For example:
If I have 5 Web servers, on web server 1 and 2 I have website http://test.com but test.com doesn't exist on web server 3 4 and 5. So on server's 3 4 and 5 we have stopped the application pools for site test.com so now the dashboard shows 3 down application pools where as we don't care! We would like to ignore those down app pools on 3 4 5 ... I thought this could be done with a subsearch or perhaps search NOT but I'm very new to search and have a hard time with the syntax.

Could anyone provide an example of how I can accomplish the exclusion? thank you

[DalJeanis · Accepted solution]

Something like this -

   index=web counter="Current Application Pool State" instance=* 
    | eval host_app=host."__".instance 
    | dedup host_app 
    | eval dropme=if(like(host_app,"^.*test[^_]+__[345]$"),1,0)
    | search dropme = 0
    | chart count by host 
    | sort 0 -Value

The intention of the regular expression in the like clause is to take any host that has the word test in it somewhere, consume all characters before first underscore, then the two underscores, then check for instances 3 thru 5 (assuming that it is a single digit for the instance number). Modify the regex as necessary.

---

edited to use sort 0 instead of sort in case there were more than 100 values to be sorted.

[Jarohnimo]

Your awesome!

[DalJeanis]

Thanks! Glad it helped!

[Jarohnimo]

index=web counter=Current Application Pool State" instance=* | eval host_app=host . "__" . instance | dedup host_app| chart count by host |sort -Value

^ This isn't the full query but I just drew up a little bit so you'd get the idea. This is returning all application pools on all servers. I would like to Ignore the application pool state of the site Test on server 3 4 and 5. How do you exclude those servers from showing results (returning results) as we are perfectly fine with those 3 app pools b

[somesoni2]

Assuming there is field called Value which will be 3 if the site is up and 0 if site is down, then I believe you just exclude all down sites events by including the filter based on Value field. like this

index=web counter=Current Application Pool State" instance=* Value=3 |...rest of the query..

[somesoni2]

Can you post any sample data or search that you're using right now in your dashboard?

[DalJeanis]

There are a ton of ways to go about this. The trivial way would be to have a lookup table of what apps are supposed to run on what web server, add that lookup to the search as an inner join, and any records that aren't supposed to run on that server will magically disappear.

If that method doesn't solve your issue, then please post your current search query, so we can review your architectural assumptions and come up with a custom solution.

[Jarohnimo]

I'll look into the Look up table option,

The full syntax is long but here's the basic

index=web counter=Current Application Pool State" instance=* | eval host_app=host . "__" . instance | dedup host_app| chart count by host |sort -Value

^ No real logic added, it has no way of knowing which app pools are supposed to be on which servers... I thought we could add the logic to the search query by telling it.. "Don't show any results for application pools state for site test on boxes 3 4 and 5.

[DalJeanis]

I assume Value is count?
what field is "test.com" going to appear in?
How do you determine server1 versus server3 or whatever?

[DalJeanis]

Never mind, think i worked it out. check the new answer posted.