Control Alerting Based on Host Groups

[rlaan]

I want to be able to create searches that will only look at hosts from different levels of our SDLC environment so for example a DEVELOPMENT group, TESTING and PRODUCTION. Also including a group to managed decommissioned servers until they return to use.

DEV = "host d, d1, d2, d3... dn"
TEST = "host t, t1, t2, t3... tn"
PRD = "host p, p1, p2, p3... pn"
NO_ALERT = "host d1, t1, p1"

I would like to be able to create searches and reports using "search DEV NOT NO_ALERT | foo "

Is this possible to avoid having to manually update each report whenever the scope of an environment changes?

[martin_mueller · Accepted solution]

You could tag your hosts and search for tag=DEV NOT tag=NO_ALERT.

[rlaan]

How do you allow all users to see tags that area created?

May they be quickly created using the deployment server for easier management? For a large amount of hosts the gui seems bulky

[martin_mueller]

As with most objects in Splunk, you can control a tag's sharing between private (default), app (requires write permissions for the app), and global (requires admin) as well as per-role read and write permissions - just go to Settings -> Tags -> All unique tags -> Permissions for the tag you want to share.

For adding a large list I'd drop down to tags.conf and insert them there in bulk - whether you do that on your search head, or through a deployment server managed app doesn't really matter. Just do it like you manage all your Splunk configuration.
Alternatively, you could maintain a lookup file that resolves hosts to a stage field, and have users search using that field. Should be easier to maintain for large lists, but can be slower to search for.