I want to be able to add only a few selected heavy forwarders in my distributed monitoring console.
so basically I want to use wildcard (or may be a text file with list of forwarders or something similar ) for hostnames of these HFs and only add these matching HF's in my Monitoring console in forwarder section..
Is this possible in splunk ?
If you are a customer that has a few heavy forwarders then it probably means you are large enough that you should consider having a stand-alone monitoring console. With a stand-alone monitoring console you should only add the heavy forwarders you care about as search peers. This way your less important heavy forwarders won't be displayed in the monitoring console. Once a server is defined as a search peer to the monitoring console it will be displayed. You can't filter it using a wildcard.
All the best.
Thanks for taking time for this post Chris, much appreciated.
Yes, at the moment I already have these HF's added as indexers.
But I'd like to add and see these HF's as HF in monitoring console in the forwarder section.
as you already guessed, I can NOT add all UF and HFs there because I have 1000s of UF's.
So Can I not add only a few select HF's in as HFs in monitoring console under forwarders?
You cannot select to have only a few forwarders in the Monitoring Console, as this is depending on having the "full view" of everything going on in your environment.
But you could go for the following solution:
A search could look like this:
| inputlookup dmc_forwarder_assets [|inputlookup your_hf_list.csv | return hostname] | search status="missing"
This should give you a list of all missing HF out of your selection.
Also, I have some 60k UF's. if I add them all , will it not be a huge risk for my MC performance? Please advice.
Start by enableing the forwarder monitoring in the MC with Settings => Forwarder Monitoring Setup. You can reduce the data colletion interval if you desire.
This will enable the MC to run internal saved searches, one of which builds the forwarder asset table. This can be accessed in a regular Splunk search with
| inputlookup dmc_forwarder_assets. From there on you can build your custom alert which will only cover your selected Heavy Forwarders. This is not a built-in MC alert anymore, as the standard alerts will alert for any missing forwarder. So your should leave these alerts turned off.
And no, this will not be a huge performance risk for your instance if you have sized it accordingly. Be aware that a MC used for such a large architecture has to be a standalone instance, with no other funcionalities.