How to manage a lot of forwarder agents

[Daniel28]

Hi team

I have problems when i monitoring many UF (~ 400 agents) with Distributed Architechture  (UF --> HF --> Indexer) as below:

1. When a new UF agent connect to Deployment Server, i can't know it.
2. How to prevent local users from uninstalling UF agent on Client host ? 

3. Monitoring, alert status UP/DOWN of UF agent. 

4. Agent have been deployed Splunk apps or not ? 

Please give me some solutions in my cases. Thanks for your concerns !

[gcusello · Accepted solution]

HI @Daniel28,

this isn't a question, these are 4 questions!

Anyway:

1)

You can know if there's a new Forwarder connected to DS, creating a list of the monitored Forwarders (in a lookup called e.g. "perimer.csv" with at least one column, called e.g. "host") and running (e.g. one time a day) a simple search like this:

| metasearch index=_internal NOT [ | inputlookup perimeter.csv | fields host]
| dedup host
| sort host
| table host

If you have results, there are new Forwarders.

2)

the only way is to have a control on users' systems.

Using Splunk you can know when this happens (see answer 1)  but not block it before.

3)

You can create an alert to know when a Forwarder isn't sending logs, using the above lookup and running (e.g. every 5 minutes) a simple search like this:

| metasearch index=_internal 
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| where total=0

 If you have results, one or more forwarders of the lookup aren't sending logs.

using the same search (without the last row) you can have a dashboard monitoring the status (up(down) of your forwarders.

4)

You can have the status of deployed Apps on your Deployment Server, using [Settings -- Forwardriding Management], or running a search that you can find in my previous answer https://community.splunk.com/t5/Getting-Data-In/How-do-I-get-a-count-of-applications-deployed-to-for...

Ciao.

Giuseppe

[Daniel28]

Many thanks to you, @gcusello 

I will follow your guide. But I have other question. 

Now, i'm using Distributed Mode with 2 Heavy Forwarder and 4 Indexer. So what the best location which i can run your command (as above) to show the results ? Because all of UF agent will forward data to 2 Heavy Forwarder. And then, HF will forward to Indexer. 

So I have tried some commands before. But it just show Heavy Forwarder's information.

Please help me clarify it. 

I really appreciate your help !  

[gcusello]

Hi @Daniel28,

searches on indexed logs (as _internal) can be done only on Search Heads that access data indexed on Indexers.

Heavy Forwarders are only concentrators and usually don't contain data.

each Indexers contains only a part of data so you cannot use them for searches, in addition, if you have an Indexers cluster, all the local searces are disabled!

Ciao.

Giuseppe

[Daniel28]

Yeah, I will try to do it. 

Thanks for your support, @gcusello