Monitoring Splunk

How to manage a lot of forwarder agents

Daniel28
Explorer

Hi team

I have problems when i monitoring many UF (~ 400 agents) with Distributed Architechture  (UF --> HF --> Indexer) as below:

1. When a new UF agent connect to Deployment Server, i can't know it.
2. How to prevent local users from uninstalling UF agent on Client host ? 

3. Monitoring, alert status UP/DOWN of UF agent. 

4. Agent have been deployed Splunk apps or not ? 

Please give me some solutions in my cases. Thanks for your concerns !

Labels (2)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

HI @Daniel28,

this isn't a question, these are 4 questions!

Anyway:

1)

You can know if there's a new Forwarder connected to DS, creating a list of the monitored Forwarders (in a lookup called e.g. "perimer.csv" with at least one column, called e.g. "host") and running (e.g. one time a day) a simple search like this:

| metasearch index=_internal NOT [ | inputlookup perimeter.csv | fields host]
| dedup host
| sort host
| table host

If you have results, there are new Forwarders.

2)

the only way is to have a control on users' systems.

Using Splunk you can know when this happens (see answer 1)  but not block it before.

3)

You can create an alert to know when a Forwarder isn't sending logs, using the above lookup and running (e.g. every 5 minutes) a simple search like this:

| metasearch index=_internal 
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| where total=0

 If you have results, one or more forwarders of the lookup aren't sending logs.

using the same search (without the last row) you can have a dashboard monitoring the status (up(down) of your forwarders.

4)

You can have the status of deployed Apps on your Deployment Server, using [Settings -- Forwardriding Management], or running a search that you can find in my previous answer https://community.splunk.com/t5/Getting-Data-In/How-do-I-get-a-count-of-applications-deployed-to-for...

Ciao.

Giuseppe

View solution in original post

gcusello
SplunkTrust
SplunkTrust

HI @Daniel28,

this isn't a question, these are 4 questions!

Anyway:

1)

You can know if there's a new Forwarder connected to DS, creating a list of the monitored Forwarders (in a lookup called e.g. "perimer.csv" with at least one column, called e.g. "host") and running (e.g. one time a day) a simple search like this:

| metasearch index=_internal NOT [ | inputlookup perimeter.csv | fields host]
| dedup host
| sort host
| table host

If you have results, there are new Forwarders.

2)

the only way is to have a control on users' systems.

Using Splunk you can know when this happens (see answer 1)  but not block it before.

3)

You can create an alert to know when a Forwarder isn't sending logs, using the above lookup and running (e.g. every 5 minutes) a simple search like this:

| metasearch index=_internal 
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| where total=0

 If you have results, one or more forwarders of the lookup aren't sending logs.

using the same search (without the last row) you can have a dashboard monitoring the status (up(down) of your forwarders.

4)

You can have the status of deployed Apps on your Deployment Server, using [Settings -- Forwardriding Management], or running a search that you can find in my previous answer https://community.splunk.com/t5/Getting-Data-In/How-do-I-get-a-count-of-applications-deployed-to-for...

Ciao.

Giuseppe

Daniel28
Explorer

Many thanks to you, @gcusello 

I will follow your guide. But I have other question. 

Now, i'm using Distributed Mode with 2 Heavy Forwarder and 4 Indexer. So what the best location which i can run your command (as above) to show the results ? Because all of UF agent will forward data to 2 Heavy Forwarder. And then, HF will forward to Indexer. 

So I have tried some commands before. But it just show Heavy Forwarder's information.

Please help me clarify it. 

I really appreciate your help !  

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Daniel28,

searches on indexed logs (as _internal) can be done only on Search Heads that access data indexed on Indexers.

Heavy Forwarders are only concentrators and usually don't contain data.

each Indexers contains only a part of data so you cannot use them for searches, in addition, if you have an Indexers cluster, all the local searces are disabled!

Ciao.

Giuseppe

0 Karma

Daniel28
Explorer

Yeah, I will try to do it. 

Thanks for your support, @gcusello 

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...