Monitoring Splunk

How to manage a lot of forwarder agents

Daniel28
Explorer

Hi team

I have problems when i monitoring many UF (~ 400 agents) with Distributed Architechture  (UF --> HF --> Indexer) as below:

1. When a new UF agent connect to Deployment Server, i can't know it.
2. How to prevent local users from uninstalling UF agent on Client host ? 

3. Monitoring, alert status UP/DOWN of UF agent. 

4. Agent have been deployed Splunk apps or not ? 

Please give me some solutions in my cases. Thanks for your concerns !

Labels (2)
0 Karma
1 Solution

gcusello
Legend

HI @Daniel28,

this isn't a question, these are 4 questions!

Anyway:

1)

You can know if there's a new Forwarder connected to DS, creating a list of the monitored Forwarders (in a lookup called e.g. "perimer.csv" with at least one column, called e.g. "host") and running (e.g. one time a day) a simple search like this:

| metasearch index=_internal NOT [ | inputlookup perimeter.csv | fields host]
| dedup host
| sort host
| table host

If you have results, there are new Forwarders.

2)

the only way is to have a control on users' systems.

Using Splunk you can know when this happens (see answer 1)  but not block it before.

3)

You can create an alert to know when a Forwarder isn't sending logs, using the above lookup and running (e.g. every 5 minutes) a simple search like this:

| metasearch index=_internal 
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| where total=0

 If you have results, one or more forwarders of the lookup aren't sending logs.

using the same search (without the last row) you can have a dashboard monitoring the status (up(down) of your forwarders.

4)

You can have the status of deployed Apps on your Deployment Server, using [Settings -- Forwardriding Management], or running a search that you can find in my previous answer https://community.splunk.com/t5/Getting-Data-In/How-do-I-get-a-count-of-applications-deployed-to-for...

Ciao.

Giuseppe

View solution in original post

gcusello
Legend

HI @Daniel28,

this isn't a question, these are 4 questions!

Anyway:

1)

You can know if there's a new Forwarder connected to DS, creating a list of the monitored Forwarders (in a lookup called e.g. "perimer.csv" with at least one column, called e.g. "host") and running (e.g. one time a day) a simple search like this:

| metasearch index=_internal NOT [ | inputlookup perimeter.csv | fields host]
| dedup host
| sort host
| table host

If you have results, there are new Forwarders.

2)

the only way is to have a control on users' systems.

Using Splunk you can know when this happens (see answer 1)  but not block it before.

3)

You can create an alert to know when a Forwarder isn't sending logs, using the above lookup and running (e.g. every 5 minutes) a simple search like this:

| metasearch index=_internal 
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| where total=0

 If you have results, one or more forwarders of the lookup aren't sending logs.

using the same search (without the last row) you can have a dashboard monitoring the status (up(down) of your forwarders.

4)

You can have the status of deployed Apps on your Deployment Server, using [Settings -- Forwardriding Management], or running a search that you can find in my previous answer https://community.splunk.com/t5/Getting-Data-In/How-do-I-get-a-count-of-applications-deployed-to-for...

Ciao.

Giuseppe

Daniel28
Explorer

Many thanks to you, @gcusello 

I will follow your guide. But I have other question. 

Now, i'm using Distributed Mode with 2 Heavy Forwarder and 4 Indexer. So what the best location which i can run your command (as above) to show the results ? Because all of UF agent will forward data to 2 Heavy Forwarder. And then, HF will forward to Indexer. 

So I have tried some commands before. But it just show Heavy Forwarder's information.

Please help me clarify it. 

I really appreciate your help !  

0 Karma

gcusello
Legend

Hi @Daniel28,

searches on indexed logs (as _internal) can be done only on Search Heads that access data indexed on Indexers.

Heavy Forwarders are only concentrators and usually don't contain data.

each Indexers contains only a part of data so you cannot use them for searches, in addition, if you have an Indexers cluster, all the local searces are disabled!

Ciao.

Giuseppe

0 Karma

Daniel28
Explorer

Yeah, I will try to do it. 

Thanks for your support, @gcusello 

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...