Deployment Architecture
Highlighted

Listing forwarders

Explorer

Hi all,

I have been trying to identify a list the current forwarders that are sending data to our single Splunk indexer. Is there a section within Splunk where i can find this or even a search query?

Thanks in advance.
Anu

Tags (1)
Highlighted

Re: Listing forwarders

Communicator

If you are forwarding _internal indexes from the forwarders, then the data should all be in the _internal index on your indexer.

forwardedindex.filter.disable = true in outputs.conf would achieve this.

The deployment monitor app would then show you all forwarders out of the box.

0 Karma
Highlighted

Re: Listing forwarders

Explorer

Thanks for that input. I am not sure if we are forwarding _internal indexes (I'm fairly new to Splunk and am still learning my way around the software) from the forwarders but i will investigate and try it out.

Cheers

0 Karma
Highlighted

Re: Listing forwarders

Legend

The universal forwarder does not have indexes. But it does forward its internal logs by default - so the effect is the same. And you don't need to do anything to get it.

If you are using a heavy forwarder, you will need to set it to forwarder rather than index. The following documentation is written for a search head - but the settings for a heavy forwarder will be exactly the same.
Best practice: Forward search head data into the indexing layer

0 Karma
Highlighted

Re: Listing forwarders

Communicator

I would suggest a query to the metadata using the search

| metadata type="hosts"

Should list the various hosts delivering you events.

If you just want the splunk forwarders you can try the following shell command:
splunk cmd btool inputs list splunktcp

Highlighted

Re: Listing forwarders

Legend

The trouble with this, is that the hosts listed will be the name of the host specified in inputs.conf; that might or might not match the actual forwarder names.
By using the _internal index, you see the actual IP address and server name of the forwarder.

0 Karma
Highlighted

Re: Listing forwarders

Legend

Here is a search that I often use to check on how much data is being sent per hour, by forwarder.

index=_internal source=*metrics.log group=tcpin_connections 
| eval sourceHost=if(isnull(hostname), sourceHost,hostname) 
| rename connectionType as connectType
| eval connectType=case(fwdType=="uf","univ fwder", fwdType=="lwf", "lightwt fwder",fwdType=="full", "heavy fwder", connectType=="cooked" or connectType=="cookedSSL","Splunk fwder", connectType=="raw" or connectType=="rawSSL","legacy fwder")
| eval version=if(isnull(version),"pre 4.2",version)
| rename version as Ver 
| fields connectType sourceIp sourceHost destPort kb tcp_eps tcp_Kprocessed tcp_KBps splunk_server Ver
| eval Indexer= splunk_server
| eval Hour=relative_time(_time,"@h")
| stats avg(tcp_KBps) sum(tcp_eps) sum(tcp_Kprocessed) sum(kb) by Hour connectType sourceIp sourceHost destPort Indexer Ver
| fieldformat Hour=strftime(Hour,"%x %H")

Just copy this search and paste into your search box - and pick a relatively short time period (like last 24 hours or less). It should run on any Splunk 4.2 or newer. It might work on older versions, but I am not sure...

You could change the stats command if you wanted a slightly different output. For example, replace the last 3 lines with the following to get an overall summary by forwarder, rather than hour by hour statistics:

| stats avg(tcp_KBps) sum(tcp_eps) sum(tcp_Kprocessed) sum(kb) by connectType sourceIp sourceHost destPort Indexer Ver

I originally found this search as part of the Spunk Deployment Monitor. I've been tweaking it ever since.

View solution in original post

Highlighted

Re: Listing forwarders

Path Finder

I just wanted to thank you - I modified your search to help me find out of date forwarders:

index=_internal source=*metrics.log group=tcpin_connections | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | dedup sourceHost | table sourceHost sourceIP os version | sort version

Highlighted

Re: Listing forwarders

Explorer

I used... this search (above)... and added:

| stats count as vercount by version

As one of our execs wanted to know how many of each version we were running.

0 Karma
Highlighted

Re: Listing forwarders

Engager
index=_internal source=*metrics.log group=tcpin_connections | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | dedup sourceHost | table sourceHost sourceIP os version | sort version

Thanks for sharing this, but when we use this search string we get duplicates, where forwarders in the results list both hostnames and a duplicate records for each as IP address. So we have two results for each forwarder, one with hostname and another with just the IP. Also, the hostname column does not populate the IP address column.

0 Karma