How to retrieve IPs of forwarders for respective host names from the Monitoring Console?

[MousumiChowdhur]

Hi,

I have a requirement of pulling a list of all Splunk instances and the forwarders with their host names, IPs and the type of instances they are (like search head, indexer, uf, heavy forwarder, cluster master, deployment server etc). I have written a search and running it from the Monitoring Console. Please find my search below.

| rest splunk_server_group=dmc_group_cluster_master splunk_server_group="dmc_indexerclustergroup_SplunkProd" /services/cluster/master/peers | eval Category="Indexer" | eval Priority="Critical"
| eval bunit="abc"
| fields label host_port_pair Category Priority bunit
| rename label as Instance, host_port_pair as IP | append [ | rest splunk_server=local /services/search/distributed/peers
| where search_groups="dmc_searchheadclustergroup_SHC1" AND server_roles="search_head"
| eval label = host
| join guid type=outer [
| rest splunk_server_group="dmc_searchheadclustergroup_SHC1" splunk_server_group=dmc_group_search_head /services/shcluster/member/members count=0
| dedup label
| eval guid = title
]
| eval Category="Search Head"
| eval Priority="Critical"
| eval bunit="abc"
| sort role | fields label role Category Priority bunit
| rename label as Instance, role as Role] | append [| inputlookup dmc_forwarder_assets
| eval Category = case(forwarder_type == "full", "Heavy Forwarder", forwarder_type == "uf", "Universal Forwarder", forwarder_type == "lwf", "Light Forwarder", 1==1, forwarder_type) | dedup hostname
| search NOT [| inputlookup dmc_assets | dedup serverName | rename serverName as hostname | fields hostname peerURI]
| fields hostname, Category peerURI
| search hostname="***"
| eval Priority="Critical"
| eval bunit="abc"
| rename hostname as Instance, peerURI as IP] | dedup Instance | eval Category1 = case(Instance == "sedcacsc0140", "Cluster Master", Instance == "sedcacsc0150", "Deplyment Server") | eval category=coalesce(Category1,Category) | fields Instance IP category Priority bunit

The above search gives me the host names, type of instances the hosts are, along with certain additional static information as bunit and priority.
Now, I would like to retrieve the IPs of the respective hosts.

Could you please me with some idea on how can I get the IPs as well?

Thank You!

[lakshman239]

If you have Splunk Enterprise Security and assets are populated, it could have IP's there.
The other option is to run nslookup [comes bundled with ES] using the hostname to get IP [ something like https://splunkbase.splunk.com/app/2922/ might help]. This assumes nslookup can work and resolve the hostnames from the DMC (where you are running searches)

[MousumiChowdhur]

Hi @lakshman239 ,

Thank you for your response. The idea is basically to have an asset of servers that will show me IPs or host names along with other server and location related information. I have all the information coming as a result of my above mentioned search.

Also, I tried nslookup, but its not able to fetch result for all the forwarders. It gave me IPs for a few of the hostnames only.

[lakshman239]

You may have to look at your Asset/IPAM repository/DNS servers for the host/IP, location and other specific information, to pull all the details. When forwarders/other splunk instances connect, i believe we will have either hostname or IP, but not both in one place [ metrics/license.log etc..]. If you have nmap installed, it could help discover host/ip as well.

[dkeck]

Hi,

try to have a look at index=_internal source=*metrics.log group=tcpin_connections

field hostname and sourceIp could give you what your are looking for

[dkeck]

A sheduled search with an output would be a good way to go.

You say thats not reliable enough to have a list based on search time, but I would guess if you have missing forwarders not sending thats something you want to investigate anyway, so you can always compare that to the list you get out of your rest search, to make sure all of them are there.

[MousumiChowdhur]

Yes that makes sense to investigate any missing forwarders but unfortunately that's not really what I'm looking for.
Basically, if I am mentioning a host name /IP as my input then it should give me the respective IP/hostname along with other information like Category, bunit, priority from my search similar to how assets work.

[dkeck]

The only thing I can thing about is the fowarder part of Management Console. Unter "Forwarder Instance" you have a list of all your forwards and you can select one an get IP and other information, but that is on a single peer basis. You might be able to find out the rest call for that dashboard?!

[MousumiChowdhur]

Hi @dkeck ,

Thanks for the comment. I am aware of this approach but I don't want to rely on _internal index. That will only show me list of forwarders that are connected at that point of time or the time frame I mention. Let me know if I'm missing out on anything.

Thank you.

[eavent_splunk]

You could use a scheduled saved search to output the _internal results to a lookup table - that way you have a list not limited by search timeframe (though new forwarders connecting before the populating search is run would not be listed)