Why are there duplicate forwarder entries IP and FQDN after upgrade from 6.1.4 to 6.2?

[larrymagstadt]

I upgraded from 6.1.4 to 6.2 for the server and the universal forwarders. Afterwards there are duplicate entries on the All Forwarders screen for each server; one with the ip address as the Forwarder designation and another with the fqdn. The record with the ip address shows:

Splunk version: pre 4.2
Platform: n/a
Last Connected: updating every ten minutes
Current Status: missing

The record with the fqdn, shows 6.2 for the version and all of the other information like before the upgrade. I thought that the records with the ip address would be cleaned up after 24 hours since they have a status of missing but since the connection time still updates they have remained.

Any clue as two why there are two forwarder records? And how would I clear up the ones that have the ip address as the forwarder name?

[larrymagstadt · Accepted solution]

I found out my issue. I did not realize I had to update the deployment monitor app. The "upgrade" was not noticeable until a co-work asked about it. I ran the upgrade and the duplicate entries are now gone.

[lmyrefelt]

Glad you found it out and shared it with us ! 🙂

[andykiely]

I contacted splunk support with the issue and they gave me the following which has worked for me:

We have a workaround for this until it's fixed properly in the code.

If you could, modify $SPLUNK_HOME/etc/apps/splunk_deployment_monitor/default/macros.conf and change this:

[forwarder_metrics]
definition = index="_internal" source="metrics.lo" group=tcpin_connections | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | eval connectionType=case(fwdType=="uf","universal forwarder", fwdType=="lwf", "lightweight forwarder",fwdType=="full", "heavy forwarder", connectionType=="cooked" or connectionType=="cookedSSL","Splunk forwarder", connectionType=="raw" or connectionType=="rawSSL","legacy forwarder")| eval build=if(isnull(build),"n/a",build) | eval version=if(isnull(version),"pre 4.2",version) | eval guid=if(isnull(guid),sourceHost,guid) | eval os=if(isnull(os),"n/a",os)| eval arch=if(isnull(arch),"n/a",arch) | fields connectionType sourceIp sourceHost sourcePort destPort kb tcp_eps tcp_Kprocessed tcp_KBps splunk_server build version os arch guid

To this:
[forwarder_metrics]
definition = index="_internal" source="metrics.lo" group=tcpin_connections NOT eventType=* | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | eval connectionType=case(fwdType=="uf","universal forwarder", fwdType=="lwf", "lightweight forwarder",fwdType=="full", "heavy forwarder", connectionType=="cooked" or connectionType=="cookedSSL","Splunk forwarder", connectionType=="raw" or connectionType=="rawSSL","legacy forwarder")| eval build=if(isnull(build),"n/a",build) | eval version=if(isnull(version),"pre 4.2",version) | eval guid=if(isnull(guid),sourceHost,guid) | eval os=if(isnull(os),"n/a",os)| eval arch=if(isnull(arch),"n/a",arch) | fields connectionType sourceIp sourceHost sourcePort destPort kb tcp_eps tcp_Kprocessed tcp_KBps splunk_server build version os arch guid

[larrymagstadt]

Thanks for adding what you found. I made the modification and now none of the forwards show up. I will just have to wait until the official fix is released.

[nwieseler]

Having the same issue... Upgraded from 6.1.4 to 6.2 and when I view "All Forwarders" in the Deployment Monitor app, I have every forwarder showing missing (listed as the forwarders' IP, pre 4.2, etc.) as well as active using DNS name (not FQDN).

Looked at the answer here and it doesn't really make much sense to me.

Nick

[nwieseler]

I should say all of my forwarders are Windows with a mix of 5.0.3, 6.0, 6.0.1 and I just installed the 6.2 forwarder on a pair of servers (one W2K12R2 one WK8R2) - same deal. IP shows missing, DNS name shows active and looks "normal."

Nick

[andykiely]

I have the same issue but on a windows platform.

If I look in the deployment monitor and do a search for one affected server name and also the ip address I can see it looks like it make two connections both at the same time (as displayed by the lastconnected field). The source port is different for each listing, if I do a capture on the machine I can only see evidence of a single connection.

[larrymagstadt]

Atleast I know that I am not the only one.

[andykiely]

It looks like the deployment monitor builds its logs from the metrics.log file coming from the forwarder. I can see a number of messages that use a sourcehost field and that is being used in the lookup. Run this to see what I mean:

index="_internal" source="metrics.lo" INFO StatusMgr

I think this therefore is some sort of query issue, I'll keep digging.

[larrymagstadt]

I checked our splunk server in our test environment (running 6.1.4) and splunkd creates a second connection to the splunk server on another port, but an extra forwarder is not being created. I am going to compare all of the config files between the two and see what is different between 6.1.4 and 6.2.

[lmyrefelt]

You can / should check $SPLUNK_HOME/etc/system/local/server.conf, $SPLUNK_HOME/etc/system/local/inputs.conf, $SPLUNK_HOME/etc/system/local/deploymentclient.conf to see that they contain the "hostname" you like, either short or fqdn .

Sometimes when installing Splunkforwarder it does not set or change the $HOSTNAME-WHATEVEVER_STUFF_COMES-HERE$ in server.conf

[larrymagstadt]

We use Puppet to deploy the Splunkforwarder so we set the hostname from there. I double checked those files and the server.conf file does have the fqdn for the serverName in the 'general' section.

[lmyrefelt]

And some data / host-names etc might come from within the logs themselves ... ie. using short name / fqdn name

[larrymagstadt]

They are all linux boxes, some opensuse, some centos. Also noticed that when I monitor the open ports (lsof -i); splunkd will be connected to the server on one port and every so often it will connect on a second port for about a second.

[jrodman]

Is this different for all platforms? What platforms do you have for forwarders?