Getting Data In

Why are there duplicate forwarder entries IP and FQDN after upgrade from 6.1.4 to 6.2?

Explorer

I upgraded from 6.1.4 to 6.2 for the server and the universal forwarders. Afterwards there are duplicate entries on the All Forwarders screen for each server; one with the ip address as the Forwarder designation and another with the fqdn. The record with the ip address shows:

Splunk version: pre 4.2
Platform: n/a
Last Connected: updating every ten minutes
Current Status: missing

The record with the fqdn, shows 6.2 for the version and all of the other information like before the upgrade. I thought that the records with the ip address would be cleaned up after 24 hours since they have a status of missing but since the connection time still updates they have remained.

Any clue as two why there are two forwarder records? And how would I clear up the ones that have the ip address as the forwarder name?

1 Solution

Explorer

I found out my issue. I did not realize I had to update the deployment monitor app. The "upgrade" was not noticeable until a co-work asked about it. I ran the upgrade and the duplicate entries are now gone.

View solution in original post

0 Karma

Explorer

I found out my issue. I did not realize I had to update the deployment monitor app. The "upgrade" was not noticeable until a co-work asked about it. I ran the upgrade and the duplicate entries are now gone.

View solution in original post

0 Karma

Builder

Glad you found it out and shared it with us ! 🙂

0 Karma

Path Finder

I contacted splunk support with the issue and they gave me the following which has worked for me:

We have a workaround for this until it's fixed properly in the code.

If you could, modify $SPLUNKHOME/etc/apps/splunkdeployment_monitor/default/macros.conf and change this:

[forwardermetrics]
definition = index="
internal" source="metrics.lo" group=tcpinconnections | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | eval connectionType=case(fwdType=="uf","universal forwarder", fwdType=="lwf", "lightweight forwarder",fwdType=="full", "heavy forwarder", connectionType=="cooked" or connectionType=="cookedSSL","Splunk forwarder", connectionType=="raw" or connectionType=="rawSSL","legacy forwarder")| eval build=if(isnull(build),"n/a",build) | eval version=if(isnull(version),"pre 4.2",version) | eval guid=if(isnull(guid),sourceHost,guid) | eval os=if(isnull(os),"n/a",os)| eval arch=if(isnull(arch),"n/a",arch) | fields connectionType sourceIp sourceHost sourcePort destPort kb tcpeps tcpKprocessed tcpKBps splunk_server build version os arch guid

To this:
[forwardermetrics]
definition = index="
internal" source="metrics.lo" group=tcpinconnections NOT eventType=* | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | eval connectionType=case(fwdType=="uf","universal forwarder", fwdType=="lwf", "lightweight forwarder",fwdType=="full", "heavy forwarder", connectionType=="cooked" or connectionType=="cookedSSL","Splunk forwarder", connectionType=="raw" or connectionType=="rawSSL","legacy forwarder")| eval build=if(isnull(build),"n/a",build) | eval version=if(isnull(version),"pre 4.2",version) | eval guid=if(isnull(guid),sourceHost,guid) | eval os=if(isnull(os),"n/a",os)| eval arch=if(isnull(arch),"n/a",arch) | fields connectionType sourceIp sourceHost sourcePort destPort kb tcpeps tcpKprocessed tcpKBps splunk_server build version os arch guid

0 Karma

Explorer

Thanks for adding what you found. I made the modification and now none of the forwards show up. I will just have to wait until the official fix is released.

0 Karma

Path Finder

Having the same issue... Upgraded from 6.1.4 to 6.2 and when I view "All Forwarders" in the Deployment Monitor app, I have every forwarder showing missing (listed as the forwarders' IP, pre 4.2, etc.) as well as active using DNS name (not FQDN).

Looked at the answer here and it doesn't really make much sense to me.

Nick

0 Karma

Path Finder

I should say all of my forwarders are Windows with a mix of 5.0.3, 6.0, 6.0.1 and I just installed the 6.2 forwarder on a pair of servers (one W2K12R2 one WK8R2) - same deal. IP shows missing, DNS name shows active and looks "normal."

Nick

0 Karma

Path Finder

I have the same issue but on a windows platform.

If I look in the deployment monitor and do a search for one affected server name and also the ip address I can see it looks like it make two connections both at the same time (as displayed by the lastconnected field). The source port is different for each listing, if I do a capture on the machine I can only see evidence of a single connection.

0 Karma

Explorer

Atleast I know that I am not the only one.

0 Karma

Path Finder

It looks like the deployment monitor builds its logs from the metrics.log file coming from the forwarder. I can see a number of messages that use a sourcehost field and that is being used in the lookup. Run this to see what I mean:

index="_internal" source="metrics.lo" INFO StatusMgr

I think this therefore is some sort of query issue, I'll keep digging.

0 Karma

Explorer

I checked our splunk server in our test environment (running 6.1.4) and splunkd creates a second connection to the splunk server on another port, but an extra forwarder is not being created. I am going to compare all of the config files between the two and see what is different between 6.1.4 and 6.2.

0 Karma

Builder

You can / should check $SPLUNKHOME/etc/system/local/server.conf, $SPLUNKHOME/etc/system/local/inputs.conf, $SPLUNK_HOME/etc/system/local/deploymentclient.conf to see that they contain the "hostname" you like, either short or fqdn .

Sometimes when installing Splunkforwarder it does not set or change the $HOSTNAME-WHATEVEVERSTUFFCOMES-HERE$ in server.conf

0 Karma

Explorer

We use Puppet to deploy the Splunkforwarder so we set the hostname from there. I double checked those files and the server.conf file does have the fqdn for the serverName in the 'general' section.

0 Karma

Builder

And some data / host-names etc might come from within the logs themselves ... ie. using short name / fqdn name

0 Karma

Explorer

They are all linux boxes, some opensuse, some centos. Also noticed that when I monitor the open ports (lsof -i); splunkd will be connected to the server on one port and every so often it will connect on a second port for about a second.

0 Karma

Splunk Employee
Splunk Employee

Is this different for all platforms? What platforms do you have for forwarders?

0 Karma