Why is one of my universal forwarders trying to contact the deployment server on port 9997?

[splunk_cv]

Hi,

I have a configuration where many Universal Forwarders are managed by a Deployment Server.
Today I installed a new UF on a Windows machine, and I have several problems:

• in the internal log I see WARN TcpOutputFd - Connect to :9997 failed. No connection could be made because the target machine actively refused it and ERROR TcpOutputFd - Connection to host=:9997 failed -> Why is the UF trying to contact the DS on 9997?

• in the internal log, I can't see the phonehome logs, but looking at the ASA logs, I can see the communication between UF and DS on port 8089 every minute and so I suppose this is the phonehome

• Main problem: I have configured the inputs.conf to monitor a path, but I can't see the logs, maybe because the errors above.

  [monitor://C:\path\ue_*.log]
  index=ftp
  sourcetype=ftp
  In the outputs.conf I have listed, as usual, my 6 indexers on port 9997.

[lguinn2 · Accepted solution]

In reply to your bullet points

1 - The TcpOutputFd component is trying to send data to an indexer - it is NOT trying to contact the deployment server. It looks like there is probably an error in outputs.conf. And there is probably more than one outputs.conf...

2 - If the forwarder is not actually forwarding, you won't see any "phone home" messages in the _internal index. You can, however, go to the splunkd.log on the forwarder and check for phone home messages. If deploymentclient.conf is set up properly, you will see them there or error messages.

3 - if the forwarder is not forwarding, of course your inputs will not be sent to the indexers and the data will not be there.

There were a lot of garbled lines in your btool output. But the last 4 lines are telling:

UF\etc\system\local\outputs.conf                        [tcpout-server://10.73.21.105:9997]
UF\etc\system\local\outputs.conf                       server = 10.73.21.105:9997
UF\etc\apps\OA-output_forwarder\default\outputs.conf   [tcpout:indexmi_9997]
UF\etc\apps\OA-output_forwarde\\default\outputs.conf   server = 10.73.21.103:9997,10.73.21.104:9997,10.73.21.157:9997,
10.73.21.158:9997,10.73.21.159:9997,10.73.21.160:9997

You have a setting in etc/system/local as well as in your app. Because the stanzas are different, they do not conflict - but how does Splunk know which stanza to use? And of course, anything in etc/system/local cannot be removed with the Deployment Server, it will have to be removed manually.

If you look at splunkd.log on the forwarder, you will probably see exactly where and how it was attempting to forward. During your uninstall process, perhaps you removed the bad configuration and then the forwarder unblocked and sent its data.

btool is your friend!

[splunk_cv]

Thanks for the answer. I found the problem on my own some days ago, analyzing the btool's output: the FW installation was made from a third person, and he indicates the DS also as an IX during the installation process, so of course the FW tryed to send data the DS instead of the IXs.

[splunk_cv]

Today i tried to unistall the Universal Forwarder on the remote host, and i noticed that after it was uninstalled (or during the uninstallation operations, i can't say it exactly) I receveid all the internal log that i haven't received previously. Someone can explain this strange behavior?

[diogofgm]

did you delivered the inputs.conf to the new UF via deployment server?

[schose]

Could you provide output of the command "spunk btool outputs list --debug" from one of the forwarders?

[splunk_cv]

Yes i delivered it via DS, i created an app with inputs.conf in it.
@schose this is a Windows UF, could you provide me instruction for the command under Windows systems?

[schose]

in command prompt type

"C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" btool outputs list --debug 

[splunk_cv]

@schose here is the ouptut:

Note: The OA-output_forwarder is an app deployed on every forwarder with the outputs.conf file configured to send data to our 6 indexers.

UF\etc\system\default\outputs.conf type = udp
UF\etc\apps\SplunkUniversalForwarder\default\outputs.conf [tcpout]
UF\etc\system\default\outputs.conf ackTimeoutOnShutdown = 30

UF\etc\system\default\outputs.conf blockOnCloning = true

UF\etc\system\default\outputs.conf compressed = false
UF\etc\system\default\outputs.conf connectionTimeout = 20

UF\etc\system\default\outputs.conf forceTimebasedAutoLB = false
UF\etc\apps\SplunkUniversalForwarder\default\outputs.conf forwardedindex.0.whitelist = .*
UF\etc\apps\SplunkUniversalForwarder\default\outputs.conf forwardedindex.1.blacklist = _.*
UF\etc\apps\SplunkUniversalForwarder\default\outputs.conf forwardedindex.2.whitelist = (_audit|_introspection)
UF\etc\apps\SplunkUniversalForwarder\default\outputs.conf forwardedindex.filter.disable = false
UF\etc\system\default\outputs.conf indexAndForward = false
UF\etc\system\default\outputs.conf maxConnectionsPerIndexer = 2
UF\etc\system\default\outputs.conf maxFailuresPerInterval = 2
UF\etc\system\default\outputs.conf readTimeout = 300
UF\etc\system\default\outputs.conf secsInFailureInterval = 1
UF\etc\system\default\outputs.conf sslQuietShutdown = false
UF\etc\system\default\outputs.conf useACK = false
UF\etc\system\local\outputs.conf [tcpout-server://10.73.21.105:9997]
UF\etc\system\local\outputs.conf server = 10.73.21.105:9997
UF\etc\apps\OA-output_forwarder\default\outputs.conf [tcpout:indexmi_9997]
UF\etc\apps\OA-output_forwarder\default\outputs.conf server = 10.73.21.103:9997,10.73.21.104:9997,10.73.21.157:9997,
10.73.21.158:9997,10.73.21.159:9997,10.73.21.160:9997

(I reformatted the btool output for readability -@lguinn)