Hi, I'm brand new to Splunk and been given an existing Splunk environment to manage. I need to get a universal forwarder installed on a couple servers. This environment already has several universal forwarders in place. I installed the forwarders and selected Windows Application, Security and System logs. The deployment is setup to listen on port 9997.
In the splunkd log on the forwarder server, I see these lines repeated and not sure what they mean. I'd appreciate any help and keep in mind, I'm still very new to this. Thanks!
09-28-2017 18:45:47.694 -0400 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=notconnected
09-28-2017 18:45:59.695 -0400 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=notconnected
09-28-2017 18:46:02.913 -0400 WARN HttpPubSubConnection - HTTP client error in http pubsub Connection closed by peer uri=https://team-splunk01:9997/services/broker/connect/A917C286-95F0-4285-9F0C-8FDE5F9C5596/TEAM-SV-FILE...
09-28-2017 18:46:02.913 -0400 WARN HttpPubSubConnection - Unable to parse message from PubSubSvr:
the first messages means that the new UFs cannot connect to the Deployment Server.
You can test this using telnet on the management port (usually 8089).
Did you configured Deployment Server?
If not, message isn't important.
If yes and connection is OK, check if your UF is seen by the Deployment Server.
When you say: "The deployment is setup to listen on port 9997." are you speaking of Indexer?
To debug connection with Indexers, at first test connection using telnet on 9997 port telnet team-splunk01 9997.
After configure outputs.conf on the forwarders to send logs to Indexers (I usually use Deployment Server, but it's possible to do this also manually.
When outputs.conf is Ok to send logs to indexers (and Splunk restart) check if Indexers are receiving internal logs (index=internal host=UniversalForwarder_hostname).
If it's OK I suggest to use SplunkTAWindows (eventually distributed by Deployment Server) to take Windows logs.