Getting Data In

Why is universal forwarder connection failing and getting "Socket error from x.x.x.x while idling: error: ...SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol."?

AllenRed
New Member

I cannot get Splunk enterprise to work. I am using the free version (6.x), and I have an all Linux environment. It is already set up to use one client server fine. I am tasked with getting it set up to work with a second server using the universal forwarder. I configured a receiver port, and inputs.conf file (on the main splunk server) and an outputs.conf file (on the client server). I looked at the splunkd.log files on both the main Splunk server and the forwarder client server.

I tried different ports (8089 and 9997). I wonder if the servers don't have some security issue preventing port 9997 (e.g., some firewall issue). Maybe port 8089 is reserved for something else. Based on the error messages, there seems to be an attempt over port 80 to establish a connection. Would port 80 need to be listening? To access the web UI, I use port 8000 in the URL of the web browser.

The splunkd.log files indicate the following errors (on the main splunk server):

Socket error from x.x.x.x while idling: error: ...SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol.

WARN TcpOutputFd - Connect to x.x.x.x:80 failed. Connection refused.
ERROR TcpOutputFd - Connection to host=x.x.x.x failed.

(I removed some text and replaced it with an ellipsis. I replaced the IP address with x.x.x.x. I verified that the IP address is correct for the client server.)

Restarting splunk services on both the client and main splunk server didn't help.

How can I get log data from the client server (with a universal forwarder) using Splunk?

How many fields in the inputs.conf and outputs.conf files are optional? My versions are sparse.

0 Karma

AllenRed
New Member

It seems like port 9997 is closed on my network. At this time of year, I cannot get someone to determine if it is working or not. iptables doesn't block this port on either machine (the client forwarder that I want to get working or the Splunk server). I installed telnet on both machines. It seems like port 9997 isn't working. However, a splunk client is working (and connecting to the splunk server). I want to get a second client working.

On the forwarder I want to get working for the first time, the output of this command (from /opt/splunkforwarder/bin/) is nothing:

# ./splunk cmd btool output list --debug 

The output of this command from /opt/splunkforwarder/bin/,

# ./splunk cmd btool inputs list splunktcp --debug

is as follows:

/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf [splunktcp]
/opt/splunkforwarder/etc/system/default/inputs.conf                        _rcvbuf = 1572864
/opt/splunkforwarder/etc/system/default/inputs.conf                        acceptFrom = *
/opt/splunkforwarder/etc/system/default/inputs.conf                        connection_host = ip
/opt/splunkforwarder/etc/system/local/inputs.conf                          host = cooltest.domainName.cloud
/opt/splunkforwarder/etc/system/default/inputs.conf                        index = default
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf route = has_key:tautology:parsingQueue;absent_key:tautology:parsingQueue

What should I do?


@mods: When I tried to respond to martin's comment about talking to networking people, I got this error: "We're sorry, but you do not have enough permissions to post a comment.
If you believe this to be in error, contact your system administrator."

0 Karma

ppablo
Retired

Thanks for the heads up @AllenRed about the permissions issue. Someone brought it up just 2 days ago. It should have been fixed by now, but if you're still getting the permissions error when trying to leave comments, shoot me an email at ppablo@splunk.com

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

The splunktcp input on the forwarder is irrelevant, it matters only on the indexer - your forwarder isn't listening for data from other forwarders.

To tell this new forwarder to send data you could either add an entry to outputs.conf, or run this:

./splunk add forward-server host:port

Use the host and port combo that worked on your other forwarder.

0 Karma

AllenRed
New Member

Check if you can reach the indexer by
running telnet indexer-host 9997

telnet isn't installed on my version of Linux. I can ssh to the indexer from the forwarder. I cannot seem to test ssh with the -p flag (to force it over one port). I get connection refused or a hanging command:

$ ssh indexer-host -p 9997

I tried different ports. Some come back with connection refused. Others hang, and I don't go to a new prompt. Thank you very much for your assistance. I hope we can troubleshoot this port issue.

How do I test the port without telnet?

0 Karma

ppablo
Retired

Hi @AllenRed

I already converted your answer to a comment appropriately under Martin's comment below since you were responding to this:

"The forwarder is using 8089 by default as its own management port, so it's okay if it's listening on that.

Check the open connections using netstat -a, you should see a listen on 8089. Check if you can reach the indexer by running telnet indexer-host 9997."

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Your forwarder is sending to port 8089, which is the Splunk Management Port by default. That can't receive forwarder data.

Your indexer is listening for forwarder data on 8098 and 9997, make the forwarder use one of these two.

The error message you get is about SSL because the HTTPS on 8089 is getting plain-text TCP from the forwarder, which it can't understand when looking for SSL.

martin_mueller
SplunkTrust
SplunkTrust

Talk to your networking people then, have them help you figure out what ports are reachable from the forwarder. Once you know that, configure the splunktcp:abcd in inputs.conf on the indexer and the server in outputs.conf on the forwarder with such a port (other than 8089).

If they ask you what port to open, I'd recommend 9997 because it lines up with the default in all the documentation.

0 Karma

AllenRed
New Member

Check if you can reach the indexer by
running telnet indexer-host 9997

Telnet isn't installed on my version of Linux. I can ssh to the indexer. I cannot seem to test ssh with the -p flag (to force it over one port). I get connection refused or a hanging command:

$ ssh indexer-host -p 9997

I tried different ports. Some come back with connection refused. Others hang, and I don't go to a new prompt. Thank you very much for your assistance. I hope we can troubleshoot this port issue.

0 Karma

ppablo
Retired

Hi @AllenRed

Please be sure to respond to @martin_mueller by typing in the "Add your comment" box directly below his response/comment. Right now, you keep posting a brand new answer in the box that says "Enter your answer here..." at the very bottom of the page each time. Your other "Answer" has too many characters to convert to a comment properly at this point, so just respond appropriately from this point on. Thanks!

Patrick

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

The forwarder is using 8089 by default as its own management port, so it's okay if it's listening on that.

Check the open connections using netstat -a, you should see a listen on 8089. Check if you can reach the indexer by running telnet indexer-host 9997.

0 Karma

AllenRed
New Member

I changed the outputs.conf file on the forwarder (to use port 9997). But now when I restart splunk on the forwarder, it is using port 8089. I used this command:

grep -rl 8089

It finds /opt/splunkforwarder/bin/splunk and /opt/splunkforwarder/bin/splunkd. This are binary files. How do I get the forwarder to use port 9997?

0 Karma

mandarpimplapur
Explorer

This port need to be enabled from Splunk Master UI :
Manager --> Forwarding and receiving --> Receive Data --> Add a new entry for Port 9997 or any.

Bounce the service once to changes affect.

0 Karma

AllenRed
New Member

I ran this on the forwarder:

$SPLUNK_HOME/bin/splunk cmd btool
outputs list --debug

/opt/splunkforwarder/etc/system/default/outputs.conf                        [syslog]
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxEventSize = 1024
/opt/splunkforwarder/etc/system/default/outputs.conf                        priority = <13>
/opt/splunkforwarder/etc/system/default/outputs.conf                        type = udp
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf [tcpout]
/opt/splunkforwarder/etc/system/default/outputs.conf                        autoLBFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        blockOnCloning = true
/opt/splunkforwarder/etc/system/default/outputs.conf                        blockWarnThreshold = 100
/opt/splunkforwarder/etc/system/default/outputs.conf                        compressed = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        connectionTimeout = 20
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf defaultGroup = indexCluster
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf disabled = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropClonedEventsOnQueueFull = 5
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf                        forceTimebasedAutoLB = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        forwardedindex.0.whitelist = .*
/opt/splunkforwarder/etc/system/default/outputs.conf                        forwardedindex.1.blacklist = _.*
/opt/splunkforwarder/etc/system/default/outputs.conf                        forwardedindex.2.whitelist = (_audit|_internal|_introspection)
/opt/splunkforwarder/etc/system/default/outputs.conf                        forwardedindex.filter.disable = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        heartbeatFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        indexAndForward = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxConnectionsPerIndexer = 2
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxFailuresPerInterval = 2
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxQueueSize = auto
/opt/splunkforwarder/etc/system/default/outputs.conf                        readTimeout = 300
/opt/splunkforwarder/etc/system/default/outputs.conf                        secsInFailureInterval = 1
/opt/splunkforwarder/etc/system/default/outputs.conf                        sendCookedData = true
/opt/splunkforwarder/etc/system/default/outputs.conf                        sslQuietShutdown = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        useACK = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        writeTimeout = 300
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf [tcpout:indexCluster]
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf server = 10.198.221.166:8089

I ran this on the indexer:

$SPLUNK_HOME/bin/splunk cmd btool
inputs list splunktcp --debug

It produced this as the output:

/opt/splunk/etc/system/default/inputs.conf      [splunktcp]
/opt/splunk/etc/system/default/inputs.conf      _rcvbuf = 1572864
/opt/splunk/etc/system/default/inputs.conf      acceptFrom = *
/opt/splunk/etc/system/default/inputs.conf      connection_host = ip
/opt/splunk/etc/system/local/inputs.conf        host = cooltest.domainName.cloud
/opt/splunk/etc/system/default/inputs.conf      index = default
/opt/splunk/etc/system/default/inputs.conf      route = has_key:_replicationBucketUUID:replicationQueue;has_key:_dstrx:typingQueue;has_key:_linebreaker:indexQueue;absent_key:_linebreaker:parsingQueue
/opt/splunk/etc/apps/launcher/local/inputs.conf [splunktcp://8098]
/opt/splunk/etc/system/default/inputs.conf      _rcvbuf = 1572864
/opt/splunk/etc/apps/launcher/local/inputs.conf connection_host = ip
/opt/splunk/etc/system/local/inputs.conf        host = cooltest.domainName.cloud
/opt/splunk/etc/system/default/inputs.conf      index = default
/opt/splunk/etc/apps/launcher/local/inputs.conf [splunktcp://9997]
/opt/splunk/etc/system/default/inputs.conf      _rcvbuf = 1572864
/opt/splunk/etc/apps/launcher/local/inputs.conf connection_host = ip
/opt/splunk/etc/system/local/inputs.conf        disabled = 0
/opt/splunk/etc/system/local/inputs.conf        host = cooltest.domainName.cloud
/opt/splunk/etc/system/default/inputs.conf      index = default

I ran this on the indexer:

$SPLUNK_HOME/bin/splunk cmd btool
inputs list splunktcp-ssl --debug

It produced no output. There was no error either.

Do you need anything further to help me?

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Do post the output of this on the forwarder:

$SPLUNK_HOME/bin/splunk cmd btool outputs list --debug

and of this on the indexer:

$SPLUNK_HOME/bin/splunk cmd btool inputs list splunktcp --debug
$SPLUNK_HOME/bin/splunk cmd btool inputs list splunktcp-ssl --debug

Alternatively, do post what you have set in outputs.conf on the forwarder and inputs.conf on the indexer regarding the forwarding and receiving.

I'm most leaning towards either a port issue or an SSL issue, or both.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...