Connection problems with Universal Forwarder for Linux ARM and Splunk Cloud (SSL error)

[DanielaHerold]

Hi everyone,

I am currently trying to run the Universal Forwarder for Linux ARM on a Raspberry Pi 2 Model B with an arch linux installed. I want to forward the data to Splunk Cloud, however, I'm having connection problems. Does the Universal Forwarder for Linux ARM work with splunk cloud?

Here is what is installed:

[root@raspi splunk]# cat /proc/version 
Linux version 3.18.8-1-ARCH (builduser@leming) (gcc version 4.9.2 20141224 (prerelease) (GCC) ) #1 SMP PREEMPT Fri Feb 27 19:37:26 MST 2015

My splunkd.log contains the following (many lines with the same):

[root@raspi splunk]# tail splunkd.log 
01-14-2016 12:35:04.697 +0000 ERROR TcpOutputFd - Connection to host=xxx.xxx.xxx.xxx:9997 failed. sock_error = 104. SSL Error = error:00000000:lib(0):func(0):reason(0)
01-14-2016 12:35:04.706 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer

The universal forwarder credentials splunkclouduf.spl are installed. For testing I am monitoring the directory /opt/splunkforwarder/var/log/

Compare the output of list monitor:

[root@raspi splunk]# /opt/splunkforwarder/bin/splunk list monitor                                                                                                                          
Monitored Directories:                                                                                                                                   
$SPLUNK_HOME/var/log/splunk/splunkd.log
/opt/splunkforwarder/var/log/splunk/audit.log
/opt/splunkforwarder/var/log/splunk/btool.log    
...
$SPLUNK_HOME/var/spool/splunk/...stash_new
Monitored Files:
$SPLUNK_HOME/etc/splunk.version

I am also running the Splunk Universal Forwarder Version 6.3.2 on a "normal" Linux (Debian) machine. There it works without problems.

Any help is appreciated! Let me know if you need any more output...

[bengoerz]

I also got sock_error = 104 when attempting connections to Splunk Cloud.

07-01-2019 15:45:03.234 +0000 ERROR TcpOutputFd - Connection to host=12.34.56.78:9997 failed. sock_error = 104. SSL Error = No error

In my case, the root cause was an upstream device doing SSL inspection (so accepting the TCP connection), but dropping the traffic after it failed to decrypt (because Splunk Cloud uses pre-shared keys instead of a key exchange).

[khourihan_splun]

can you do a

$SPLUNK_HOME/bin/splunk btool outputs list --debug 

and post it here. Make sure you don't post the sslPassword = part!

[DanielaHerold]

With telnet I'm getting this:

   [root@raspi splunk]# telnet xxx.xxx.xxx.xxx 9997
    Trying xxx.xxx.xxx.xxx...
    Connected to xxx.xxx.xxx.xxx.

So I suppose this is what we want, right?

[khourihan_splun]

yeah, that means that your UF can connect to your Splunk Cloud receiver on the right port.

I sent you an email. Let me know if you didn't get it.

[siddesh333]

Hi,
Is this issue resolved, if yes, could anyone help me with resolution steps
Thanks

[vanders]

Did you guys ever solve this? I'm having the same issue on my Raspberry Pi 3 - can telnet to the Splunk Cloud receiver on 9997, but am getting the same SSL errors as the OP.

Thanks,
Matt

[DanielaHerold]

Ok, thanks. No, I was not intending to clone my data.

So I followed your first suggestion: I moved /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder to /tmp and restarted my forwarder.

Unfortunately, it still doesn't work. My splunkd.log still contains this line:

01-18-2016 09:26:13.140 +0000 ERROR TcpOutputFd - Connection to host=xxx.xxx.xxx.xxx:9997 failed. sock_error = 104. SSL Error = error:00000000:lib(0):func(0):reason(0)

Do you have any other ideas?

[khourihan_splun]

can you telnet to the host address on port 9997? maybe a firewall is blocking you?

from: http://openssl.6102.n7.nabble.com/SSL-negotiation-failed-error-00000000-lib-0-func-0-reason-0-td4957...

SSL negotiation failed: error:00000000:lib(0):func(0):reason(0)

It means no SSL error occurred. Typically you'll see this in a server
environment when a client initiates a connection to the server, but
then immediately disconnects, or sends data other than beginning
SSL negotiation.

So please test connectivity and if you are able to connect we can try something else.

[khourihan_splun]

Looks like you have overlapping outputs.conf settings.

from /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder

 /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf [tcpout:my_indexers]
 /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf server = xxx.xxx.xxx.xxx:9997

and from /opt/splunkforwarder/etc/apps/splunkclouduf/

/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf sslCommonNameToCheck = input-prd-p-xxx.cloud.splunk.com

Try moving /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder to /tmp or somewhere safe. Then restart your forwarder.

i.e. /opt/splunkforwarder/bin/splunk restart

and see if that helps. If you need intend to clone your data to two different tcpout locations create this file: /opt/splunkforwarder/etc/apps/splunkclouduf/local/outputs.conf and add thes lines:

[tcpout]
defaultGroup=my_indexers,splunkcloud

You might want to try moving the /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder directory to /tmp first and test it, then if you plan to clone, you can do as I said above.

GL!
Kyle

[DanielaHerold]

Hello,

thanks for your answer. Here's the output:

[root@raspi splunkforwarder]# /opt/splunkforwarder/bin/splunk btool outputs list --debug                
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf [tcpout]
/opt/splunkforwarder/etc/system/default/outputs.conf                        autoLBFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        blockOnCloning = true
/opt/splunkforwarder/etc/system/default/outputs.conf                        blockWarnThreshold = 100
/opt/splunkforwarder/etc/system/default/outputs.conf                        compressed = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        connectionTimeout = 20
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf defaultGroup = my_indexers
/opt/splunkforwarder/etc/system/default/outputs.conf                        disabled = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropClonedEventsOnQueueFull = 5
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf                        forceTimebasedAutoLB = false
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = _audit
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        heartbeatFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        indexAndForward = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxConnectionsPerIndexer = 2
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxFailuresPerInterval = 2
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxQueueSize = auto
/opt/splunkforwarder/etc/system/default/outputs.conf                        readTimeout = 300
/opt/splunkforwarder/etc/system/default/outputs.conf                        secsInFailureInterval = 1
/opt/splunkforwarder/etc/system/default/outputs.conf                        sendCookedData = true
/opt/splunkforwarder/etc/system/default/outputs.conf                        useACK = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        writeTimeout = 300
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf [tcpout:my_indexers]
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf server = xxx.xxx.xxx.xxx:9997
/opt/splunkforwarder/etc/apps/splunkclouduf/local/outputs.conf              [tcpout:splunkcloud]
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf            compressed = false
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf            disabled = false
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf            server = input-prd-p-xxx.cloud.splunk.com:9997
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf            sslCertPath = $SPLUNK_HOME/etc/apps/splunkclouduf/default/client.pem
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf            sslCommonNameToCheck = input-prd-p-xxx.cloud.splunk.com
/opt/splunkforwarder/etc/apps/splunkclouduf/local/outputs.conf              sslPassword = ****
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf            sslRootCAPath = $SPLUNK_HOME/etc/apps/splunkclouduf/default/cacert.pem
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf            sslVerifyServerCert = true
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf            useACK = true