topic Re: Connection problems with Universal Forwarder for Linux ARM and Splunk Cloud (SSL error) in Deployment Architecture

Connection problems with Universal Forwarder for Linux ARM and Splunk Cloud (SSL error)

DanielaHerold — Thu, 14 Jan 2016 14:29:25 GMT

Hi everyone,

I am currently trying to run the Universal Forwarder for Linux ARM on a Raspberry Pi 2 Model B with an arch linux installed. I want to forward the data to Splunk Cloud, however, I'm having connection problems. Does the Universal Forwarder for Linux ARM work with splunk cloud?

Here is what is installed:

[root@raspi splunk]# cat /proc/version 
Linux version 3.18.8-1-ARCH (builduser@leming) (gcc version 4.9.2 20141224 (prerelease) (GCC) ) #1 SMP PREEMPT Fri Feb 27 19:37:26 MST 2015

My splunkd.log contains the following (many lines with the same):

[root@raspi splunk]# tail splunkd.log 
01-14-2016 12:35:04.697 +0000 ERROR TcpOutputFd - Connection to host=xxx.xxx.xxx.xxx:9997 failed. sock_error = 104. SSL Error = error:00000000:lib(0):func(0):reason(0)
01-14-2016 12:35:04.706 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer

The universal forwarder credentials splunkclouduf.spl are installed. For testing I am monitoring the directory /opt/splunkforwarder/var/log/

Compare the output of list monitor:

[root@raspi splunk]# /opt/splunkforwarder/bin/splunk list monitor                                                                                                                          
Monitored Directories:                                                                                                                                   
$SPLUNK_HOME/var/log/splunk/splunkd.log
/opt/splunkforwarder/var/log/splunk/audit.log
/opt/splunkforwarder/var/log/splunk/btool.log    
...
$SPLUNK_HOME/var/spool/splunk/...stash_new
Monitored Files:
$SPLUNK_HOME/etc/splunk.version

I am also running the Splunk Universal Forwarder Version 6.3.2 on a "normal" Linux (Debian) machine. There it works without problems.

Any help is appreciated! Let me know if you need any more output...

Re: Connection problems with Universal Forwarder for Linux ARM and Splunk Cloud (SSL error)

khourihan_splun — Fri, 15 Jan 2016 00:38:57 GMT

can you do a

$SPLUNK_HOME/bin/splunk btool outputs list --debug

and post it here. Make sure you don't post the sslPassword = part!

Re: Connection problems with Universal Forwarder for Linux ARM and Splunk Cloud (SSL error)

DanielaHerold — Fri, 15 Jan 2016 08:07:40 GMT

Hello,

thanks for your answer. Here's the output:

[root@raspi splunkforwarder]# /opt/splunkforwarder/bin/splunk btool outputs list --debug                
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf [tcpout]
/opt/splunkforwarder/etc/system/default/outputs.conf                        autoLBFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        blockOnCloning = true
/opt/splunkforwarder/etc/system/default/outputs.conf                        blockWarnThreshold = 100
/opt/splunkforwarder/etc/system/default/outputs.conf                        compressed = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        connectionTimeout = 20
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf defaultGroup = my_indexers
/opt/splunkforwarder/etc/system/default/outputs.conf                        disabled = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropClonedEventsOnQueueFull = 5
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf                        forceTimebasedAutoLB = false
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = _audit
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        heartbeatFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        indexAndForward = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxConnectionsPerIndexer = 2
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxFailuresPerInterval = 2
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxQueueSize = auto
/opt/splunkforwarder/etc/system/default/outputs.conf                        readTimeout = 300
/opt/splunkforwarder/etc/system/default/outputs.conf                        secsInFailureInterval = 1
/opt/splunkforwarder/etc/system/default/outputs.conf                        sendCookedData = true
/opt/splunkforwarder/etc/system/default/outputs.conf                        useACK = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        writeTimeout = 300
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf [tcpout:my_indexers]
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf server = xxx.xxx.xxx.xxx:9997
/opt/splunkforwarder/etc/apps/splunkclouduf/local/outputs.conf              [tcpout:splunkcloud]
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf            compressed = false
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf            disabled = false
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf            server = input-prd-p-xxx.cloud.splunk.com:9997
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf            sslCertPath = $SPLUNK_HOME/etc/apps/splunkclouduf/default/client.pem
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf            sslCommonNameToCheck = input-prd-p-xxx.cloud.splunk.com
/opt/splunkforwarder/etc/apps/splunkclouduf/local/outputs.conf              sslPassword = ****
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf            sslRootCAPath = $SPLUNK_HOME/etc/apps/splunkclouduf/default/cacert.pem
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf            sslVerifyServerCert = true
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf            useACK = true

Re: Connection problems with Universal Forwarder for Linux ARM and Splunk Cloud (SSL error)

khourihan_splun — Sat, 16 Jan 2016 23:16:19 GMT

Looks like you have overlapping outputs.conf settings.

from /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder

 /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf [tcpout:my_indexers]
 /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf server = xxx.xxx.xxx.xxx:9997

and from /opt/splunkforwarder/etc/apps/splunkclouduf/

/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf sslCommonNameToCheck = input-prd-p-xxx.cloud.splunk.com

Try moving /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder to /tmp or somewhere safe. Then restart your forwarder.

i.e. /opt/splunkforwarder/bin/splunk restart

and see if that helps. If you need intend to clone your data to two different tcpout locations create this file: /opt/splunkforwarder/etc/apps/splunkclouduf/local/outputs.conf and add thes lines:

[tcpout]
defaultGroup=my_indexers,splunkcloud

You might want to try moving the /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder directory to /tmp first and test it, then if you plan to clone, you can do as I said above.

GL!
Kyle

Re: Connection problems with Universal Forwarder for Linux ARM and Splunk Cloud (SSL error)

DanielaHerold — Mon, 18 Jan 2016 09:31:41 GMT

Ok, thanks. No, I was not intending to clone my data.

So I followed your first suggestion: I moved /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder to /tmp and restarted my forwarder.

Unfortunately, it still doesn't work. My splunkd.log still contains this line:

01-18-2016 09:26:13.140 +0000 ERROR TcpOutputFd - Connection to host=xxx.xxx.xxx.xxx:9997 failed. sock_error = 104. SSL Error = error:00000000:lib(0):func(0):reason(0)

Do you have any other ideas?

Re: Connection problems with Universal Forwarder for Linux ARM and Splunk Cloud (SSL error)

khourihan_splun — Tue, 19 Jan 2016 19:12:36 GMT

can you telnet to the host address on port 9997? maybe a firewall is blocking you?

from: http://openssl.6102.n7.nabble.com/SSL-negotiation-failed-error-00000000-lib-0-func-0-reason-0-td49570.html

SSL negotiation failed: error:00000000:lib(0):func(0):reason(0)

It means no SSL error occurred. Typically you'll see this in a server
environment when a client initiates a connection to the server, but
then immediately disconnects, or sends data other than beginning
SSL negotiation.

So please test connectivity and if you are able to connect we can try something else.

Re: Connection problems with Universal Forwarder for Linux ARM and Splunk Cloud (SSL error)

DanielaHerold — Tue, 19 Jan 2016 20:26:48 GMT

With telnet I'm getting this:

   [root@raspi splunk]# telnet xxx.xxx.xxx.xxx 9997
    Trying xxx.xxx.xxx.xxx...
    Connected to xxx.xxx.xxx.xxx.

So I suppose this is what we want, right?

Re: Connection problems with Universal Forwarder for Linux ARM and Splunk Cloud (SSL error)

khourihan_splun — Tue, 19 Jan 2016 20:41:30 GMT

yeah, that means that your UF can connect to your Splunk Cloud receiver on the right port.

I sent you an email. Let me know if you didn't get it.

Re: Connection problems with Universal Forwarder for Linux ARM and Splunk Cloud (SSL error)

vanders — Sat, 30 Apr 2016 08:19:45 GMT

Did you guys ever solve this? I'm having the same issue on my Raspberry Pi 3 - can telnet to the Splunk Cloud receiver on 9997, but am getting the same SSL errors as the OP.

Thanks,
Matt

Re: Connection problems with Universal Forwarder for Linux ARM and Splunk Cloud (SSL error)

bengoerz — Tue, 02 Jul 2019 00:32:03 GMT

I also got sock_error = 104 when attempting connections to Splunk Cloud.

07-01-2019 15:45:03.234 +0000 ERROR TcpOutputFd - Connection to host=12.34.56.78:9997 failed. sock_error = 104. SSL Error = No error

In my case, the root cause was an upstream device doing SSL inspection (so accepting the TCP connection), but dropping the traffic after it failed to decrypt (because Splunk Cloud uses pre-shared keys instead of a key exchange).

Re: Connection problems with Universal Forwarder for Linux ARM and Splunk Cloud (SSL error)

siddesh333 — Thu, 20 Feb 2020 00:02:08 GMT

Hi,
Is this issue resolved, if yes, could anyone help me with resolution steps
Thanks

Re: Connection problems with Universal Forwarder for Linux ARM and Splunk Cloud (SSL error)

lakshman239 — Wed, 14 May 2025 17:33:44 GMT

@bengoerz - does that mean, we shouldn't SSL inspect the traffic from On-prem splunk instance to splunk cloud traffic, to avoid sock_error = 104? thx

Re: Connection problems with Universal Forwarder for Linux ARM and Splunk Cloud (SSL error)

lakshman239 — Fri, 16 May 2025 10:35:25 GMT

it was an issue with firewall rules, as it dropped packets.