Hi everyone,
I am currently trying to run the Universal Forwarder for Linux ARM on a Raspberry Pi 2 Model B with an arch linux installed. I want to forward the data to Splunk Cloud, however, I'm having connection problems. Does the Universal Forwarder for Linux ARM work with splunk cloud?
Here is what is installed:
[root@raspi splunk]# cat /proc/version
Linux version 3.18.8-1-ARCH (builduser@leming) (gcc version 4.9.2 20141224 (prerelease) (GCC) ) #1 SMP PREEMPT Fri Feb 27 19:37:26 MST 2015
My splunkd.log contains the following (many lines with the same):
[root@raspi splunk]# tail splunkd.log
01-14-2016 12:35:04.697 +0000 ERROR TcpOutputFd - Connection to host=xxx.xxx.xxx.xxx:9997 failed. sock_error = 104. SSL Error = error:00000000:lib(0):func(0):reason(0)
01-14-2016 12:35:04.706 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
The universal forwarder credentials splunkclouduf.spl are installed. For testing I am monitoring the directory /opt/splunkforwarder/var/log/
Compare the output of list monitor:
[root@raspi splunk]# /opt/splunkforwarder/bin/splunk list monitor
Monitored Directories:
$SPLUNK_HOME/var/log/splunk/splunkd.log
/opt/splunkforwarder/var/log/splunk/audit.log
/opt/splunkforwarder/var/log/splunk/btool.log
...
$SPLUNK_HOME/var/spool/splunk/...stash_new
Monitored Files:
$SPLUNK_HOME/etc/splunk.version
I am also running the Splunk Universal Forwarder Version 6.3.2 on a "normal" Linux (Debian) machine. There it works without problems.
Any help is appreciated! Let me know if you need any more output...
I also got sock_error = 104 when attempting connections to Splunk Cloud.
07-01-2019 15:45:03.234 +0000 ERROR TcpOutputFd - Connection to host=12.34.56.78:9997 failed. sock_error = 104. SSL Error = No error
In my case, the root cause was an upstream device doing SSL inspection (so accepting the TCP connection), but dropping the traffic after it failed to decrypt (because Splunk Cloud uses pre-shared keys instead of a key exchange).
can you do a
$SPLUNK_HOME/bin/splunk btool outputs list --debug
and post it here. Make sure you don't post the sslPassword = part!
With telnet I'm getting this:
[root@raspi splunk]# telnet xxx.xxx.xxx.xxx 9997
Trying xxx.xxx.xxx.xxx...
Connected to xxx.xxx.xxx.xxx.
So I suppose this is what we want, right?
yeah, that means that your UF can connect to your Splunk Cloud receiver on the right port.
I sent you an email. Let me know if you didn't get it.
Hi,
Is this issue resolved, if yes, could anyone help me with resolution steps
Thanks
Did you guys ever solve this? I'm having the same issue on my Raspberry Pi 3 - can telnet to the Splunk Cloud receiver on 9997, but am getting the same SSL errors as the OP.
Thanks,
Matt
Ok, thanks. No, I was not intending to clone my data.
So I followed your first suggestion: I moved /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder to /tmp and restarted my forwarder.
Unfortunately, it still doesn't work. My splunkd.log still contains this line:
01-18-2016 09:26:13.140 +0000 ERROR TcpOutputFd - Connection to host=xxx.xxx.xxx.xxx:9997 failed. sock_error = 104. SSL Error = error:00000000:lib(0):func(0):reason(0)
Do you have any other ideas?
can you telnet to the host address on port 9997? maybe a firewall is blocking you?
SSL negotiation failed: error:00000000:lib(0):func(0):reason(0)
It means no SSL error occurred. Typically you'll see this in a server
environment when a client initiates a connection to the server, but
then immediately disconnects, or sends data other than beginning
SSL negotiation.
So please test connectivity and if you are able to connect we can try something else.
Looks like you have overlapping outputs.conf settings.
from /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf [tcpout:my_indexers]
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf server = xxx.xxx.xxx.xxx:9997
and from /opt/splunkforwarder/etc/apps/splunkclouduf/
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf sslCommonNameToCheck = input-prd-p-xxx.cloud.splunk.com
Try moving /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder to /tmp or somewhere safe. Then restart your forwarder.
i.e. /opt/splunkforwarder/bin/splunk restart
and see if that helps. If you need intend to clone your data to two different tcpout locations create this file: /opt/splunkforwarder/etc/apps/splunkclouduf/local/outputs.conf and add thes lines:
[tcpout]
defaultGroup=my_indexers,splunkcloud
You might want to try moving the /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder directory to /tmp first and test it, then if you plan to clone, you can do as I said above.
GL!
Kyle
Hello,
thanks for your answer. Here's the output:
[root@raspi splunkforwarder]# /opt/splunkforwarder/bin/splunk btool outputs list --debug
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf [tcpout]
/opt/splunkforwarder/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunkforwarder/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunkforwarder/etc/system/default/outputs.conf compressed = false
/opt/splunkforwarder/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf defaultGroup = my_indexers
/opt/splunkforwarder/etc/system/default/outputs.conf disabled = false
/opt/splunkforwarder/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunkforwarder/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = _audit
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunkforwarder/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf indexAndForward = false
/opt/splunkforwarder/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunkforwarder/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunkforwarder/etc/system/default/outputs.conf maxQueueSize = auto
/opt/splunkforwarder/etc/system/default/outputs.conf readTimeout = 300
/opt/splunkforwarder/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunkforwarder/etc/system/default/outputs.conf sendCookedData = true
/opt/splunkforwarder/etc/system/default/outputs.conf useACK = false
/opt/splunkforwarder/etc/system/default/outputs.conf writeTimeout = 300
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf [tcpout:my_indexers]
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf server = xxx.xxx.xxx.xxx:9997
/opt/splunkforwarder/etc/apps/splunkclouduf/local/outputs.conf [tcpout:splunkcloud]
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf compressed = false
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf disabled = false
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf server = input-prd-p-xxx.cloud.splunk.com:9997
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf sslCertPath = $SPLUNK_HOME/etc/apps/splunkclouduf/default/client.pem
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf sslCommonNameToCheck = input-prd-p-xxx.cloud.splunk.com
/opt/splunkforwarder/etc/apps/splunkclouduf/local/outputs.conf sslPassword = ****
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf sslRootCAPath = $SPLUNK_HOME/etc/apps/splunkclouduf/default/cacert.pem
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf sslVerifyServerCert = true
/opt/splunkforwarder/etc/apps/splunkclouduf/default/outputs.conf useACK = true