Archive
Highlighted

Splunk not receiving data from forwarder - tried everything in documentation

Engager

Let me preface by saying I've read through multiple threads and tried their recommendations with no luck.

I have a splunk enterprise indexer that is not receiving data from a splunk universal forwarder on a remote server.
When I set it up, it initially sent the data, but since has not updated with new information.

I confirmed that my local box is recieving the connection. There are live established connections between the two over port 9999 (which I set). I confirmed that the firewall rules between here and there are perfectly fine. The connections are happening, but no data is flowing.

I have a data input set up in the indexer and it's enabled.

My inputs.conf in $SPLUNK/etc/apps/search/local :

[monitor:///home/admin/web/MYSERVER/logs/MYSERVER.log]
disabled = false

My inputs.conf in $SPLUNK/etc/system/local:

[default]
host = MYSERVER.NET

The tail end of my splunkd.log on the forwarder:

01-13-2018 16:07:02.330 -0500 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNKHOME/var/spool/splunk/...stashnew.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNKHOME/etc/splunk.version.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK
HOME/var/log/splunk.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNKHOME/var/log/splunk/licenseusagesummary.log.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK
HOME/var/log/splunk/metrics.log.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNKHOME/var/log/splunk/splunkd.log.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor:///home/admin/web/MYSERVER.net/logs/MYSERVER.net.log.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://home/admin/web/MYSERVER.net/logs/MYSERVER.net.log.
01-13-2018 16:07:02.332 -0500 ERROR TailingProcessor - Input stanza path, 'home/admin/web/MYSERVER.net/logs/MYSERVER.net.log' is not absolute. This is a configuration error and may not work / break things. Change this path to an absolute path.
01-13-2018 16:07:02.332 -0500 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
01-13-2018 16:07:02.332 -0500 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Adding watch on path: /home/admin/web/MYSERVER.net/logs/MYSERVER.net.log.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Adding watch on path: /home/admin/web/MYSERVER.net/splunkforwarder/etc/splunk.version.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Adding watch on path: /home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Adding watch on path: /home/admin/web/MYSERVER.net/splunkforwarder/var/spool/splunk.
01-13-2018 16:07:02.333 -0500 INFO loader - Limiting REST HTTP server to 21845 sockets
01-13-2018 16:07:02.333 -0500 INFO loader - Limiting REST HTTP server to 170 threads
01-13-2018 16:07:02.333 -0500 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see:
01-13-2018 16:07:02.343 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/mongod.log'.
01-13-2018 16:07:02.345 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/splunkd
uiaccess.log'.
01-13-2018 16:07:02.346 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/license
usagesummary.log'.
01-13-2018 16:07:02.350 -0500 INFO WatchedFile - Will begin reading at offset=1556 for file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/splunkd
stderr.log'.
01-13-2018 16:07:02.403 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/btool.log'.
01-13-2018 16:07:02.431 -0500 INFO WatchedFile - Will begin reading at offset=10800 for file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/splunkd-utility.log'.
01-13-2018 16:07:02.437 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/searchhistory.log'.
01-13-2018 16:07:02.440 -0500 INFO WatchedFile - Will begin reading at offset=4740 for file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/conf.log'.
01-13-2018 16:07:02.465 -0500 INFO WatchedFile - Will begin reading at offset=3303363 for file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/metrics.log'.
01-13-2018 16:07:02.468 -0500 INFO WatchedFile - Will begin reading at offset=87350 for file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/audit.log'.
01-13-2018 16:07:02.471 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/licenseusage.log'.
01-13-2018 16:07:02.474 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/remote
searches.log'.
01-13-2018 16:07:02.476 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/scheduler.log'.
01-13-2018 16:07:02.503 -0500 INFO WatchedFile - Will begin reading at offset=2426 for file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/splunkdstdout.log'.
01-13-2018 16:07:02.515 -0500 INFO TcpOutputProc - Connected to idx=MY.INDEXER.I.P:9999, pset=0, reuse=0.
01-13-2018 16:07:14.118 -0500 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not
connected
01-13-2018 16:07:26.119 -0500 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=notconnected
01-13-2018 16:07:33.715 -0500 INFO ProxyConfig - Failed to initialize http
proxy from server.conf for splunkd. Please make sure that the httpproxy property is set as httpproxy=http://host:port in case HTTP proxying needs to be enabled.
01-13-2018 16:07:33.715 -0500 INFO ProxyConfig - Failed to initialize httpsproxy from server.conf for splunkd. Please make sure that the httpsproxy property is set as httpsproxy=http://host:port in case HTTP proxying needs to be enabled.
01-13-2018 16:07:33.715 -0500 INFO ProxyConfig - Failed to initialize the no
proxy setting from server.conf for splunkd. Please provide a valid set of noproxy rules in case HTTP proxying needs to be enabled.
01-13-2018 16:07:33.925 -0500 INFO HttpPubSubConnection - SSL connection with id: connection
MY.FORWARDER.I.P8089MY.FORWARDER.I.PserverB47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:07:34.149 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connectionMY.FORWARDER.I.P8089MY.FORWARDER.I.PserverB47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:07:38.119 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection
MY.FORWARDER.I.P8089MY.FORWARDER.I.PserverB47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:07:38.176 -0500 INFO DC:HandshakeReplyHandler - Handshake done.
01-13-2018 16:08:38.176 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connectionMY.FORWARDER.I.P8089MY.FORWARDER.I.PserverB47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:08:38.372 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection
MY.FORWARDER.I.P8089MY.FORWARDER.I.PserverB47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:09:38.419 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connectionMY.FORWARDER.I.P8089MY.FORWARDER.I.PserverB47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:10:38.615 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection
MY.FORWARDER.I.P8089MY.FORWARDER.I.PserverB47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:11:38.908 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connectionMY.FORWARDER.I.P8089MY.FORWARDER.I.PserverB47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:12:39.109 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection
MY.FORWARDER.I.P8089MY.FORWARDER.I.PserverB47A56B6-7904-4954-98AE-8D56B372CFCF

I'm a noob to splunk and am not sure what else to do, I've followed the steps in the documentation.

Any ideas ?? Thanks ahead of time for your help.

0 Karma
Highlighted

Re: Splunk not receiving data from forwarder - tried everything in documentation

Engager

Adding my /etc/system/local/outputs.conf:
[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = MY.INDEXER.IP:9999

[tcpout-server://MY.INDEXER.IP:9999]

0 Karma
Highlighted

Re: Splunk not receiving data from forwarder - tried everything in documentation

SplunkTrust
SplunkTrust

Hey looking at splunkd.log

You got a ERROR

01-13-2018 16:07:02.332 -0500 ERROR TailingProcessor - Input stanza path, 'home/admin/web/MYSERVER.net/logs/MYSERVER.net.log' is not absolute. This is a configuration error and may not work / break things. Change this path to an absolute path.

So in your monitor stanza provide full path with the root directory.

If you want to know what is absolute path?
So here is the answer,an absolute path is defined as the specifying the location of a file or directory from the root directory(/). In other words we can say absolute path is a complete path from start of actual filesystem from / directory.

Also enable receiving on the indexer if you have not:

To enable receiving,login on indexer:
Go to Settings » Forwarding and receiving » Receive data » Add new Put 9999 and click save.

0 Karma
Highlighted

Re: Splunk not receiving data from forwarder - tried everything in documentation

Ultra Champion

Please refer to I can't find my data!

0 Karma
Highlighted

Re: Splunk not receiving data from forwarder - tried everything in documentation

SplunkTrust
SplunkTrust

Did you restart the Splunk service after making changes to your conf files?

0 Karma
Highlighted

Re: Splunk not receiving data from forwarder - tried everything in documentation

Splunk Employee
Splunk Employee

In addition to all of these, have you checked internal on your indexer to see if you can see the forwarders internal logs?

index=_internal | stats count by host

That will validate if the UF / Forwarder is connecting, and if the problem is in your inputs. Additionally try oneshot'ing a file from your forwarder and see if you can search it.