Splunk not receiving data from forwarder - tried everything in documentation

Leavittinc — Tue, 29 Sep 2020 17:43:04 GMT

Let me preface by saying I've read through multiple threads and tried their recommendations with no luck.

I have a splunk enterprise indexer that is not receiving data from a splunk universal forwarder on a remote server.
When I set it up, it initially sent the data, but since has not updated with new information.

I confirmed that my local box is recieving the connection. There are live established connections between the two over port 9999 (which I set). I confirmed that the firewall rules between here and there are perfectly fine. The connections are happening, but no data is flowing.

I have a data input set up in the indexer and it's enabled.

My inputs.conf in $SPLUNK/etc/apps/search/local :

[monitor:///home/admin/web/MYSERVER/logs/MYSERVER.log]
disabled = false

My inputs.conf in $SPLUNK/etc/system/local:

[default]
host = MYSERVER.NET

The tail end of my splunkd.log on the forwarder:

01-13-2018 16:07:02.330 -0500 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor:///home/admin/web/MYSERVER.net/logs/MYSERVER.net.log.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://home/admin/web/MYSERVER.net/logs/MYSERVER.net.log.
01-13-2018 16:07:02.332 -0500 ERROR TailingProcessor - Input stanza path, 'home/admin/web/MYSERVER.net/logs/MYSERVER.net.log' is not absolute. This is a configuration error and may not work / break things. Change this path to an absolute path.
01-13-2018 16:07:02.332 -0500 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
01-13-2018 16:07:02.332 -0500 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Adding watch on path: /home/admin/web/MYSERVER.net/logs/MYSERVER.net.log.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Adding watch on path: /home/admin/web/MYSERVER.net/splunkforwarder/etc/splunk.version.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Adding watch on path: /home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Adding watch on path: /home/admin/web/MYSERVER.net/splunkforwarder/var/spool/splunk.
01-13-2018 16:07:02.333 -0500 INFO loader - Limiting REST HTTP server to 21845 sockets
01-13-2018 16:07:02.333 -0500 INFO loader - Limiting REST HTTP server to 170 threads
01-13-2018 16:07:02.333 -0500 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see:
01-13-2018 16:07:02.343 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/mongod.log'.
01-13-2018 16:07:02.345 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/splunkd_ui_access.log'.
01-13-2018 16:07:02.346 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/license_usage_summary.log'.
01-13-2018 16:07:02.350 -0500 INFO WatchedFile - Will begin reading at offset=1556 for file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/splunkd_stderr.log'.
01-13-2018 16:07:02.403 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/btool.log'.
01-13-2018 16:07:02.431 -0500 INFO WatchedFile - Will begin reading at offset=10800 for file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/splunkd-utility.log'.
01-13-2018 16:07:02.437 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/searchhistory.log'.
01-13-2018 16:07:02.440 -0500 INFO WatchedFile - Will begin reading at offset=4740 for file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/conf.log'.
01-13-2018 16:07:02.465 -0500 INFO WatchedFile - Will begin reading at offset=3303363 for file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/metrics.log'.
01-13-2018 16:07:02.468 -0500 INFO WatchedFile - Will begin reading at offset=87350 for file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/audit.log'.
01-13-2018 16:07:02.471 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/license_usage.log'.
01-13-2018 16:07:02.474 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/remote_searches.log'.
01-13-2018 16:07:02.476 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/scheduler.log'.
01-13-2018 16:07:02.503 -0500 INFO WatchedFile - Will begin reading at offset=2426 for file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/splunkd_stdout.log'.
01-13-2018 16:07:02.515 -0500 INFO TcpOutputProc - Connected to idx=MY.INDEXER.I.P:9999, pset=0, reuse=0.
01-13-2018 16:07:14.118 -0500 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
01-13-2018 16:07:26.119 -0500 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
01-13-2018 16:07:33.715 -0500 INFO ProxyConfig - Failed to initialize http_proxy from server.conf for splunkd. Please make sure that the http_proxy property is set as http_proxy=http://host:port in case HTTP proxying needs to be enabled.
01-13-2018 16:07:33.715 -0500 INFO ProxyConfig - Failed to initialize https_proxy from server.conf for splunkd. Please make sure that the https_proxy property is set as https_proxy=http://host:port in case HTTP proxying needs to be enabled.
01-13-2018 16:07:33.715 -0500 INFO ProxyConfig - Failed to initialize the no_proxy setting from server.conf for splunkd. Please provide a valid set of no_proxy rules in case HTTP proxying needs to be enabled.
01-13-2018 16:07:33.925 -0500 INFO HttpPubSubConnection - SSL connection with id: connection_MY.FORWARDER.I.P_8089_MY.FORWARDER.I.P_server_B47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:07:34.149 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_MY.FORWARDER.I.P_8089_MY.FORWARDER.I.P_server_B47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:07:38.119 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_MY.FORWARDER.I.P_8089_MY.FORWARDER.I.P_server_B47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:07:38.176 -0500 INFO DC:HandshakeReplyHandler - Handshake done.
01-13-2018 16:08:38.176 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_MY.FORWARDER.I.P_8089_MY.FORWARDER.I.P_server_B47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:08:38.372 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_MY.FORWARDER.I.P_8089_MY.FORWARDER.I.P_server_B47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:09:38.419 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_MY.FORWARDER.I.P_8089_MY.FORWARDER.I.P_server_B47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:10:38.615 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_MY.FORWARDER.I.P_8089_MY.FORWARDER.I.P_server_B47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:11:38.908 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_MY.FORWARDER.I.P_8089_MY.FORWARDER.I.P_server_B47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:12:39.109 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_MY.FORWARDER.I.P_8089_MY.FORWARDER.I.P_server_B47A56B6-7904-4954-98AE-8D56B372CFCF

I'm a noob to splunk and am not sure what else to do, I've followed the steps in the documentation.

Any ideas ?? Thanks ahead of time for your help.

Re: Splunk not receiving data from forwarder - tried everything in documentation

Leavittinc — Sat, 13 Jan 2018 22:25:05 GMT

Adding my /etc/system/local/outputs.conf:
[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = MY.INDEXER.IP:9999

[tcpout-server://MY.INDEXER.IP:9999]

Re: Splunk not receiving data from forwarder - tried everything in documentation

mayurr98 — Sun, 14 Jan 2018 15:00:05 GMT

Hey looking at splunkd.log

You got a ERROR

01-13-2018 16:07:02.332 -0500 ERROR TailingProcessor - Input stanza path, 'home/admin/web/MYSERVER.net/logs/MYSERVER.net.log' is not absolute. This is a configuration error and may not work / break things. Change this path to an absolute path.

So in your monitor stanza provide full path with the root directory.

If you want to know what is absolute path?
So here is the answer,an absolute path is defined as the specifying the location of a file or directory from the root directory(/). In other words we can say absolute path is a complete path from start of actual filesystem from / directory.

Also enable receiving on the indexer if you have not:

To enable receiving,login on indexer:
Go to Settings » Forwarding and receiving » Receive data » Add new Put 9999 and click save.

Re: Splunk not receiving data from forwarder - tried everything in documentation

ddrillic — Sun, 14 Jan 2018 15:16:03 GMT

Please refer to I can't find my data!

Re: Splunk not receiving data from forwarder - tried everything in documentation

skoelpin — Sun, 14 Jan 2018 16:54:10 GMT

Did you restart the Splunk service after making changes to your conf files?

Re: Splunk not receiving data from forwarder - tried everything in documentation

esix_splunk — Mon, 15 Jan 2018 04:10:57 GMT

In addition to all of these, have you checked internal on your indexer to see if you can see the forwarders internal logs?

index=_internal | stats count by host

That will validate if the UF / Forwarder is connecting, and if the problem is in your inputs. Additionally try oneshot'ing a file from your forwarder and see if you can search it.

topic Re: Splunk not receiving data from forwarder - tried everything in documentation in Getting Data In

Splunk not receiving data from forwarder - tried everything in documentation

Re: Splunk not receiving data from forwarder - tried everything in documentation

Re: Splunk not receiving data from forwarder - tried everything in documentation

Re: Splunk not receiving data from forwarder - tried everything in documentation

Re: Splunk not receiving data from forwarder - tried everything in documentation

Re: Splunk not receiving data from forwarder - tried everything in documentation