Splunk not receiving data from forwarder - tried everything in documentation

[Leavittinc]

Let me preface by saying I've read through multiple threads and tried their recommendations with no luck.

I have a splunk enterprise indexer that is not receiving data from a splunk universal forwarder on a remote server.
When I set it up, it initially sent the data, but since has not updated with new information.

I confirmed that my local box is recieving the connection. There are live established connections between the two over port 9999 (which I set). I confirmed that the firewall rules between here and there are perfectly fine. The connections are happening, but no data is flowing.

I have a data input set up in the indexer and it's enabled.

My inputs.conf in $SPLUNK/etc/apps/search/local :

[monitor:///home/admin/web/MYSERVER/logs/MYSERVER.log]
disabled = false

My inputs.conf in $SPLUNK/etc/system/local:

[default]
host = MYSERVER.NET

The tail end of my splunkd.log on the forwarder:

01-13-2018 16:07:02.330 -0500 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor:///home/admin/web/MYSERVER.net/logs/MYSERVER.net.log.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Parsing configuration stanza: monitor://home/admin/web/MYSERVER.net/logs/MYSERVER.net.log.
01-13-2018 16:07:02.332 -0500 ERROR TailingProcessor - Input stanza path, 'home/admin/web/MYSERVER.net/logs/MYSERVER.net.log' is not absolute. This is a configuration error and may not work / break things. Change this path to an absolute path.
01-13-2018 16:07:02.332 -0500 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
01-13-2018 16:07:02.332 -0500 INFO TailReader - State transitioning from 1 to 0 (initOrResume).
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Adding watch on path: /home/admin/web/MYSERVER.net/logs/MYSERVER.net.log.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Adding watch on path: /home/admin/web/MYSERVER.net/splunkforwarder/etc/splunk.version.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Adding watch on path: /home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk.
01-13-2018 16:07:02.332 -0500 INFO TailingProcessor - Adding watch on path: /home/admin/web/MYSERVER.net/splunkforwarder/var/spool/splunk.
01-13-2018 16:07:02.333 -0500 INFO loader - Limiting REST HTTP server to 21845 sockets
01-13-2018 16:07:02.333 -0500 INFO loader - Limiting REST HTTP server to 170 threads
01-13-2018 16:07:02.333 -0500 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see:
01-13-2018 16:07:02.343 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/mongod.log'.
01-13-2018 16:07:02.345 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/splunkd_ui_access.log'.
01-13-2018 16:07:02.346 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/license_usage_summary.log'.
01-13-2018 16:07:02.350 -0500 INFO WatchedFile - Will begin reading at offset=1556 for file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/splunkd_stderr.log'.
01-13-2018 16:07:02.403 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/btool.log'.
01-13-2018 16:07:02.431 -0500 INFO WatchedFile - Will begin reading at offset=10800 for file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/splunkd-utility.log'.
01-13-2018 16:07:02.437 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/searchhistory.log'.
01-13-2018 16:07:02.440 -0500 INFO WatchedFile - Will begin reading at offset=4740 for file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/conf.log'.
01-13-2018 16:07:02.465 -0500 INFO WatchedFile - Will begin reading at offset=3303363 for file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/metrics.log'.
01-13-2018 16:07:02.468 -0500 INFO WatchedFile - Will begin reading at offset=87350 for file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/audit.log'.
01-13-2018 16:07:02.471 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/license_usage.log'.
01-13-2018 16:07:02.474 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/remote_searches.log'.
01-13-2018 16:07:02.476 -0500 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/scheduler.log'.
01-13-2018 16:07:02.503 -0500 INFO WatchedFile - Will begin reading at offset=2426 for file='/home/admin/web/MYSERVER.net/splunkforwarder/var/log/splunk/splunkd_stdout.log'.
01-13-2018 16:07:02.515 -0500 INFO TcpOutputProc - Connected to idx=MY.INDEXER.I.P:9999, pset=0, reuse=0.
01-13-2018 16:07:14.118 -0500 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
01-13-2018 16:07:26.119 -0500 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
01-13-2018 16:07:33.715 -0500 INFO ProxyConfig - Failed to initialize http_proxy from server.conf for splunkd. Please make sure that the http_proxy property is set as http_proxy=http://host:port in case HTTP proxying needs to be enabled.
01-13-2018 16:07:33.715 -0500 INFO ProxyConfig - Failed to initialize https_proxy from server.conf for splunkd. Please make sure that the https_proxy property is set as https_proxy=http://host:port in case HTTP proxying needs to be enabled.
01-13-2018 16:07:33.715 -0500 INFO ProxyConfig - Failed to initialize the no_proxy setting from server.conf for splunkd. Please provide a valid set of no_proxy rules in case HTTP proxying needs to be enabled.
01-13-2018 16:07:33.925 -0500 INFO HttpPubSubConnection - SSL connection with id: connection_MY.FORWARDER.I.P_8089_MY.FORWARDER.I.P_server_B47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:07:34.149 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_MY.FORWARDER.I.P_8089_MY.FORWARDER.I.P_server_B47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:07:38.119 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_MY.FORWARDER.I.P_8089_MY.FORWARDER.I.P_server_B47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:07:38.176 -0500 INFO DC:HandshakeReplyHandler - Handshake done.
01-13-2018 16:08:38.176 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_MY.FORWARDER.I.P_8089_MY.FORWARDER.I.P_server_B47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:08:38.372 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_MY.FORWARDER.I.P_8089_MY.FORWARDER.I.P_server_B47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:09:38.419 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_MY.FORWARDER.I.P_8089_MY.FORWARDER.I.P_server_B47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:10:38.615 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_MY.FORWARDER.I.P_8089_MY.FORWARDER.I.P_server_B47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:11:38.908 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_MY.FORWARDER.I.P_8089_MY.FORWARDER.I.P_server_B47A56B6-7904-4954-98AE-8D56B372CFCF
01-13-2018 16:12:39.109 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_MY.FORWARDER.I.P_8089_MY.FORWARDER.I.P_server_B47A56B6-7904-4954-98AE-8D56B372CFCF

I'm a noob to splunk and am not sure what else to do, I've followed the steps in the documentation.

Any ideas ?? Thanks ahead of time for your help.

[esix_splunk]

In addition to all of these, have you checked internal on your indexer to see if you can see the forwarders internal logs?

index=_internal | stats count by host

That will validate if the UF / Forwarder is connecting, and if the problem is in your inputs. Additionally try oneshot'ing a file from your forwarder and see if you can search it.

[skoelpin]

Did you restart the Splunk service after making changes to your conf files?

[ddrillic]

Please refer to I can't find my data!

[mayurr98]

Hey looking at splunkd.log

You got a ERROR

01-13-2018 16:07:02.332 -0500 ERROR TailingProcessor - Input stanza path, 'home/admin/web/MYSERVER.net/logs/MYSERVER.net.log' is not absolute. This is a configuration error and may not work / break things. Change this path to an absolute path.

So in your monitor stanza provide full path with the root directory.

If you want to know what is absolute path?
So here is the answer,an absolute path is defined as the specifying the location of a file or directory from the root directory(/). In other words we can say absolute path is a complete path from start of actual filesystem from / directory.

Also enable receiving on the indexer if you have not:

To enable receiving,login on indexer:
Go to Settings » Forwarding and receiving » Receive data » Add new Put 9999 and click save.

[Leavittinc]

Adding my /etc/system/local/outputs.conf:
[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = MY.INDEXER.IP:9999

[tcpout-server://MY.INDEXER.IP:9999]