topic Re: Splunk indexer is not able to get the data from the splunk forwarder in Deployment Architecture

Splunk indexer is not able to get the data from the splunk forwarder

samarkumar — Tue, 29 Sep 2020 16:41:18 GMT

I am trying to install splunk enterprise trial version in our infrastructure for the evaluation purpose.

I configured splunk enterprise in one of dev linux box and forwarder in another dev linux box. Trying to forward the data through forwarder.

When i see the log in splunk forwarder linux box it seems to me it is connected to Splunk enterprise server

Checked the log: /opt/app/test/splunkforwarder/var/log/splunk/splunkd.log | grep TcpOutputProc

11-09-2017 15:08:12.538 -0700 INFO TcpOutputProc - Connected to idx=001.001.001.95:9997, pset=0, reuse=0.

Note: IP 001.001.001.95 is changed, as it is for splunk enterprise server.

When i check the Splunk enterprise for port 9997, i see below details

[XXXXXXXX ~]$ netstat -anp|grep 9997
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:9997 0.0.0.0:* LISTEN 1893/splunkd
tcp 0 0 001.001.001.95:9997 001.001.001.02:30189 ESTABLISHED 1893/splunkd
tcp 0 0 001.001.001.95:9997 001.001.001.02:54039 ESTABLISHED 1893/splunkd

ESTABLISHED connection is showing for my splunk forwarder linux server - not sure if it is correct

My Splunk forwarder configuration is as below:

=====splunkforwarder/etc/system/local/inputs.conf

[default]
host = n01aplXXX.XXX.YY.PPPP.GGG
[splunktcp://9997]
connection_host = none
[monitor:///opt/app/test/testlog/testLog.log]
sourcetype = 3dev1
disabled = 0
index= np_test

=====splunkforwarder/etc/system/local/outputs.conf --- 001.001.001.95 is splunk enterprise server IP

defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
disabled = false
server = 001.001.001.95:9997
[tcpout-server:// 001.001.001.95:9997]

===== Splunk enterprise server configuration
added Forwarding and receiving » Receive data -- port as 9997

under Forwarder Management i am able to see my splunk forwarder clients details

I am seeing data from splunk forwarder, please suggest what i am missing.

Re: Splunk indexer is not able to get the data from the splunk forwarder

esix_splunk — Fri, 10 Nov 2017 06:44:07 GMT

Your last sentence say that you are seeing data from the Splunk Forwarder. So this means you are getting the data... I

index=_internal | stats count by host

Run that search and you should see both your local Splunk server and the remote server...

Re: Splunk indexer is not able to get the data from the splunk forwarder

mayurr98 — Fri, 10 Nov 2017 06:50:21 GMT

Remove this setting from inputs.conf forwarder
[default]
host = n01aplXXX.XXX.YY.PPPP.GGG
[splunktcp://9997]
connection_host = none

Just write
[monitor:///opt/app/test/testlog/testLog.log]
index= np_test
sourcetype = 3dev1

also, have you configured forwarder correctly?
in order to send the data to indexer
you need to run
cd /opt/splunk/bin
./splunk add forward-server index_ip:9997

refer this doc for forwarder configuration: http://docs.splunk.com/Documentation/Forwarder/6.5.3/Forwarder/Configuretheuniversalforwarder

refer below doc for how to forward data to indexer:
http://docs.splunk.com/Documentation/Forwarder/7.0.0/Forwarder/HowtoforwarddatatoSplunkEnterprise

also, you can check for forwarder logs to troubleshoot
on the forwarder go to vi /opt/splunkforwarder/var/log/splunk/splunkd.log

check for this link it might help you:
https://answers.splunk.com/answers/465/ive-set-up-a-forwarder-but-im-not-receving-any-events-on-the-splunk-indexer.html

Let me know if it works!

Re: Splunk indexer is not able to get the data from the splunk forwarder

samarkumar — Tue, 29 Sep 2020 16:42:10 GMT

Thanks for your input.

When i ran index=_internal | stats count by host i am able to see splunk forwarder following data, but i am not seeing the data which i am monitoring: [monitor:///opt/app/test/testlog/testLog.log]

splunkd
splunkd_access
splunk_btool
splunkd_stdout-too_small
splunkd_stderr
splunkd_conf

further i changed inputs.conf as below:

[monitor:///opt/app/test/testlog/cccmLog.log]
index= np_test
sourcetype = 3dev1

===
my output.conf look as below:

====
[tcpout]
defaultGroup = splunk
[tcpout:splunk]

server = server ip as (xx.xx.xx.xx):9997

i checked splunkd.log and find following :

kforwarder/var/log/splunk/license_usage.log'.
11-10-2017 12:13:29.117 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/test/splunkforwarder/var/log/splunk/mongod.log'.
11-10-2017 12:13:29.125 -0700 INFO WatchedFile - Will begin reading at offset=2440 for file='/opt/app/test/splunkforwarder/var/log/splunk/splunkd_stderr.log'.
11-10-2017 12:13:29.128 -0700 INFO TcpOutputProc - Connected to idx=xx.xx.xx.xx:9997, pset=0, reuse=0.
11-10-2017 12:13:29.128 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/test/splunkforwarder/var/log/splunk/searchhistory.log'.
11-10-2017 12:13:29.129 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/test/splunkforwarder/var/log/splunk/scheduler.log'.
11-10-2017 12:13:29.144 -0700 INFO WatchedFile - Will begin reading at offset=7590331 for file='/opt/app/test/splunkforwarder/var/log/splunk/metrics.log'.
11-10-2017 12:13:29.161 -0700 INFO WatchedFile - Will begin reading at offset=80220 for file='/opt/app/test/splunkforwarder/var/log/splunk/audit.log'.
11-10-2017 12:13:29.163 -0700 INFO WatchedFile - Will begin reading at offset=17159 for file='/opt/app/test/splunkforwarder/var/log/splunk/splunkd-utility.log'.
11-10-2017 12:13:29.166 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/test/splunkforwarder/var/log/splunk/remote_searches.log'.
11-10-2017 12:13:29.169 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/test/splunkforwarder/var/log/splunk/splunkd_ui_access.log'.
11-10-2017 12:13:29.172 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/test/splunkforwarder/var/log/splunk/license_usage_summary.log'.
11-10-2017 12:13:29.184 -0700 INFO WatchedFile - Will begin reading at offset=7000 for file='/opt/app/test/splunkforwarder/var/log/splunk/conf.log'.
11-10-2017 12:13:40.933 -0700 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-10-2017 12:13:52.934 -0700 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-10-2017 12:14:04.934 -0700 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-10-2017 12:14:14.693 -0700 INFO ProxyConfig - Failed to initialize http_proxy from server.conf for splunkd. Please make sure that the http_proxy property is set as http_proxy=http://host:port in case HTTP proxying needs to be enabled.
11-10-2017 12:14:14.693 -0700 INFO ProxyConfig - Failed to initialize https_proxy from server.conf for splunkd. Please make sure that the https_proxy property is set as https_proxy=http://host:port in case HTTP proxying needs to be enabled.
11-10-2017 12:14:14.693 -0700 INFO ProxyConfig - Failed to initialize the no_proxy setting from server.conf for splunkd. Please provide a valid set of no_proxy rules in case HTTP proxying needs to be enabled.

do i need to change any thing in server.conf ??

Re: Splunk indexer is not able to get the data from the splunk forwarder

samarkumar — Tue, 29 Sep 2020 16:42:15 GMT

Hi
i did ./splunk list forward-server and found that active forwards to indexer Ip and Port

indexer IP:9997

Configured but inactive forwards:
None

i checked the url https://answers.splunk.com/answers/465/ive-set-up-a-forwarder-but-im-not-receving-any-events-on-the-splunk-indexer.html

and found that using index=_internal source=*metrics.log tcpin_connections

in splunk logs as below:

=====
11-10-2017 16:44:41.999 -0700 INFO Metrics - group=tcpin_connections, 1XX.XX8.X4.X0:47185:9997, connectionType=cooked, sourcePort=47185, sourceHost=1XX.XX8.X4.X0, sourceIp=1XX.XX8.X4.X0, destPort=9997, kb=7.93, _tcp_Bps=261.99, _tcp_KBps=0.26, _tcp_avg_thruput=0.27, _tcp_Kprocessed=2271.78, _tcp_eps=0.19, _process_time_ms=0, evt_misc_kBps=0.00, evt_raw_kBps=0.23, evt_fields_kBps=0.00, evt_fn_kBps=0.00, evt_fv_kBps=0.00, evt_fn_str_kBps=0.00, evt_fn_meta_dyn_kBps=0.00, evt_fn_meta_predef_kBps=0.00, evt_fn_meta_str_kBps=0.00, evt_fv_num_kBps=0.00, evt_fv_str_kBps=0.00, evt_fv_predef_kBps=0.00, evt_fv_offlen_kBps=0.00, evt_fv_fp_kBps=0.00, build=c8a78efdd40f, version=7.0.0, os=Linux, arch=x86_64, hostname=XXXXXXXXXXXXXXXXXXX, guid=XXXXXXXXXXXXX, fwdType=uf, ssl=false, lastIndexer=XXX.XX.XX.XX:9997, ack=false

====
It looks like every other logs i am getting other than monitoring file which is pointed to index np_test and source type 3dev1. [monitor:///opt/app/test/testlog/testLog.log]

please suggest how to rectify this.