Deployment Architecture

Splunk indexer is not able to get the data from the splunk forwarder

samarkumar
Path Finder

Hi

I am trying to install splunk enterprise trial version in our infrastructure for the evaluation purpose.

I configured splunk enterprise in one of dev linux box and forwarder in another dev linux box. Trying to forward the data through forwarder.

When i see the log in splunk forwarder linux box it seems to me it is connected to Splunk enterprise server

Checked the log: /opt/app/test/splunkforwarder/var/log/splunk/splunkd.log | grep TcpOutputProc

11-09-2017 15:08:12.538 -0700 INFO TcpOutputProc - Connected to idx=001.001.001.95:9997, pset=0, reuse=0.

Note: IP 001.001.001.95 is changed, as it is for splunk enterprise server.

When i check the Splunk enterprise for port 9997, i see below details

[XXXXXXXX ~]$ netstat -anp|grep 9997
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:9997 0.0.0.0:* LISTEN 1893/splunkd
tcp 0 0 001.001.001.95:9997 001.001.001.02:30189 ESTABLISHED 1893/splunkd
tcp 0 0 001.001.001.95:9997 001.001.001.02:54039 ESTABLISHED 1893/splunkd

ESTABLISHED connection is showing for my splunk forwarder linux server - not sure if it is correct

My Splunk forwarder configuration is as below:

=====splunkforwarder/etc/system/local/inputs.conf

[default]
host = n01aplXXX.XXX.YY.PPPP.GGG
[splunktcp://9997]
connection_host = none
[monitor:///opt/app/test/testlog/testLog.log]
sourcetype = 3dev1
disabled = 0
index= np_test

=====splunkforwarder/etc/system/local/outputs.conf --- 001.001.001.95 is splunk enterprise server IP

defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
disabled = false
server = 001.001.001.95:9997
[tcpout-server:// 001.001.001.95:9997]

===== Splunk enterprise server configuration
added Forwarding and receiving » Receive data -- port as 9997

under Forwarder Management i am able to see my splunk forwarder clients details

I am seeing data from splunk forwarder, please suggest what i am missing.

0 Karma

samarkumar
Path Finder

Thanks for your input.

When i ran index=_internal | stats count by host i am able to see splunk forwarder following data, but i am not seeing the data which i am monitoring: [monitor:///opt/app/test/testlog/testLog.log]

  • splunkd
  • splunkd_access
  • splunk_btool
  • splunkd_stdout-too_small
  • splunkd_stderr
  • splunkd_conf

further i changed inputs.conf as below:

[monitor:///opt/app/test/testlog/cccmLog.log]
index= np_test
sourcetype = 3dev1

===
my output.conf look as below:

====
[tcpout]
defaultGroup = splunk
[tcpout:splunk]

server = server ip as (xx.xx.xx.xx):9997

i checked splunkd.log and find following :

kforwarder/var/log/splunk/license_usage.log'.
11-10-2017 12:13:29.117 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/test/splunkforwarder/var/log/splunk/mongod.log'.
11-10-2017 12:13:29.125 -0700 INFO WatchedFile - Will begin reading at offset=2440 for file='/opt/app/test/splunkforwarder/var/log/splunk/splunkd_stderr.log'.
11-10-2017 12:13:29.128 -0700 INFO TcpOutputProc - Connected to idx=xx.xx.xx.xx:9997, pset=0, reuse=0.
11-10-2017 12:13:29.128 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/test/splunkforwarder/var/log/splunk/searchhistory.log'.
11-10-2017 12:13:29.129 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/test/splunkforwarder/var/log/splunk/scheduler.log'.
11-10-2017 12:13:29.144 -0700 INFO WatchedFile - Will begin reading at offset=7590331 for file='/opt/app/test/splunkforwarder/var/log/splunk/metrics.log'.
11-10-2017 12:13:29.161 -0700 INFO WatchedFile - Will begin reading at offset=80220 for file='/opt/app/test/splunkforwarder/var/log/splunk/audit.log'.
11-10-2017 12:13:29.163 -0700 INFO WatchedFile - Will begin reading at offset=17159 for file='/opt/app/test/splunkforwarder/var/log/splunk/splunkd-utility.log'.
11-10-2017 12:13:29.166 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/test/splunkforwarder/var/log/splunk/remote_searches.log'.
11-10-2017 12:13:29.169 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/test/splunkforwarder/var/log/splunk/splunkd_ui_access.log'.
11-10-2017 12:13:29.172 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/test/splunkforwarder/var/log/splunk/license_usage_summary.log'.
11-10-2017 12:13:29.184 -0700 INFO WatchedFile - Will begin reading at offset=7000 for file='/opt/app/test/splunkforwarder/var/log/splunk/conf.log'.
11-10-2017 12:13:40.933 -0700 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-10-2017 12:13:52.934 -0700 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-10-2017 12:14:04.934 -0700 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-10-2017 12:14:14.693 -0700 INFO ProxyConfig - Failed to initialize http_proxy from server.conf for splunkd. Please make sure that the http_proxy property is set as http_proxy=http://host:port in case HTTP proxying needs to be enabled.
11-10-2017 12:14:14.693 -0700 INFO ProxyConfig - Failed to initialize https_proxy from server.conf for splunkd. Please make sure that the https_proxy property is set as https_proxy=http://host:port in case HTTP proxying needs to be enabled.
11-10-2017 12:14:14.693 -0700 INFO ProxyConfig - Failed to initialize the no_proxy setting from server.conf for splunkd. Please provide a valid set of no_proxy rules in case HTTP proxying needs to be enabled.

do i need to change any thing in server.conf ??

0 Karma

mayurr98
Super Champion

Remove this setting from inputs.conf forwarder
[default]
host = n01aplXXX.XXX.YY.PPPP.GGG
[splunktcp://9997]
connection_host = none

Just write
[monitor:///opt/app/test/testlog/testLog.log]
index= np_test
sourcetype = 3dev1

also, have you configured forwarder correctly?
in order to send the data to indexer
you need to run
cd /opt/splunk/bin
./splunk add forward-server index_ip:9997

refer this doc for forwarder configuration: http://docs.splunk.com/Documentation/Forwarder/6.5.3/Forwarder/Configuretheuniversalforwarder

refer below doc for how to forward data to indexer:
http://docs.splunk.com/Documentation/Forwarder/7.0.0/Forwarder/HowtoforwarddatatoSplunkEnterprise

also, you can check for forwarder logs to troubleshoot
on the forwarder go to vi /opt/splunkforwarder/var/log/splunk/splunkd.log

check for this link it might help you:
https://answers.splunk.com/answers/465/ive-set-up-a-forwarder-but-im-not-receving-any-events-on-the-...

Let me know if it works!

0 Karma

samarkumar
Path Finder

Hi
i did ./splunk list forward-server and found that active forwards to indexer Ip and Port

indexer IP:9997

Configured but inactive forwards:
None

i checked the url https://answers.splunk.com/answers/465/ive-set-up-a-forwarder-but-im-not-receving-any-events-on-the-...

and found that using index=_internal source=*metrics.log tcpin_connections

in splunk logs as below:

=====
11-10-2017 16:44:41.999 -0700 INFO Metrics - group=tcpin_connections, 1XX.XX8.X4.X0:47185:9997, connectionType=cooked, sourcePort=47185, sourceHost=1XX.XX8.X4.X0, sourceIp=1XX.XX8.X4.X0, destPort=9997, kb=7.93, _tcp_Bps=261.99, _tcp_KBps=0.26, _tcp_avg_thruput=0.27, _tcp_Kprocessed=2271.78, _tcp_eps=0.19, _process_time_ms=0, evt_misc_kBps=0.00, evt_raw_kBps=0.23, evt_fields_kBps=0.00, evt_fn_kBps=0.00, evt_fv_kBps=0.00, evt_fn_str_kBps=0.00, evt_fn_meta_dyn_kBps=0.00, evt_fn_meta_predef_kBps=0.00, evt_fn_meta_str_kBps=0.00, evt_fv_num_kBps=0.00, evt_fv_str_kBps=0.00, evt_fv_predef_kBps=0.00, evt_fv_offlen_kBps=0.00, evt_fv_fp_kBps=0.00, build=c8a78efdd40f, version=7.0.0, os=Linux, arch=x86_64, hostname=XXXXXXXXXXXXXXXXXXX, guid=XXXXXXXXXXXXX, fwdType=uf, ssl=false, lastIndexer=XXX.XX.XX.XX:9997, ack=false

====
It looks like every other logs i am getting other than monitoring file which is pointed to index np_test and source type 3dev1. [monitor:///opt/app/test/testlog/testLog.log]

please suggest how to rectify this.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Your last sentence say that you are seeing data from the Splunk Forwarder. So this means you are getting the data... I

index=_internal | stats count by host

Run that search and you should see both your local Splunk server and the remote server...

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...