Deployment Architecture
Highlighted

Splunk indexer is not able to get the data from the splunk forwarder

Path Finder

Hi

I am trying to install splunk enterprise trial version in our infrastructure for the evaluation purpose.

I configured splunk enterprise in one of dev linux box and forwarder in another dev linux box. Trying to forward the data through forwarder.

When i see the log in splunk forwarder linux box it seems to me it is connected to Splunk enterprise server

Checked the log: /opt/app/test/splunkforwarder/var/log/splunk/splunkd.log | grep TcpOutputProc

11-09-2017 15:08:12.538 -0700 INFO TcpOutputProc - Connected to idx=001.001.001.95:9997, pset=0, reuse=0.

Note: IP 001.001.001.95 is changed, as it is for splunk enterprise server.

When i check the Splunk enterprise for port 9997, i see below details

[XXXXXXXX ~]$ netstat -anp|grep 9997
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:9997 0.0.0.0:* LISTEN 1893/splunkd
tcp 0 0 001.001.001.95:9997 001.001.001.02:30189 ESTABLISHED 1893/splunkd
tcp 0 0 001.001.001.95:9997 001.001.001.02:54039 ESTABLISHED 1893/splunkd

ESTABLISHED connection is showing for my splunk forwarder linux server - not sure if it is correct

My Splunk forwarder configuration is as below:

=====splunkforwarder/etc/system/local/inputs.conf

[default]
host = n01aplXXX.XXX.YY.PPPP.GGG
[splunktcp://9997]
connectionhost = none
[monitor:///opt/app/test/testlog/testLog.log]
sourcetype = 3dev1
disabled = 0
index= np
test

=====splunkforwarder/etc/system/local/outputs.conf --- 001.001.001.95 is splunk enterprise server IP

defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
disabled = false
server = 001.001.001.95:9997
[tcpout-server:// 001.001.001.95:9997]

===== Splunk enterprise server configuration
added Forwarding and receiving » Receive data -- port as 9997

under Forwarder Management i am able to see my splunk forwarder clients details

I am seeing data from splunk forwarder, please suggest what i am missing.

0 Karma
Highlighted

Re: Splunk indexer is not able to get the data from the splunk forwarder

Splunk Employee
Splunk Employee

Your last sentence say that you are seeing data from the Splunk Forwarder. So this means you are getting the data... I

index=_internal | stats count by host

Run that search and you should see both your local Splunk server and the remote server...

0 Karma
Highlighted

Re: Splunk indexer is not able to get the data from the splunk forwarder

SplunkTrust
SplunkTrust

Remove this setting from inputs.conf forwarder
[default]
host = n01aplXXX.XXX.YY.PPPP.GGG
[splunktcp://9997]
connection_host = none

Just write
[monitor:///opt/app/test/testlog/testLog.log]
index= np_test
sourcetype = 3dev1

also, have you configured forwarder correctly?
in order to send the data to indexer
you need to run
cd /opt/splunk/bin
./splunk add forward-server index_ip:9997

refer this doc for forwarder configuration: http://docs.splunk.com/Documentation/Forwarder/6.5.3/Forwarder/Configuretheuniversalforwarder

refer below doc for how to forward data to indexer:
http://docs.splunk.com/Documentation/Forwarder/7.0.0/Forwarder/HowtoforwarddatatoSplunkEnterprise

also, you can check for forwarder logs to troubleshoot
on the forwarder go to vi /opt/splunkforwarder/var/log/splunk/splunkd.log

check for this link it might help you:
https://answers.splunk.com/answers/465/ive-set-up-a-forwarder-but-im-not-receving-any-events-on-the-...

Let me know if it works!

0 Karma
Highlighted

Re: Splunk indexer is not able to get the data from the splunk forwarder

Path Finder

Hi
i did ./splunk list forward-server and found that active forwards to indexer Ip and Port

indexer IP:9997

Configured but inactive forwards:
None

i checked the url https://answers.splunk.com/answers/465/ive-set-up-a-forwarder-but-im-not-receving-any-events-on-the-...

and found that using index=internal source=*metrics.log tcpinconnections

in splunk logs as below:

=====
11-10-2017 16:44:41.999 -0700 INFO Metrics - group=tcpinconnections, 1XX.XX8.X4.X0:47185:9997, connectionType=cooked, sourcePort=47185, sourceHost=1XX.XX8.X4.X0, sourceIp=1XX.XX8.X4.X0, destPort=9997, kb=7.93, _tcpBps=261.99, tcpKBps=0.26, tcpavgthruput=0.27, _tcpKprocessed=2271.78, tcpeps=0.19, processtimems=0, evtmisckBps=0.00, evtrawkBps=0.23, evtfieldskBps=0.00, evtfnkBps=0.00, evtfvkBps=0.00, evtfnstrkBps=0.00, evtfnmetadynkBps=0.00, evtfnmetapredefkBps=0.00, evtfnmetastrkBps=0.00, evtfvnumkBps=0.00, evtfvstrkBps=0.00, evtfvpredefkBps=0.00, evtfvofflenkBps=0.00, evtfvfpkBps=0.00, build=c8a78efdd40f, version=7.0.0, os=Linux, arch=x8664, hostname=XXXXXXXXXXXXXXXXXXX, guid=XXXXXXXXXXXXX, fwdType=uf, ssl=false, lastIndexer=XXX.XX.XX.XX:9997, ack=false

====
It looks like every other logs i am getting other than monitoring file which is pointed to index np_test and source type 3dev1. [monitor:///opt/app/test/testlog/testLog.log]

please suggest how to rectify this.

0 Karma
Highlighted

Re: Splunk indexer is not able to get the data from the splunk forwarder

Path Finder

Thanks for your input.

When i ran index=_internal | stats count by host i am able to see splunk forwarder following data, but i am not seeing the data which i am monitoring: [monitor:///opt/app/test/testlog/testLog.log]

  • splunkd
  • splunkd_access
  • splunk_btool
  • splunkdstdout-toosmall
  • splunkd_stderr
  • splunkd_conf

further i changed inputs.conf as below:

[monitor:///opt/app/test/testlog/cccmLog.log]
index= np_test
sourcetype = 3dev1

===
my output.conf look as below:

====
[tcpout]
defaultGroup = splunk
[tcpout:splunk]

server = server ip as (xx.xx.xx.xx):9997

i checked splunkd.log and find following :

kforwarder/var/log/splunk/licenseusage.log'.
11-10-2017 12:13:29.117 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/test/splunkforwarder/var/log/splunk/mongod.log'.
11-10-2017 12:13:29.125 -0700 INFO WatchedFile - Will begin reading at offset=2440 for file='/opt/app/test/splunkforwarder/var/log/splunk/splunkd
stderr.log'.
11-10-2017 12:13:29.128 -0700 INFO TcpOutputProc - Connected to idx=xx.xx.xx.xx:9997, pset=0, reuse=0.
11-10-2017 12:13:29.128 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/test/splunkforwarder/var/log/splunk/searchhistory.log'.
11-10-2017 12:13:29.129 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/test/splunkforwarder/var/log/splunk/scheduler.log'.
11-10-2017 12:13:29.144 -0700 INFO WatchedFile - Will begin reading at offset=7590331 for file='/opt/app/test/splunkforwarder/var/log/splunk/metrics.log'.
11-10-2017 12:13:29.161 -0700 INFO WatchedFile - Will begin reading at offset=80220 for file='/opt/app/test/splunkforwarder/var/log/splunk/audit.log'.
11-10-2017 12:13:29.163 -0700 INFO WatchedFile - Will begin reading at offset=17159 for file='/opt/app/test/splunkforwarder/var/log/splunk/splunkd-utility.log'.
11-10-2017 12:13:29.166 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/test/splunkforwarder/var/log/splunk/remotesearches.log'.
11-10-2017 12:13:29.169 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/test/splunkforwarder/var/log/splunk/splunkd
uiaccess.log'.
11-10-2017 12:13:29.172 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/app/test/splunkforwarder/var/log/splunk/license
usagesummary.log'.
11-10-2017 12:13:29.184 -0700 INFO WatchedFile - Will begin reading at offset=7000 for file='/opt/app/test/splunkforwarder/var/log/splunk/conf.log'.
11-10-2017 12:13:40.933 -0700 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not
connected
11-10-2017 12:13:52.934 -0700 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=notconnected
11-10-2017 12:14:04.934 -0700 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not
connected
11-10-2017 12:14:14.693 -0700 INFO ProxyConfig - Failed to initialize httpproxy from server.conf for splunkd. Please make sure that the httpproxy property is set as httpproxy=http://host:port in case HTTP proxying needs to be enabled.
11-10-2017 12:14:14.693 -0700 INFO ProxyConfig - Failed to initialize https
proxy from server.conf for splunkd. Please make sure that the httpsproxy property is set as httpsproxy=http://host:port in case HTTP proxying needs to be enabled.
11-10-2017 12:14:14.693 -0700 INFO ProxyConfig - Failed to initialize the noproxy setting from server.conf for splunkd. Please provide a valid set of noproxy rules in case HTTP proxying needs to be enabled.

do i need to change any thing in server.conf ??

0 Karma