Posting on behalf of someone.
I want to setup a Splunk clustered environment with 4SH (cluster), 4IDX (cluster), FWD deployed on App box across 2 data centers, But as of now I am doing some testing with following configurations. I am new to Splunk, Can someone help please?
My configuration
1 forwarder
2 indexer
2 search heads
Forwarder config
The config files on forwarder are as below
cat inputs.conf
[monitor:////var/logs/myserver.log]
disabled = false
sourcetype = mysourcetye
index=myindex
outputs.conf
[tcpout:xxxx]
server=server1.com:9997,server2:9997
autoLB = true
autoLBFrequency = 300
forceTimebasedAutoLB = true
useACK = true
Indexer config
On indexer, the inputs.conf is in /opt/splunk/etc/apps/myapp/local
cat inputs.conf
[splunktcp://9997]
disabled = 1
The server.conf in /opt/splunk/etc/system/local location has following stanza
[general]
pass4SymmKey = $1$xxxxxxx
serverName = myserver.com
[clustering]
master_uri = https://myclustermaster.com:8089
mode = slave
[license]
master_uri = https://mylicensemaster.com:8089
Forwarder error
I am seeing following error in forwarder splunkd.log
07-14-2016 11:58:09.776 +0100 INFO WatchedFile - Will begin reading at offset=966525 for file='/var/xxx/logs/jetty/jetty.log'.
07-14-2016 11:58:09.794 +0100 INFO WatchedFile - Will begin reading at offset=316928 for file='/opt/splunkforwarder/var/log/splunk/metrics.log'.
07-14-2016 11:58:09.968 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_stdout.log'.
07-14-2016 11:58:09.969 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/mongod.log'.
07-14-2016 11:58:09.971 +0100 INFO WatchedFile - Will begin reading at offset=9129 for file='/opt/splunkforwarder/var/log/splunk/splunkd-utility.log'.
07-14-2016 11:58:09.974 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage.log'.
07-14-2016 11:58:09.976 +0100 INFO WatchedFile - Will begin reading at offset=3230 for file='/opt/splunkforwarder/var/log/splunk/conf.log'.
07-14-2016 11:58:09.978 +0100 INFO WatchedFile - Will begin reading at offset=1230 for file='/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log'.
07-14-2016 11:58:10.004 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/remote_searches.log'.
07-14-2016 11:58:10.006 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/scheduler.log'.
07-14-2016 11:58:10.010 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log'.
07-14-2016 11:58:10.045 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/searchhistory.log'.
07-14-2016 11:58:10.048 +0100 INFO WatchedFile - Will begin reading at offset=68593 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.
07-14-2016 11:58:29.697 +0100 WARN TcpOutputProc - Cooked connection to ip=Inderxer1:9997 timed out
07-14-2016 11:58:49.697 +0100 WARN TcpOutputProc - Cooked connection to ip=indexer2:9997 timed out
Hi vinitatsky, I believe the issue is that you set splunktcp://9997 to disabled on your indexer. Try
[splunktcp://9997]
disabled = 0
Please let me know if this answers your question! 😄
Hi vinitatsky, I believe the issue is that you set splunktcp://9997 to disabled on your indexer. Try
[splunktcp://9997]
disabled = 0
Please let me know if this answers your question! 😄
Thanks @muebel
It was an issue with our index configuration and we managed to solve the issue.
Thanks for your quick response..!!
Thanks for prompt reply..!!
On indexer, myapp was in two location and the inputs.conf in first location had disabled = 1
1. /opt/splunk/etc/apps/myapp/local
2. /opt/splunk/etc/slave-apps/myapp/local
cd /opt/splunk/etc/apps/myapp/local
cat inputs.conf
[splunktcp://9997]
disabled = 1
cd /opt/splunk/etc/slave-apps/myapp/local
cat inputs.conf
[splunktcp://9997]
Removed the first location app, restarted indexers and it worked!!
Can you try to telnet <indexer> 9997
from the forwarder?
Telnet is working fine
yes, I can
About the Cooked connection at TcpOutputProc - Cooked connection to ip=x.x.x.x:9997 timed out
thanks. We managed to solve it by modifying indexer configuration as suggested by muebel