Splunk configurations for SH, FWD and INDEXER

[vinitatsky]

Posting on behalf of someone.

I want to setup a Splunk clustered environment with 4SH (cluster), 4IDX (cluster), FWD deployed on App box across 2 data centers, But as of now I am doing some testing with following configurations. I am new to Splunk, Can someone help please?

My configuration
1 forwarder
2 indexer
2 search heads
Forwarder config
The config files on forwarder are as below
cat inputs.conf
[monitor:////var/logs/myserver.log]
disabled = false
sourcetype = mysourcetye
index=myindex

outputs.conf
[tcpout:xxxx]
server=server1.com:9997,server2:9997
autoLB = true
autoLBFrequency = 300
forceTimebasedAutoLB = true
useACK = true
Indexer config
On indexer, the inputs.conf is in /opt/splunk/etc/apps/myapp/local
cat inputs.conf
[splunktcp://9997]
disabled = 1

The server.conf in /opt/splunk/etc/system/local location has following stanza
[general]
pass4SymmKey = $1$xxxxxxx
serverName = myserver.com

[clustering]
master_uri = https://myclustermaster.com:8089
mode = slave

[license]
master_uri = https://mylicensemaster.com:8089

Forwarder error
I am seeing following error in forwarder splunkd.log

07-14-2016 11:58:09.776 +0100 INFO WatchedFile - Will begin reading at offset=966525 for file='/var/xxx/logs/jetty/jetty.log'.
07-14-2016 11:58:09.794 +0100 INFO WatchedFile - Will begin reading at offset=316928 for file='/opt/splunkforwarder/var/log/splunk/metrics.log'.
07-14-2016 11:58:09.968 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_stdout.log'.
07-14-2016 11:58:09.969 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/mongod.log'.
07-14-2016 11:58:09.971 +0100 INFO WatchedFile - Will begin reading at offset=9129 for file='/opt/splunkforwarder/var/log/splunk/splunkd-utility.log'.
07-14-2016 11:58:09.974 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage.log'.
07-14-2016 11:58:09.976 +0100 INFO WatchedFile - Will begin reading at offset=3230 for file='/opt/splunkforwarder/var/log/splunk/conf.log'.
07-14-2016 11:58:09.978 +0100 INFO WatchedFile - Will begin reading at offset=1230 for file='/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log'.
07-14-2016 11:58:10.004 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/remote_searches.log'.
07-14-2016 11:58:10.006 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/scheduler.log'.
07-14-2016 11:58:10.010 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log'.
07-14-2016 11:58:10.045 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/searchhistory.log'.
07-14-2016 11:58:10.048 +0100 INFO WatchedFile - Will begin reading at offset=68593 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.
07-14-2016 11:58:29.697 +0100 WARN TcpOutputProc - Cooked connection to ip=Inderxer1:9997 timed out
07-14-2016 11:58:49.697 +0100 WARN TcpOutputProc - Cooked connection to ip=indexer2:9997 timed out

[muebel · Accepted solution]

Hi vinitatsky, I believe the issue is that you set splunktcp://9997 to disabled on your indexer. Try

[splunktcp://9997]
disabled = 0

Please let me know if this answers your question! 😄

[vinitatsky]

Thanks @muebel
It was an issue with our index configuration and we managed to solve the issue.
Thanks for your quick response..!!

[sanjayagrey]

Thanks for prompt reply..!!

[sanjayagrey]

On indexer, myapp was in two location and the inputs.conf in first location had disabled = 1
1. /opt/splunk/etc/apps/myapp/local
2. /opt/splunk/etc/slave-apps/myapp/local
cd /opt/splunk/etc/apps/myapp/local
cat inputs.conf
[splunktcp://9997]
disabled = 1
cd /opt/splunk/etc/slave-apps/myapp/local
cat inputs.conf
[splunktcp://9997]
Removed the first location app, restarted indexers and it worked!!

[ddrillic]

Can you try to telnet <indexer> 9997 from the forwarder?

[vinitatsky]

Telnet is working fine

[sanjayagrey]

yes, I can

[ddrillic]

About the Cooked connection at TcpOutputProc - Cooked connection to ip=x.x.x.x:9997 timed out

[vinitatsky]

thanks. We managed to solve it by modifying indexer configuration as suggested by muebel