Getting Data In

Splunk configurations for SH, FWD and INDEXER

vinitatsky
Communicator

Posting on behalf of someone.

I want to setup a Splunk clustered environment with 4SH (cluster), 4IDX (cluster), FWD deployed on App box across 2 data centers, But as of now I am doing some testing with following configurations. I am new to Splunk, Can someone help please?

My configuration
1 forwarder
2 indexer
2 search heads
Forwarder config
The config files on forwarder are as below
cat inputs.conf
[monitor:////var/logs/myserver.log]
disabled = false
sourcetype = mysourcetye
index=myindex

outputs.conf
[tcpout:xxxx]
server=server1.com:9997,server2:9997
autoLB = true
autoLBFrequency = 300
forceTimebasedAutoLB = true
useACK = true
Indexer config
On indexer, the inputs.conf is in /opt/splunk/etc/apps/myapp/local
cat inputs.conf
[splunktcp://9997]
disabled = 1

The server.conf in /opt/splunk/etc/system/local location has following stanza
[general]
pass4SymmKey = $1$xxxxxxx
serverName = myserver.com

[clustering]
master_uri = https://myclustermaster.com:8089
mode = slave

[license]
master_uri = https://mylicensemaster.com:8089

Forwarder error
I am seeing following error in forwarder splunkd.log

07-14-2016 11:58:09.776 +0100 INFO WatchedFile - Will begin reading at offset=966525 for file='/var/xxx/logs/jetty/jetty.log'.
07-14-2016 11:58:09.794 +0100 INFO WatchedFile - Will begin reading at offset=316928 for file='/opt/splunkforwarder/var/log/splunk/metrics.log'.
07-14-2016 11:58:09.968 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_stdout.log'.
07-14-2016 11:58:09.969 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/mongod.log'.
07-14-2016 11:58:09.971 +0100 INFO WatchedFile - Will begin reading at offset=9129 for file='/opt/splunkforwarder/var/log/splunk/splunkd-utility.log'.
07-14-2016 11:58:09.974 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage.log'.
07-14-2016 11:58:09.976 +0100 INFO WatchedFile - Will begin reading at offset=3230 for file='/opt/splunkforwarder/var/log/splunk/conf.log'.
07-14-2016 11:58:09.978 +0100 INFO WatchedFile - Will begin reading at offset=1230 for file='/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log'.
07-14-2016 11:58:10.004 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/remote_searches.log'.
07-14-2016 11:58:10.006 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/scheduler.log'.
07-14-2016 11:58:10.010 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log'.
07-14-2016 11:58:10.045 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/searchhistory.log'.
07-14-2016 11:58:10.048 +0100 INFO WatchedFile - Will begin reading at offset=68593 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.
07-14-2016 11:58:29.697 +0100 WARN TcpOutputProc - Cooked connection to ip=Inderxer1:9997 timed out
07-14-2016 11:58:49.697 +0100 WARN TcpOutputProc - Cooked connection to ip=indexer2:9997 timed out

0 Karma
1 Solution

muebel
SplunkTrust
SplunkTrust

Hi vinitatsky, I believe the issue is that you set splunktcp://9997 to disabled on your indexer. Try

[splunktcp://9997]
disabled = 0

Please let me know if this answers your question! 😄

View solution in original post

0 Karma

muebel
SplunkTrust
SplunkTrust

Hi vinitatsky, I believe the issue is that you set splunktcp://9997 to disabled on your indexer. Try

[splunktcp://9997]
disabled = 0

Please let me know if this answers your question! 😄

0 Karma

vinitatsky
Communicator

Thanks @muebel
It was an issue with our index configuration and we managed to solve the issue.
Thanks for your quick response..!!

0 Karma

sanjayagrey
New Member

Thanks for prompt reply..!!

0 Karma

sanjayagrey
New Member

On indexer, myapp was in two location and the inputs.conf in first location had disabled = 1
1. /opt/splunk/etc/apps/myapp/local
2. /opt/splunk/etc/slave-apps/myapp/local
cd /opt/splunk/etc/apps/myapp/local
cat inputs.conf
[splunktcp://9997]
disabled = 1
cd /opt/splunk/etc/slave-apps/myapp/local
cat inputs.conf
[splunktcp://9997]
Removed the first location app, restarted indexers and it worked!!

0 Karma

ddrillic
Ultra Champion

Can you try to telnet <indexer> 9997 from the forwarder?

0 Karma

vinitatsky
Communicator

Telnet is working fine

0 Karma

sanjayagrey
New Member

yes, I can

0 Karma

ddrillic
Ultra Champion
0 Karma

vinitatsky
Communicator

thanks. We managed to solve it by modifying indexer configuration as suggested by muebel

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...