Getting Data In

Splunk configurations for SH, FWD and INDEXER

vinitatsky
Communicator

Posting on behalf of someone.

I want to setup a Splunk clustered environment with 4SH (cluster), 4IDX (cluster), FWD deployed on App box across 2 data centers, But as of now I am doing some testing with following configurations. I am new to Splunk, Can someone help please?

My configuration
1 forwarder
2 indexer
2 search heads
Forwarder config
The config files on forwarder are as below
cat inputs.conf
[monitor:////var/logs/myserver.log]
disabled = false
sourcetype = mysourcetye
index=myindex

outputs.conf
[tcpout:xxxx]
server=server1.com:9997,server2:9997
autoLB = true
autoLBFrequency = 300
forceTimebasedAutoLB = true
useACK = true
Indexer config
On indexer, the inputs.conf is in /opt/splunk/etc/apps/myapp/local
cat inputs.conf
[splunktcp://9997]
disabled = 1

The server.conf in /opt/splunk/etc/system/local location has following stanza
[general]
pass4SymmKey = $1$xxxxxxx
serverName = myserver.com

[clustering]
master_uri = https://myclustermaster.com:8089
mode = slave

[license]
master_uri = https://mylicensemaster.com:8089

Forwarder error
I am seeing following error in forwarder splunkd.log

07-14-2016 11:58:09.776 +0100 INFO WatchedFile - Will begin reading at offset=966525 for file='/var/xxx/logs/jetty/jetty.log'.
07-14-2016 11:58:09.794 +0100 INFO WatchedFile - Will begin reading at offset=316928 for file='/opt/splunkforwarder/var/log/splunk/metrics.log'.
07-14-2016 11:58:09.968 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_stdout.log'.
07-14-2016 11:58:09.969 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/mongod.log'.
07-14-2016 11:58:09.971 +0100 INFO WatchedFile - Will begin reading at offset=9129 for file='/opt/splunkforwarder/var/log/splunk/splunkd-utility.log'.
07-14-2016 11:58:09.974 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage.log'.
07-14-2016 11:58:09.976 +0100 INFO WatchedFile - Will begin reading at offset=3230 for file='/opt/splunkforwarder/var/log/splunk/conf.log'.
07-14-2016 11:58:09.978 +0100 INFO WatchedFile - Will begin reading at offset=1230 for file='/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log'.
07-14-2016 11:58:10.004 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/remote_searches.log'.
07-14-2016 11:58:10.006 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/scheduler.log'.
07-14-2016 11:58:10.010 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log'.
07-14-2016 11:58:10.045 +0100 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/searchhistory.log'.
07-14-2016 11:58:10.048 +0100 INFO WatchedFile - Will begin reading at offset=68593 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.
07-14-2016 11:58:29.697 +0100 WARN TcpOutputProc - Cooked connection to ip=Inderxer1:9997 timed out
07-14-2016 11:58:49.697 +0100 WARN TcpOutputProc - Cooked connection to ip=indexer2:9997 timed out

0 Karma
1 Solution

muebel
SplunkTrust
SplunkTrust

Hi vinitatsky, I believe the issue is that you set splunktcp://9997 to disabled on your indexer. Try

[splunktcp://9997]
disabled = 0

Please let me know if this answers your question! 😄

View solution in original post

0 Karma

muebel
SplunkTrust
SplunkTrust

Hi vinitatsky, I believe the issue is that you set splunktcp://9997 to disabled on your indexer. Try

[splunktcp://9997]
disabled = 0

Please let me know if this answers your question! 😄

0 Karma

vinitatsky
Communicator

Thanks @muebel
It was an issue with our index configuration and we managed to solve the issue.
Thanks for your quick response..!!

0 Karma

sanjayagrey
New Member

Thanks for prompt reply..!!

0 Karma

sanjayagrey
New Member

On indexer, myapp was in two location and the inputs.conf in first location had disabled = 1
1. /opt/splunk/etc/apps/myapp/local
2. /opt/splunk/etc/slave-apps/myapp/local
cd /opt/splunk/etc/apps/myapp/local
cat inputs.conf
[splunktcp://9997]
disabled = 1
cd /opt/splunk/etc/slave-apps/myapp/local
cat inputs.conf
[splunktcp://9997]
Removed the first location app, restarted indexers and it worked!!

0 Karma

ddrillic
Ultra Champion

Can you try to telnet <indexer> 9997 from the forwarder?

0 Karma

vinitatsky
Communicator

Telnet is working fine

0 Karma

sanjayagrey
New Member

yes, I can

0 Karma

ddrillic
Ultra Champion
0 Karma

vinitatsky
Communicator

thanks. We managed to solve it by modifying indexer configuration as suggested by muebel

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...