Splunk Enterprise

Index and forward, intermediate Splunk Indexer not forwarding data

somesoni2
Revered Legend

I am trying to implement a multiindexer environment where my two indexers are on different version of Splunk. IDX1 is Splunk 5.0.1 and IDX2 is 6.0. IDX is current working Indexer. It has data from all the forwarders.

What I am trying to setup IDX1 to index as well as forward all incoming data to IDX2. I have configured IDX2 to receive on port 9998, IDX1 is configured to send data to IDX 2. Below is the outputs.conf on IDX1.

[tcpout]
defaultGroup = IDX2
indexAndForward = true

[tcpout:IDX2]
compressed = true
server = <IDX2 Host>:9998
sslCertPath = XXXX.pem
sslPassword = $1$7kxWZaCKikBUbg==
sslRootCAPath = CAXXX.pem
sslVerifyServerCert = true
useACK = true

I restarted both the servers after configuration, but the IDX1 is neither indexing nor forwarding data.
Error message on _internal index on IDX1

11-15-2013 13:14:56.455 -0500 WARN TcpOutputProc - Applying quarantine to ip=X.X.X.X port=9998 _numberOfFailures=2 
11-15-2013 13:14:56.455 -0500 INFO TcpOutputProc - Connection to X.X.X.X:9998 closed. Read error. Connection reset by peer 
11-15-2013 13:14:56.455 -0500 INFO TcpOutputProc - Connected to idx=X.X.X.X:9998 
11-15-2013 13:14:56.455 -0500 INFO StatusMgr - destHost=IDX2.example.com, destIp=X.X.X.X, destPort=9998, eventType=connect_fail, publisher=tcpout, sourcePort=8089, statusee=TcpOutputProcessor 
11-15-2013 13:14:55.453 -0500 INFO TcpOutputProc - Connected to idx=X.X.X.X:9998 

Is there any configuration I am missing? Appreciate your help.

Tags (1)
1 Solution

somesoni2
Revered Legend

Resolved. With following configuration.Splunk 6.0 Indexer
Inputs.conf

[SSL]
password = certpassword
rootCA = $SPLUNK_HOME/etc/auth/blah_CA05_root.pem
serverCert = $SPLUNK_HOME/etc/auth/sslKeysfileDEV.pem
requireClientCert = false

[splunktcp-ssl:9998]
compressed = true

Splunk 5.0 Indexer
outputs.conf

[tcpout]
defaultGroup = DEV_INDEXERS_6_0
indexAndForward = true
disabled = false

[tcpout:DEV_INDEXERS_6_0]
compressed = true
server = <splunk6 indexer server>:9998
sslCertPath = $SPLUNK_HOME/etc/auth/sslKeysfileDEV.pem
sslPassword = certpassword
sslRootCAPath = $SPLUNK_HOME/etc/auth/blah_CA05_root.pem
sslVerifyServerCert = false   
useACK = true
sendCookedData = true

Restarted both indexers and boom.

View solution in original post

somesoni2
Revered Legend

Resolved. With following configuration.Splunk 6.0 Indexer
Inputs.conf

[SSL]
password = certpassword
rootCA = $SPLUNK_HOME/etc/auth/blah_CA05_root.pem
serverCert = $SPLUNK_HOME/etc/auth/sslKeysfileDEV.pem
requireClientCert = false

[splunktcp-ssl:9998]
compressed = true

Splunk 5.0 Indexer
outputs.conf

[tcpout]
defaultGroup = DEV_INDEXERS_6_0
indexAndForward = true
disabled = false

[tcpout:DEV_INDEXERS_6_0]
compressed = true
server = <splunk6 indexer server>:9998
sslCertPath = $SPLUNK_HOME/etc/auth/sslKeysfileDEV.pem
sslPassword = certpassword
sslRootCAPath = $SPLUNK_HOME/etc/auth/blah_CA05_root.pem
sslVerifyServerCert = false   
useACK = true
sendCookedData = true

Restarted both indexers and boom.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...

Design, Compete, Win: Submit Your Best Splunk Dashboards for a .conf26 Pass

Hello Splunkers,  We’re excited to kick off a Splunk Dashboard contest! We know that dashboards are a primary ...

May 2026 Splunk Expert Sessions: Security & Observability

Level Up Your Operations: May 2026 Splunk Expert Sessions Whether you are refining your security posture or ...