Search Head cant see data in Indexers

[robertlynch2020]

Hi

For the first time i am trying to configure a distributed search (Non Clustered).
http://docs.splunk.com/Documentation/Splunk/7.2.0/DistSearch/Overviewofconfiguration

I have created 2 new Indexers and i have taken my main install (I used to have a search head and an indexer on it), i have disabled the indexer on it. So now i have one search head and 2 new indexers.

The output.conf looks like this

# Turn off indexing on the search head
[indexAndForward]
index = false
[tcpout]
defaultGroup = my_search_peers
forwardedindex.filter.disable = true  
indexAndForward = false
[tcpout:my_search_peers]
server=10.25.5.169:5997,10.25.53.57:5997

I can see that the search head is connected from the logs
11-09-2018 19:12:40.260 +0100 INFO TcpOutputProc - Connected to idx=10.25.5.169:5997, pset=0, reuse=0.
11-09-2018 19:12:42.543 +0100 INFO TcpOutputProc - Connected to idx=10.25.53.57:5997, pset=1, reuse=0.

inputs.conf (On the forwarder)

[default]
host = hp400srv_5000

[splunktcp://5997]
connection_host = ip

I have added the indexers to the search head, i think they are ok, but not sure how to check?

I can see data on one of my indexers by logging in via web (I will disable web when i have this all working)

But the issue is when i log into my search head (That is now connected to my 2 new Indexers).
I can't see any data for the same command "index=mlc_live" for a 5 minute real time search. So i have the 2 windows side by side, i can see data coming into one of the Indexers, but i cant see the same on the the search head.
Am i missing something? Is it a user right issue, on the index or something.

The data is coming into an app that i have created, i manually copied it over to the indexers(for now) to make sure they had an index and data-models for the forwarded data to go.

I am getting some errors in the logs but i don't think they are related to this?

11-09-2018 19:40:35.516 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:36.190 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:36.963 +0100 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='protocol version'.
11-09-2018 19:40:36.963 +0100 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
11-09-2018 19:40:37.042 +0100 WARN  IConfCache - Stanza has an expansion [script:///hp737srv1/apps/SPLUNK_WEEKLY_BACKUP/04-11-2018_00-30/splunk/etc/apps/TA-sos/bin/lsof_sos.sh], ignoring alternate expansion [script:///hp737srv1/apps/SPLUNK_WEEKLY_BACKUP/04-11-2018_00-30/splunk/etc/apps/sos/bin/lsof_sos.sh] in inputs.conf
11-09-2018 19:40:37.042 +0100 WARN  IConfCache - Stanza has an expansion [script:///hp737srv1/apps/SPLUNK_WEEKLY_BACKUP/04-11-2018_00-30/splunk/etc/apps/TA-sos/bin/nfs-iostat_sos.py], ignoring alternate expansion [script:///hp737srv1/apps/SPLUNK_WEEKLY_BACKUP/04-11-2018_00-30/splunk/etc/apps/sos/bin/nfs-iostat_sos.py] in inputs.conf
11-09-2018 19:40:37.042 +0100 WARN  IConfCache - Stanza has an expansion [script:///hp737srv1/apps/SPLUNK_WEEKLY_BACKUP/04-11-2018_00-30/splunk/etc/apps/TA-sos/bin/ps_sos.sh], ignoring alternate expansion [script:///hp737srv1/apps/SPLUNK_WEEKLY_BACKUP/04-11-2018_00-30/splunk/etc/apps/sos/bin/ps_sos.sh] in inputs.conf
11-09-2018 19:40:37.044 +0100 INFO  TcpOutputProc - Connected to idx=10.25.53.57:5997, pset=1, reuse=0.
11-09-2018 19:40:37.197 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:38.194 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:39.185 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:39.770 +0100 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='protocol version'.
11-09-2018 19:40:39.770 +0100 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
11-09-2018 19:40:40.196 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:41.185 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:42.185 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:42.503 +0100 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='protocol version'.
11-09-2018 19:40:42.503 +0100 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
11-09-2018 19:40:43.185 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:44.185 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:45.185 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:45.281 +0100 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='protocol version'.
11-09-2018 19:40:45.281 +0100 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
11-09-2018 19:40:46.185 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:47.286 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.

Any help would be so so cool - cheers 🙂

[woodcock · Accepted solution]

You should have an outputs.conf on every non-indexer that looks like this:

[tcpout]
defaultGroup = primary_indexers

# Correct an issue with the default outputs.conf for the Universal Forwarder
# or the SplunkLightForwarder app; these don't forward _internal events.
forwardedindex.2.whitelist = (_audit|_introspection|_internal)

[tcpout:primary_indexers]
server = indexer_one:9997, indexer_two:9997

You should have an inputs.conf like this on every indexer:

[splunktcp://9997]

In your case, it looks like you are swapping 9997 for 5997; that's fine, just make sure that both files have the same port number.
Lastly, you need to configure your indexers as search peers on the Search Head (the GUI is very easy):
https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Configuredistributedsearch

[robertlynch2020]

MR Woodcock, i hope you are well 🙂

Thanks for the answer, this is what worked

inputs.conf
[default]
host = hp400srv_5000

[splunktcp://5997]
connection_host = ip

outputs.conf
# Turn off indexing on the search head
[indexAndForward]
index = false
[tcpout]
defaultGroup = my_search_peers
forwardedindex.filter.disable = true  
indexAndForward = false
[tcpout:my_search_peers]
server=10.25.5.169:5997,10.25.53.57:5997

Cheers
Rob

[gjanders]

As per ddrillic try index=* OR index=_internal from the search heads and see if data returns.
If not start looking at splunkd for ERROR or WARN level information and see what shows up...

[davpx]

Why are you using such odd ports? Nuance practices like these will get you in a lot of trouble.

[ddrillic]

The indexer port, 5997 in this case, is really up to the application.

[ddrillic]

From the search head do you see data for index=_internal?