Getting Data In

Search Head cant see data in Indexers

robertlynch2020
Motivator

Hi

For the first time i am trying to configure a distributed search (Non Clustered).
http://docs.splunk.com/Documentation/Splunk/7.2.0/DistSearch/Overviewofconfiguration

I have created 2 new Indexers and i have taken my main install (I used to have a search head and an indexer on it), i have disabled the indexer on it. So now i have one search head and 2 new indexers.

The output.conf looks like this

# Turn off indexing on the search head
[indexAndForward]
index = false
[tcpout]
defaultGroup = my_search_peers
forwardedindex.filter.disable = true  
indexAndForward = false
[tcpout:my_search_peers]
server=10.25.5.169:5997,10.25.53.57:5997

I can see that the search head is connected from the logs
11-09-2018 19:12:40.260 +0100 INFO TcpOutputProc - Connected to idx=10.25.5.169:5997, pset=0, reuse=0.
11-09-2018 19:12:42.543 +0100 INFO TcpOutputProc - Connected to idx=10.25.53.57:5997, pset=1, reuse=0.

inputs.conf (On the forwarder)

[default]
host = hp400srv_5000

[splunktcp://5997]
connection_host = ip

I have added the indexers to the search head, i think they are ok, but not sure how to check?
alt text

I can see data on one of my indexers by logging in via web (I will disable web when i have this all working)
alt text

But the issue is when i log into my search head (That is now connected to my 2 new Indexers).
I can't see any data for the same command "index=mlc_live" for a 5 minute real time search. So i have the 2 windows side by side, i can see data coming into one of the Indexers, but i cant see the same on the the search head.
Am i missing something? Is it a user right issue, on the index or something.

The data is coming into an app that i have created, i manually copied it over to the indexers(for now) to make sure they had an index and data-models for the forwarded data to go.

I am getting some errors in the logs but i don't think they are related to this?

11-09-2018 19:40:35.516 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:36.190 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:36.963 +0100 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='protocol version'.
11-09-2018 19:40:36.963 +0100 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
11-09-2018 19:40:37.042 +0100 WARN  IConfCache - Stanza has an expansion [script:///hp737srv1/apps/SPLUNK_WEEKLY_BACKUP/04-11-2018_00-30/splunk/etc/apps/TA-sos/bin/lsof_sos.sh], ignoring alternate expansion [script:///hp737srv1/apps/SPLUNK_WEEKLY_BACKUP/04-11-2018_00-30/splunk/etc/apps/sos/bin/lsof_sos.sh] in inputs.conf
11-09-2018 19:40:37.042 +0100 WARN  IConfCache - Stanza has an expansion [script:///hp737srv1/apps/SPLUNK_WEEKLY_BACKUP/04-11-2018_00-30/splunk/etc/apps/TA-sos/bin/nfs-iostat_sos.py], ignoring alternate expansion [script:///hp737srv1/apps/SPLUNK_WEEKLY_BACKUP/04-11-2018_00-30/splunk/etc/apps/sos/bin/nfs-iostat_sos.py] in inputs.conf
11-09-2018 19:40:37.042 +0100 WARN  IConfCache - Stanza has an expansion [script:///hp737srv1/apps/SPLUNK_WEEKLY_BACKUP/04-11-2018_00-30/splunk/etc/apps/TA-sos/bin/ps_sos.sh], ignoring alternate expansion [script:///hp737srv1/apps/SPLUNK_WEEKLY_BACKUP/04-11-2018_00-30/splunk/etc/apps/sos/bin/ps_sos.sh] in inputs.conf
11-09-2018 19:40:37.044 +0100 INFO  TcpOutputProc - Connected to idx=10.25.53.57:5997, pset=1, reuse=0.
11-09-2018 19:40:37.197 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:38.194 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:39.185 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:39.770 +0100 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='protocol version'.
11-09-2018 19:40:39.770 +0100 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
11-09-2018 19:40:40.196 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:41.185 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:42.185 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:42.503 +0100 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='protocol version'.
11-09-2018 19:40:42.503 +0100 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
11-09-2018 19:40:43.185 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:44.185 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:45.185 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:45.281 +0100 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='protocol version'.
11-09-2018 19:40:45.281 +0100 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
11-09-2018 19:40:46.185 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:47.286 +0100 WARN  MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.

Any help would be so so cool - cheers 🙂

0 Karma
1 Solution

woodcock
Esteemed Legend

You should have an outputs.conf on every non-indexer that looks like this:

[tcpout]
defaultGroup = primary_indexers

# Correct an issue with the default outputs.conf for the Universal Forwarder
# or the SplunkLightForwarder app; these don't forward _internal events.
forwardedindex.2.whitelist = (_audit|_introspection|_internal)

[tcpout:primary_indexers]
server = indexer_one:9997, indexer_two:9997

You should have an inputs.conf like this on every indexer:

[splunktcp://9997]

In your case, it looks like you are swapping 9997 for 5997; that's fine, just make sure that both files have the same port number.
Lastly, you need to configure your indexers as search peers on the Search Head (the GUI is very easy):
https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Configuredistributedsearch

View solution in original post

0 Karma

woodcock
Esteemed Legend

You should have an outputs.conf on every non-indexer that looks like this:

[tcpout]
defaultGroup = primary_indexers

# Correct an issue with the default outputs.conf for the Universal Forwarder
# or the SplunkLightForwarder app; these don't forward _internal events.
forwardedindex.2.whitelist = (_audit|_introspection|_internal)

[tcpout:primary_indexers]
server = indexer_one:9997, indexer_two:9997

You should have an inputs.conf like this on every indexer:

[splunktcp://9997]

In your case, it looks like you are swapping 9997 for 5997; that's fine, just make sure that both files have the same port number.
Lastly, you need to configure your indexers as search peers on the Search Head (the GUI is very easy):
https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Configuredistributedsearch

View solution in original post

0 Karma

robertlynch2020
Motivator

MR Woodcock, i hope you are well 🙂

Thanks for the answer, this is what worked

inputs.conf
[default]
host = hp400srv_5000

[splunktcp://5997]
connection_host = ip

outputs.conf
# Turn off indexing on the search head
[indexAndForward]
index = false
[tcpout]
defaultGroup = my_search_peers
forwardedindex.filter.disable = true  
indexAndForward = false
[tcpout:my_search_peers]
server=10.25.5.169:5997,10.25.53.57:5997

Cheers
Rob

0 Karma

gjanders
SplunkTrust
SplunkTrust

As per ddrillic try index=* OR index=_internal from the search heads and see if data returns.
If not start looking at splunkd for ERROR or WARN level information and see what shows up...

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/
0 Karma

davpx
Communicator

Why are you using such odd ports? Nuance practices like these will get you in a lot of trouble.

0 Karma

ddrillic
Ultra Champion

The indexer port, 5997 in this case, is really up to the application.

0 Karma

ddrillic
Ultra Champion

From the search head do you see data for index=_internal?

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!