Hi
For the first time i am trying to configure a distributed search (Non Clustered).
http://docs.splunk.com/Documentation/Splunk/7.2.0/DistSearch/Overviewofconfiguration
I have created 2 new Indexers and i have taken my main install (I used to have a search head and an indexer on it), i have disabled the indexer on it. So now i have one search head and 2 new indexers.
The output.conf looks like this
# Turn off indexing on the search head
[indexAndForward]
index = false
[tcpout]
defaultGroup = my_search_peers
forwardedindex.filter.disable = true
indexAndForward = false
[tcpout:my_search_peers]
server=10.25.5.169:5997,10.25.53.57:5997
I can see that the search head is connected from the logs
11-09-2018 19:12:40.260 +0100 INFO TcpOutputProc - Connected to idx=10.25.5.169:5997, pset=0, reuse=0.
11-09-2018 19:12:42.543 +0100 INFO TcpOutputProc - Connected to idx=10.25.53.57:5997, pset=1, reuse=0.
inputs.conf (On the forwarder)
[default]
host = hp400srv_5000
[splunktcp://5997]
connection_host = ip
I have added the indexers to the search head, i think they are ok, but not sure how to check?
I can see data on one of my indexers by logging in via web (I will disable web when i have this all working)
But the issue is when i log into my search head (That is now connected to my 2 new Indexers).
I can't see any data for the same command "index=mlc_live" for a 5 minute real time search. So i have the 2 windows side by side, i can see data coming into one of the Indexers, but i cant see the same on the the search head.
Am i missing something? Is it a user right issue, on the index or something.
The data is coming into an app that i have created, i manually copied it over to the indexers(for now) to make sure they had an index and data-models for the forwarded data to go.
I am getting some errors in the logs but i don't think they are related to this?
11-09-2018 19:40:35.516 +0100 WARN MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:36.190 +0100 WARN MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:36.963 +0100 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='protocol version'.
11-09-2018 19:40:36.963 +0100 WARN HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
11-09-2018 19:40:37.042 +0100 WARN IConfCache - Stanza has an expansion [script:///hp737srv1/apps/SPLUNK_WEEKLY_BACKUP/04-11-2018_00-30/splunk/etc/apps/TA-sos/bin/lsof_sos.sh], ignoring alternate expansion [script:///hp737srv1/apps/SPLUNK_WEEKLY_BACKUP/04-11-2018_00-30/splunk/etc/apps/sos/bin/lsof_sos.sh] in inputs.conf
11-09-2018 19:40:37.042 +0100 WARN IConfCache - Stanza has an expansion [script:///hp737srv1/apps/SPLUNK_WEEKLY_BACKUP/04-11-2018_00-30/splunk/etc/apps/TA-sos/bin/nfs-iostat_sos.py], ignoring alternate expansion [script:///hp737srv1/apps/SPLUNK_WEEKLY_BACKUP/04-11-2018_00-30/splunk/etc/apps/sos/bin/nfs-iostat_sos.py] in inputs.conf
11-09-2018 19:40:37.042 +0100 WARN IConfCache - Stanza has an expansion [script:///hp737srv1/apps/SPLUNK_WEEKLY_BACKUP/04-11-2018_00-30/splunk/etc/apps/TA-sos/bin/ps_sos.sh], ignoring alternate expansion [script:///hp737srv1/apps/SPLUNK_WEEKLY_BACKUP/04-11-2018_00-30/splunk/etc/apps/sos/bin/ps_sos.sh] in inputs.conf
11-09-2018 19:40:37.044 +0100 INFO TcpOutputProc - Connected to idx=10.25.53.57:5997, pset=1, reuse=0.
11-09-2018 19:40:37.197 +0100 WARN MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:38.194 +0100 WARN MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:39.185 +0100 WARN MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:39.770 +0100 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='protocol version'.
11-09-2018 19:40:39.770 +0100 WARN HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
11-09-2018 19:40:40.196 +0100 WARN MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:41.185 +0100 WARN MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:42.185 +0100 WARN MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:42.503 +0100 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='protocol version'.
11-09-2018 19:40:42.503 +0100 WARN HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
11-09-2018 19:40:43.185 +0100 WARN MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:44.185 +0100 WARN MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:45.185 +0100 WARN MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:45.281 +0100 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='protocol version'.
11-09-2018 19:40:45.281 +0100 WARN HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
11-09-2018 19:40:46.185 +0100 WARN MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
11-09-2018 19:40:47.286 +0100 WARN MongoModificationsTracker - Could not load configuration for collection 'MXTIMING_MONITORING' in application 'murex_mlc'. Collection will be ignored.
Any help would be so so cool - cheers 🙂
You should have an outputs.conf
on every non-indexer that looks like this:
[tcpout]
defaultGroup = primary_indexers
# Correct an issue with the default outputs.conf for the Universal Forwarder
# or the SplunkLightForwarder app; these don't forward _internal events.
forwardedindex.2.whitelist = (_audit|_introspection|_internal)
[tcpout:primary_indexers]
server = indexer_one:9997, indexer_two:9997
You should have an inputs.conf
like this on every indexer:
[splunktcp://9997]
In your case, it looks like you are swapping 9997
for 5997
; that's fine, just make sure that both files have the same port number.
Lastly, you need to configure your indexers as search peers on the Search Head (the GUI is very easy):
https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Configuredistributedsearch
You should have an outputs.conf
on every non-indexer that looks like this:
[tcpout]
defaultGroup = primary_indexers
# Correct an issue with the default outputs.conf for the Universal Forwarder
# or the SplunkLightForwarder app; these don't forward _internal events.
forwardedindex.2.whitelist = (_audit|_introspection|_internal)
[tcpout:primary_indexers]
server = indexer_one:9997, indexer_two:9997
You should have an inputs.conf
like this on every indexer:
[splunktcp://9997]
In your case, it looks like you are swapping 9997
for 5997
; that's fine, just make sure that both files have the same port number.
Lastly, you need to configure your indexers as search peers on the Search Head (the GUI is very easy):
https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Configuredistributedsearch
MR Woodcock, i hope you are well 🙂
Thanks for the answer, this is what worked
inputs.conf
[default]
host = hp400srv_5000
[splunktcp://5997]
connection_host = ip
outputs.conf
# Turn off indexing on the search head
[indexAndForward]
index = false
[tcpout]
defaultGroup = my_search_peers
forwardedindex.filter.disable = true
indexAndForward = false
[tcpout:my_search_peers]
server=10.25.5.169:5997,10.25.53.57:5997
Cheers
Rob
As per ddrillic try index=* OR index=_internal from the search heads and see if data returns.
If not start looking at splunkd for ERROR or WARN level information and see what shows up...
Why are you using such odd ports? Nuance practices like these will get you in a lot of trouble.
The indexer port, 5997
in this case, is really up to the application.
From the search head do you see data for index=_internal
?