Hello there, We have a Search Head Cluster in 6.5.3 which configured by default in "member by member". Our configuration : shc_role_quota_enforcement="0" shc_local_quota_check="1" Splunk Documentation : To enforce quotas on a member-by-member basis, use this configuration: shc_role_quota_enforcement=false shc_local_quota_check=true Version Default enforcement 6.3-6.4 cluster-wide 6.5+ member-by-member However, while investigating skip ratio, it appears that logs are saying the opposite : index=_internal sourcetype=scheduler status=skipped | stats count by concurrency_context cluster-wide 1636 saved-search_cluster-wide 144 Does anyone knows if this is a logging issue or else ? Thanks for any hint! ... View more

Hello there, On a Search Head Cluster (6.5.3), when performing an Health Check, I have had a warning for having a high skip ratio - between 60 & 80 %. It seemed like it only affected the SHC captain. I found out that, in order to reduce the load on the SHC captain - which is executing savedsearches, ad hoc searches and delegating savedsearches between other peers -, it was recommended to configure the captain to run ad hoc searches only. It is documented here : https://docs.splunk.com/Documentation/Splunk/7.0.0/DistSearch/Adhocclustermember#Configure_the_captain_to_run_ad_hoc_searches_only This way, the captain only launches ad hoc searches & still delegates seavedsearches between cluster members. I believed this would resolve our issue. However, the skip ratio is now of 100%, still only on the captain. It is always 100% which is weird. To me it is more like an issue in the way internal log are being generated. Logs are saying : this savedsearch has been skipped on this host which is the captain, reason : the max number of auto summarization has been reached ... While it should rather be saying : this savedsearch has been skipped on this host which is the captain, reason : captain configured to run ad hoc searches only. The thing that makes me doubt about this is the reason savedsearches are being skipped on the captain : "the max number of auto summarization has been reached" So I am wondering if : it is really a good practice & it's more like a logging issue or it is not a good practice Note that there are no errors on the Data Models savedsearches are flagged as being skipped -> Build 100 % Would anyone have an idea on this ? Thanks for any feedback! ... View more

Hi there, I have deployed Splunk Stream on a distributed environment to ingest DNS first. I have followed howtos here and there and everything is fine with collected data. One thing remains unclear. I have Splunk Stream + Stream TA on my ES Search Head Stream TA on another Search Head > just for parsing Stream TA on Deployement Server > just for parsing Stream TA on Indexer > for indexing, timestamp etc. Stream TA + inputs on DNS servers However I do receive error messages from SH, DS & IDX mentioning permission issues : Unable to initialize modular input "streamfwd" defined inside the app "Splunk_TA_stream": Introspecting scheme=streamfwd: Unable to run "/opt/splunk/etc/apps/Splunk_TA_stream/linux_x86_64/bin/streamfwd --scheme": child failed to start: Permission denied I have already used set_permissions.sh so that might be due to the the fact that Splunk is running as non-root. However, on these instance, the TA is not there for capturing any stream, so isn't better to just turn off TA's network capturing capability ? I am wondering what files should I removed from the TA to do this and if this is is a good idea to do so. Thanks in advance, ... View more

Hi there, Does anyone here have succeeded in configuring SA-ldapsearch using TLS on a SHC ? We have successfully configured it on a Heavy Forwarder part of our architecture but it does not work on a Search Head member of our Search Head Cluster where it does not seem to event load SSL settings. Here is some details: SSL Config is the same on both instances : SH $ splunk cmd btool --app=SA-ldapsearch ssl list [sslConfig] caCertFile = /opt/splunk/etc/auth/ca.pem sslVersions = tls HF $ splunk cmd btool --app=SA-ldapsearch ssl list [sslConfig] caCertFile = /opt/splunk/etc/auth/ca.pem sslVersions = tls There is also an sslConfig stanza in another App, also indentical : [sslConfig] sslRootCAPath = $SPLUNK_HOME/etc/auth/ca.pem serverCert = $SPLUNK_HOME/etc/auth/splunk-cert.pem requireClientCert = true sslVerifyServerCert = true certCreateScript = sslVersions = tls1.1, tls1.2 sslVersionsForClient = tls1.1, tls1.2 SA-ldapsearch.log on the HF OK : 2017-09-11 18:56:32,218, Level=DEBUG, Pid=16868, File=search_command.py, Line=294, LdapTestConnectionCommand arguments: ['/opt/splunk/etc/apps/SA-ldapsearch/bin/ldaptestconnection.py', 'EXECUTE', 'domain="default"'] 2017-09-11 18:56:32,220, Level=DEBUG, Pid=16868, File=search_command_internals.py, Line=296, LdapTestConnectionCommand: ldaptestconnection domain="default" 2017-09-11 18:56:32,220, Level=DEBUG, Pid=16868, File=ldaptestconnection.py, Line=48, Command = ldaptestconnection domain="default" 2017-09-11 18:56:32,220, Level=DEBUG, Pid=16868, File=configuration.py, Line=47, Command = ldaptestconnection domain="default" 2017-09-11 18:56:32,242, Level=DEBUG, Pid=16868, File=configuration.py, Line=536, Configuration = ldaptestconnection(server=[Server(host='domain.local', port=636, use_ssl=True, allowed_referral_hosts=[(u'', True)], tls=Tls(validate=2, version=2, ca_certs_file='/opt/splunk/etc/auth/ca.pem'), get_info=3), Server(host='domain.local', port=636, use_ssl=True, allowed_referral_hosts=[(u'', True)], tls=Tls(validate=2, version=2, ca_certs_file='/opt/splunk/etc/auth/ca.pem'), get_info=3), Server(host='domain.local', port=636, use_ssl=True, allowed_referral_hosts=[(u'*', True)], tls=Tls(validate=2, version=2, ca_certs_file='/opt/splunk/etc/auth/ca.pem'), get_info=3)], credentials=CN=Splunk,OU=Admins,DC=domain,DC=local, alternatedomain=domain.local, basedn=dc=domain,dc=local, decode=True, paged_size=1000) 2017-09-11 18:56:32,242, Level=DEBUG, Pid=16868, File=ldaptestconnection.py, Line=65, Testing the connection to ldaps://domain.local:636 SA-ldapsearch.log on the SH KO, seems like SSL parameters are not loaded: 2017-09-11 18:54:06,998, Level=DEBUG, Pid=9063, File=search_command.py, Line=294, LdapTestConnectionCommand arguments: ['/opt/splunk/etc/apps/SA-ldapsearch/bin/ldaptestconnection.py', 'EXECUTE', 'domain="default"'] 2017-09-11 18:54:07,000, Level=DEBUG, Pid=9063, File=search_command_internals.py, Line=296, LdapTestConnectionCommand: ldaptestconnection domain="default" 2017-09-11 18:54:07,000, Level=DEBUG, Pid=9063, File=ldaptestconnection.py, Line=48, Command = ldaptestconnection domain="default" 2017-09-11 18:54:07,001, Level=DEBUG, Pid=9063, File=configuration.py, Line=47, Command = ldaptestconnection domain="default" 2017-09-11 18:54:07,006, Level=ERROR, Pid=9063, File=search_command.py, Line=346, Traceback (most recent call last): File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/searchcommands/search_command.py", line 320, in process self.execute(operation, reader, writer) File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/searchcommands/generating_command.py", line 79, in _execute for record in operation(): File "/opt/splunk/etc/apps/SA-ldapsearch/bin/ldaptestconnection.py", line 49, in generate configuration = app.Configuration(self) File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/app/configuration.py", line 52, in __init_ self.read_configuration() File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/app/configuration.py", line 432, in _read_configuration settings = self._read_default_configuration() File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/app/configuration.py", line 457, in _read_default_configuration response = service.get('properties/ldap/default', namespace.owner, namespace.app, namespace.sharing) File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/binding.py", line 241, in wrapper return request_fun(self, *args, **kwargs) File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/binding.py", line 62, in new_f val = f(*args, **kwargs) File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/binding.py", line 586, in get response = self.http.get(path, self._auth_headers, **query) File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/binding.py", line 1056, in get return self.request(url, { 'method': "GET", 'headers': headers }) File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/binding.py", line 1108, in request response = self.handler(url, message, **kwargs) File "/opt/splunk/etc/apps/SA-ldapsearch/bin/packages/splunklib/binding.py", line 1226, in request connection.request(method, path, body, head) File "/opt/splunk/lib/python2.7/httplib.py", line 1057, in request self._send_request(method, url, body, headers) File "/opt/splunk/lib/python2.7/httplib.py", line 1097, in _send_request self.endheaders(body) File "/opt/splunk/lib/python2.7/httplib.py", line 1053, in endheaders self._send_output(message_body) File "/opt/splunk/lib/python2.7/httplib.py", line 897, in _send_output self.send(msg) File "/opt/splunk/lib/python2.7/httplib.py", line 859, in send self.connect() File "/opt/splunk/lib/python2.7/httplib.py", line 1278, in connect server_hostname=server_hostname) File "/opt/splunk/lib/python2.7/ssl.py", line 352, in wrap_socket _context=self) File "/opt/splunk/lib/python2.7/ssl.py", line 579, in __init_ self.do_handshake() File "/opt/splunk/lib/python2.7/ssl.py", line 808, in do_handshake self._sslobj.do_handshake() SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:603) Thanks in advance for any feedback! ... View more