I am using Splunk Version 6.2.2 with search head and few indexer. My searches are failing with following error Search filters specified using splunk_server/splunk_server_group do not match any search peer. ... View more

Our master node is still losing sight of indexer nodes. We had a problem similar to this that was affecting up to half of our cluster at a time. Currently, it is affecting 1 or 2 indexers at a time. Master reports the indexer as down, but the indexer is in fact up. This flapping of the In total you have 20 indexer, where each indexer has around 25K buckets . SO in total you have 20*25,000= 500,000 bucket. ... View more

Followed Splunk documentation steps to enable SSL on splunk web using self signed certificates. When opening the site after restarting splunk, getting “NET::ERR_CERT_INVALID” error in both Chrome and Internet Explorer. Firefox allows an exception to be written to get past the error, Chrome and IE do not and so Splunk is inaccessible for users without FF. ... View more

We have need to set ui preference with default time range picker globally across of the searches run in Splunk. We need to deploy these configuration across all three search Head Cluster Member in SHCM (search Head Cluster Memeber Pool) ... View more

Cluster Member on Version : 6.2.1 and has three Search Head Cluster Memeber: *Issue : * 1) When user run’s adhoc search from the Search head cluster Captain (even simple search like index= ) the result start to stream right away , also when the search is being typed in Search Bar the guiding Matching searches is displayed. 2) On the other had when the same adhoc search is run from Non-Captain SHC member , after that search is typed and asked to run, the search bar turns GREY and initially no results are streamed, after some delay the bust of result get displayed message {** “Gave up waiting for the captain to establish a common bundle version across all search peers; using most recent bundles on all peers instead** is displayed. Also for Non-Captain SHC member when the search is typed in Search Bar the guiding Matching search never gets displayed. ... View more

We recently deployed the following config to 500 Windows Universal Forwarders: [WinEventLog://Security] disabled = 0 start_from = oldest evt_resolve_ad_obj = 1 And that almost killed our primary domain controller. For some reason all the forwarders tried to query this PDC instead of contacting their local domain controller so we have disabled the SID translation for now. Couple of questions: Is there any way to specify evt_dc_name in such a way that the universal fw uses its local domain controller instead of going to the PDC? Could we potentially specify "evt_dc_name = localhost" to force the universal forwarders to translate SIDs locally? Will that work? I know I could deploy different config files per sites simply by using whitelists and machine names, but this is not 100% reliable, how do you guys deal with event logs and sid translation in large infrastructures? Finally, is there any way to tell the universal forwarders to cache SID previously translated for a certain period of time? it seems to me like a waster of resources to be querying the domain controllers all the time. ... View more

How to check if the report acceleration is used on my environment where we are using Search Head Clustering( 3 members) and Cluster Replication(3 indexer) . ... View more

Splunk Universal forwarder upgrade from version 5.0.4 to any 6.x on 'AIX' failed. -The upgrade failed exactly the same way on two different systems - one was AIX 6.1 and the other was AIX 5.3. -All of the following upgrade attempts failed Upgrade from Splunk Universal Forwarder 5.0.4 > Splunk Universal Forwarder 6.0.1 Upgrade from Splunk Universal Forwarder 5.0.4 > Splunk Universal Forwarder 6.0.2 Upgrade from Splunk Universal Forwarder 5.0.4 > Splunk Universal Forwarder 6.0.3 Impact : Customer has over 100 Splunk Universal Forwarder to be upgraded, so impact is widespread. This worked: -In the past, they have upgraded ‘Splunk Universal Forwarder’ from 5.0.2 to 5.0.4 without issue. -Fresh install of Splunk Universal forwarder 6.x works fine -Full Instance of Splunk upgraded fine from 5.0.4 to 6.0.x Below are the error messages when starting Splunk post-upgrade. ********** BEGIN PREVIEW OF CONFIGURATION FILE MIGRATION ********** Could not load program /usr/local/splunkforwarder/bin/splunkd: Dependent module /usr/local/splunkforwarder/lib/libpcre.a(libpcre.so.1) could not be loaded. Member libpcre.so.1 is not found in archive ********** END PREVIEW OF CONFIGURATION FILE MIGRATION ********** dmr2o@xxxx[QA-ILMT][/usr/local/splunkforwarder]ls -l total 232 -r--r--r-- 1 root system 510 Dec 13 16:25 README-splunk.txt drwxr-xr-x 3 root system 4096 Dec 13 16:36 bin -r--r--r-- 1 root system 57 Dec 13 16:28 copyright.txt drwxr-xr-x 12 root system 4096 Dec 13 16:36 etc -rw-r--r-- 1 root system 0 Dec 13 16:28 ftr drwxr-xr-x 2 root system 256 Dec 13 16:36 include drwxr-xr-x 3 root system 4096 Dec 13 16:36 lib …………. ... View more

we have deployed a mobile access server and function normally. since we have 3 splunk enterprise servers, and i can't find a way for configuration of mobile access server to establish several connection to multiple servers, so we would like to know is it possible to config it. ... View more

Can the Cluster Peer be re-added to the Cluster Master without restarting Cluster master or the Cluster Peer? I have three Cluster Peers with RF=2 and SF=2. For some reason, one warm bucket of one of the indexes is not getting replicated to other peers. The data in the bucket is searchable. ... View more

I am on Splunk Version : 6.1.3 and trying to use splunk supported cipherSuite from TLSv1.2, but it is causing the CLI command to fail and as a result he is unable to posh the cluster bundle or use the REST end point of the Peer. Here is Configuration: ../etc/system/local/server.conf [sslConfig] ../etc/system/local/server.conf allowSslCompression = false ../etc/system/default/server.conf allowSslRenegotiation = true ../etc/system/default/server.conf caCertFile = cacert.pem ../etc/system/default/server.conf caPath = ../etc/auth ../etc/system/default/server.conf certCreateScript = ../bin/splunk, createssl, server-cert ../etc/system/local/server.conf cipherSuite = AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384 ../etc/system/local/server.conf enableSplunkdSSL = true ../etc/system/default/server.conf sendStrictTransportSecurityHeader = false ../etc/system/default/server.conf sslKeysfile = server.pem ../etc/system/local/server.conf sslKeysfilePassword = ##### ../etc/system/local/server.conf supportSSLV3Only = True ../etc/system/local/server.conf useClientSSLCompression = false ../etc/system/local/server.conf useSplunkdClientSSLCompression = false Issue is that CLI command or REST endpoints is failing with error : ./splunk list monitor result in error * Splunk is not running, and it must be for this operation. To start splunk, run "splunk start". (02) CLI command to check the bundle status also result in error splunk show cluster-bundle-status *Failed to contact the master. ERROR: Couldn't complete HTTP request: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshak*e failure The REST endpoint gives error curl -u admin:changeme -k https://localhost:8089/services/search/jobs -d"search=search *" curl: (35) SSL connect error ... View more