I am working on a script to thaw frozen buckets. Part of my script is to validate that the selected buckets are valid. We have an index cluster that currently just freeze to a shared path. I run /opt/splunk/bin/splunk check-integrity -bucketPath /some/path/to/bucket to validate the buckets. I noticed that some of my buckets have a trailing digit at the end. They look like rb_12345678_12345678_1234_GUID-0 . There are others that run up to -3 When the check-integrity command runs it reports: Constraints given leave no buckets to operate on If I rename that bucket (in a different path) to remove the -0 I get a valid bucket response: Total buckets checked=1, succeeded=1, failed=0 I do not see -0 directories in my warm or cold directories. I'm guessing that since we are freezing to a shared path that Splunk is appending a -digit to the end of the frozen bucket name as not to overwrite something that is already there. This would make me believe that I could ignore the -# buckets IF I have a corresponding bucket that does not have the extra -#. I would also want to eventually purge the extra buckets. If I'm missing a normally named bucket should I move to rename one -# bucket to make it a "real" bucket? ... View more

We have a requirement to collect data from testing enclaves (that have copies of production devices) to our primary Splunk environment. I have event data going to a separate index through a heavy forwarder. What I am concerned about is the Internal index. I will need to track if I have Splunk UFs on clients in the enclave. Is there a way to mark or tag the data coming through the Heavy Forwarder to indicate that it is coming from that testing environment? ... View more

I am working on securing our Splunk environment. I'm starting with the Splunk Web parts. I have a few non-clustered Search Heads and they went fine with no issues. I'm adding the SSL settings to the $SPLUNK_HOME/etc/system/local/web.conf. When I do the same thing to my Search Head Cluster the Splunk Webs do not respond. Looking in the splunkd.log I can see issues with connecting to each other. I can't find any of the documentation for securing the Search Head Cluster. I am following the Splunk SSL Best Practices .conf session for most of this (can't post links yet). They don't really discuss the SHC side of things. I was building certificates for each SHC member with a common Subject Alternative Name as well as the host name. Figuring on the Splunk Web parts would use common name and then I could set them up to use the host name part of the certificate to use other Splunk parts (splunkd, kvstore). Does anyone have any good documentation or guidance on how to do secure Search Head Clusters? With it being a Cluster can I only do this through apps pushed out through the Deployer? Should I be using a common certificate for Splunk Web? ... View more

We were having trouble with one of the Search Head cluster members. We went in to ensure that all of the shclustering pass4SymmKey values were correct. After restarting the Deployer splunkd service we found that the value for the pass4SymmKey did not get encrypted. Other pass4SymmKeys are being encrypted (clustering and general) but not the shclustering. This has happened on both 6.4.0 and 6.4.1 versions. We double checked that the field was listed in the encrypt fields and it is. I'm figuring we are missing something. Any suggestions? ... View more

I am working on getting Splunk secured with certificates. We have a requirement to ensure the integrity of our audit logs as they are transported to Splunk. This would mean that I need to use SSL/TLS between the Forwarders and the Indexers. When I read the SSL documentation, it wants a cert file and a password in the config settings for each forwarder. This might work for a few forwarders, but we are planning on doing 1,000+ Windows clients, so this would become a management issue. I know I can force the clients to request certificates from our Enterprise CA through GPOs without much problem. Is there a way to tell the Universal Forwarders to use the machine/host certificates without having to manually set the certificate settings or even using one certificate for all the UFs? ... View more

We are building out our new Splunk environment and I wanted to see what everyone else might be using to monitor their health across the environment. We are going to be distributed, so monitoring one server isn't going to work. What is making me question this is that I found a .conf slide deck that mentioned all three of these apps. What has been your experience with using these? Is there more overlap than unique features that one isn't as useful, or are they all fairly useful at giving us a good picture of our health? ... View more