I have a scheduled search that runs every minute, querying the previous one minute of time, and alerts if an event is found. Because the scheduled search interval is the same as the time period being searched, is it theoretically possible for an event to pop up in between the scheduled searches and not get alerted? If this is a possibility then it feels like the only option would be to search a slightly longer timeframe so that there is some overlap in the searches, but that could mean duplicate events. The other option would be to use a realtime search rather than a scheduled search., but in general I try to avoid  scheduled realtime searches. ... View more

I currently have 4 indexers as part of my Splunk deployment. I am upgrading these indexers with new hardware. I am going to join the 4 new indexers to the existing indexer cluster and then ultimately retire the 4 old indexers once the data is redistributed across the cluster. But, once all of the indexers are in the same cluster I seem to have two options (I think) for making sure that data is distributed across the new indexers: Option 1 Rebalance data across all 8 indexers...   splunk rebalance cluster-data -action start   ...and then retire the old indexers as normal. Option 2 Put each indexer in detention one by one and then retire in the following way, which as I understand it will move data off the indexer in the process...   splunk offline --enforce-counts   I've read the documentation around these topics, however Option 2 was mentioned to me in a previous post and so I just wanted clarification. Many thanks. Edit: Or, thinking about it some more, would I just use Option 1 to rebalance the data and then use Option 2 to remove the old indexers one by one? ... View more

I currently have a Splunk cluster that looks like this: Splunk CentOS Version Splunk Version Master 7.5 7.0.0 Forwarder 7.5 Universal Forwarder 6.6.3 Search Head 6.5 7.0.0 Indexer 1 6.5 7.0.0 Indexer 2 6.5 7.0.0 Indexer 3 6.5 7.0.0 Indexer 4 6.5 7.0.0   I have 4 new, better servers that I want to build as indexers to replace the 4 indexers that I currently have (while moving from Splunk 7.0.0 to Splunk 8.x in the process). It seems to me that I have two options: Option 1 Create a new indexer cluster with the 4 new indexers, so that I have an old indexer cluster and a new indexer cluster.  Send new data to the new indexer cluster only and in time the data on the old indexer cluster will age off and I can retire that cluster. During this time I will configure the search head to search across both indexer clusters so that the old and new data is searchable. Option 2 Add the new indexers to the existing indexer cluster, so that I have 4 old indexers and 4 new indexers in the same cluster. Allow the data to replicate across the 8 node cluster, and then retire the old indexers one by one. Leaving me with the 4 new indexers only. In both scenarios above the old indexers will be CentOS 6.x and the new indexers will be CentOS 7.x, but as I understand it this won't be a problem. Would anyone be able to offer any advice on which method would be best/easiest or point out any pitfalls or anything obvious that I may not have considered? ... View more

I currently have a Splunk cluster that looks like this: Splunk CentOS Version Splunk Version Master 7.5 7.0.0 Forwarder 7.5 Universal Forwarder 6.6.3 Search Head 6.5 7.0.0 Indexer 1 6.5 7.0.0 Indexer 2 6.5 7.0.0 Indexer 3 6.5 7.0.0 Indexer 4 6.5 7.0.0   I have 4 new servers that I want to build as indexers to replace the 4 indexers that I currently have (while moving from Splunk 7.0.0 to Splunk 8.x in the process). My first plan was to build the 4 new indexers and then join them to the current indexer cluster so that existing data could replicate between them and then I could retire the old indexers from the cluster. The problem with this idea is that my existing indexers are running CentOS 6.x and my new indexers will be running CentOS 7.x - as I understand it from reading documentation it is not possible to have different OS versions running in the same indexer cluster. As there is no easy way to upgrade from CentOS 6.x to CentOS 7.x, I'd rather avoid having to do this. So my next idea was to build the new indexers as a separate cluster - so that I have one cluster of CentOS 6.x indexers and one cluster of CentOS 7.x indexers - then send new data only to the new cluster and let old data eventually age off on the old cluster at which point I can retire it. During this time I will have a Search Head that searches across both clusters, so that old and new data is searchable. My question on that is: Will my Search Head running on CentOS 6.x be able to search across both a CentOS 6.x indexer cluster and a CentOs 7.x indexer cluster? Should I instead look at creating the second cluster of indexers and then manually migrating data between the old and new clusters? Although that seems like a harder process. How would you approach this?   ... View more