Revisited this question as i had a similar request come up again and pleasantly surprised to find new comments and additions!  Thanks again Nick! @sideview  ... View more

That lookup doesn't contain the descriptions field just titles of service and kpi. It may have before but it does not now. ... View more

lol its been more than 5 years! 😉 Network connectivity between the new hosts on new (at the time) ports wasn't fully enabled as such the communication between members and the cluster master did not work. If you see this error now (in the year 2020) this can be caused by other different issues such as the bundle being too large. ... View more

On your cluster master gui click the indexes tab and click bucket status. Have "select index" on "all". Any "in progress" or "pending" jobs? If so have a look at the time in fix up and the current status. This will give a hint as to why your rep/search factors are not met. ... View more

If its just for testing that you can try and put "| localop " prior to the command you want to run specifically on the search head. ... View more

Best practice. Separate you logs into different indexes. Apply normal restrictions at the indexing tier via srchIndexesAllowed in authorize.conf - https://docs.splunk.com/Documentation/Splunk/latest/Admin/Authorizeconf I'd suggest not using search filters for a non-metadata based field as they can be bypassed. ... View more

also remove the white space in your splunk_server. You've quoted whitespace that will make it fail even if you have distsearch to the master configured. ... View more

do you have distsearch to your cluster master? it wouldn't normally be enabled as it's not normally used as a search peer. ... View more

needs to be performed on the cluster master. Alternatively have dist search to the cluster master from your search heads but you'll have to add splunk_server= to the query. ... View more

My original search doesn't work anymore on recent versions of splunk as it doesn't like those escape characters in the evals. You can just use the fields in the base search : | rest /services/cluster/master/generation/master | fields pending_last_reason, search_factor_met, replication_factor_met ... View more

I don't think I said anywhere the specific number of nodes I had in my clusters. Can you show/quote me on what I said as I can't find it. ... View more

You can still do it without using clustering (it is actually easier). ... View more

Can't use 2 search heads in a search head cluster environment. Tip: you can have multiple instances on a single machine to bring the numbers up 🙂 ... View more

Thanks for the reply but I think you misunderstood the question. Given a fixed window (for testing purposes) I was able to see different numbers of events depending on when the search was run. Fundamentally incorrect results from a users point of view. The question really boiled down to “Why should the number of events returned change for the same period of time?”. Indextime is (as you said) ever increasing so you should never see new events with old indextimes being inserted way into the past (which is what we were seeing). My original use case was forwarding of critical security events. These needed to be sent once, only once, but atleast once. The nature of Splunk is that it is difficult to craft a search to forward these in a timely manner without the use of a real-time search (disabled on our environment). Seeing as a fair chunk of time has passed since the original question I changed my methodology and took the events in flight before it hit the indexing tier. It was too inconsistent for what we needed to do. For other users that wanted search time processing it can still be an issue for out of order / late events. So it still made for an interesting question on designing a search to handle this. Now that being said, I believe that there is/was hardware latency across a number of different hardware and software components in different indexers that was effecting my “time to search”. And as a result consistancy in results returned if it wasn’t taken into account. I did also find a bug with Multisite index clustering in which I could prove that hot events were not streaming to the other site until the bucket rolled to warm. This resulted in having to switch to using site0 on search head cluster members. I’d had previously logged tickets about trying to know what my “time to search” is as it differs from The basic _time - _indextime calculation. This stat doesn’t exist in any meaningful way as far as I know. It would be nice to know for example that within xx seconds of an event being indexed is it being returned in search events. ... View more

This works great thanks! I really appreciate the time you spent to get this working. This actually makes all the ldap dependent dashboards in the windows app work now 🙂 ... View more

oh thank god! 😄 ... View more