. ... View more

Revisited this question as i had a similar request come up again and pleasantly surprised to find new comments and additions!  Thanks again Nick! @sideview  ... View more

That lookup doesn't contain the descriptions field just titles of service and kpi. It may have before but it does not now. ... View more

lol its been more than 5 years! 😉 Network connectivity between the new hosts on new (at the time) ports wasn't fully enabled as such the communication between members and the cluster master did not work. If you see this error now (in the year 2020) this can be caused by other different issues such as the bundle being too large. ... View more

On your cluster master gui click the indexes tab and click bucket status. Have "select index" on "all". Any "in progress" or "pending" jobs? If so have a look at the time in fix up and the current status. This will give a hint as to why your rep/search factors are not met. ... View more

If its just for testing that you can try and put "| localop " prior to the command you want to run specifically on the search head. ... View more

Best practice. Separate you logs into different indexes. Apply normal restrictions at the indexing tier via srchIndexesAllowed in authorize.conf - https://docs.splunk.com/Documentation/Splunk/latest/Admin/Authorizeconf I'd suggest not using search filters for a non-metadata based field as they can be bypassed. ... View more

also remove the white space in your splunk_server. You've quoted whitespace that will make it fail even if you have distsearch to the master configured. ... View more

do you have distsearch to your cluster master? it wouldn't normally be enabled as it's not normally used as a search peer. ... View more

needs to be performed on the cluster master. Alternatively have dist search to the cluster master from your search heads but you'll have to add splunk_server= to the query. ... View more

My original search doesn't work anymore on recent versions of splunk as it doesn't like those escape characters in the evals. You can just use the fields in the base search : | rest /services/cluster/master/generation/master | fields pending_last_reason, search_factor_met, replication_factor_met ... View more

I don't think I said anywhere the specific number of nodes I had in my clusters. Can you show/quote me on what I said as I can't find it. ... View more

You can still do it without using clustering (it is actually easier). ... View more

Can't use 2 search heads in a search head cluster environment. Tip: you can have multiple instances on a single machine to bring the numbers up 🙂 ... View more

Thanks for the reply but I think you misunderstood the question. Given a fixed window (for testing purposes) I was able to see different numbers of events depending on when the search was run. Fundamentally incorrect results from a users point of view. The question really boiled down to “Why should the number of events returned change for the same period of time?”. Indextime is (as you said) ever increasing so you should never see new events with old indextimes being inserted way into the past (which is what we were seeing). My original use case was forwarding of critical security events. These needed to be sent once, only once, but atleast once. The nature of Splunk is that it is difficult to craft a search to forward these in a timely manner without the use of a real-time search (disabled on our environment). Seeing as a fair chunk of time has passed since the original question I changed my methodology and took the events in flight before it hit the indexing tier. It was too inconsistent for what we needed to do. For other users that wanted search time processing it can still be an issue for out of order / late events. So it still made for an interesting question on designing a search to handle this. Now that being said, I believe that there is/was hardware latency across a number of different hardware and software components in different indexers that was effecting my “time to search”. And as a result consistancy in results returned if it wasn’t taken into account. I did also find a bug with Multisite index clustering in which I could prove that hot events were not streaming to the other site until the bucket rolled to warm. This resulted in having to switch to using site0 on search head cluster members. I’d had previously logged tickets about trying to know what my “time to search” is as it differs from The basic _time - _indextime calculation. This stat doesn’t exist in any meaningful way as far as I know. It would be nice to know for example that within xx seconds of an event being indexed is it being returned in search events. ... View more

This works great thanks! I really appreciate the time you spent to get this working. This actually makes all the ldap dependent dashboards in the windows app work now 🙂 ... View more

oh thank god! 😄 ... View more

I understand it isn't supposed to be an exact location but from a users perspective there isn't THAT many large cities in Australia to compare it to the US (we literally have 7 major ones total.) I would have thought splunk would have used the standard TZ list. https://en.wikipedia.org/wiki/List_of_tz_database_time_zones Most users and Australians in general are on our eastern side of the country. Melbourne, Sydney, Brisbane (12 million) vs Hobart (200k). As such as have people that not know if Hobart has the same Daylight savings as all those 3 (hint, they don't), so it isn't clear to people that they are selecting the right thing, or not but choosing Hobart. Which is more of a big town than a city 😉 Remember, Splunk is a time series index so time is VERY important, almost the most important thing. Set in events and also via the UI. Getting it wrong changes the perspective of how the data is presented. We've had numerous people keep asking. Should I be selecting Hobart? Where is Melbourne or Sydney? Its been an ongoing joke that it hasn't been updated in the past 5 years. It is very embarrassing when we have to tell managers this software we're paying $10's of millions for each year for can't have an official timezone included in it. If it is deemed officially as too hard and the solution is "just use hobart" then that is fine. I'll just customize every search head to include the code fix myself. ... View more

I'd put in an enhancement request ( SPL-148763 ) and asked our account manager to escalate it 🙂 ... View more

6 standard Clustered search heads not enterprise security. Data is about 5-10 different kvstores of various sizes. Contents per kv store is around 2-3 million rows. So not huge really. All data is valid as they do upserts and total rewrites (append=false). The only way to release that test kvstore's 20gb was deletion of EVERYTHING inside a collection and then taking a cluster member and removing all the kvstore information from it and resyncing. The only possible way I can see this working is the migration from one kvstore to another. Fully empty that kvstore. Copy back and then delete the temporary one. Resync all members again. Redeligate the kv store master to one of the resynced members. Resync the former master. ... View more