Hello, I have an automated upgrade plan that does the following: Puts the cluster in maintenance mode splunk enable maintenance-mode Goes 1 by 1 on each of the 3 indexer peers and runs: splunk offline Extracts upgrade tar file to necessary location runs splunk start and accepts license and answers yes. repeats for the next peer Disables maintenance mode. I am trying to upgrade the peers without the end users seeing messages but unfortunately users see things like the following:   Unable to distribute to peer named X because peer has status=Down. Verify uri-scheme, connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available. ^ Even though the peer is Up according to the Cluster Master   Connection Refused for peer=X ^ Which seems like the search heads are sending queries or still have an established connection with the peer. I would expect the search head to know that a peer is down and not communicate with it till it indexes have been validates and deemed searchable.   Anyone have recommendation on making the indexer upgrade as seamless to the end user as possible? Things tried: adjusted the restart_timeout, quiet_period, and decomission_node_force_timeout on the cluster master Thanks, J ... View more

We are trying to identify how much of our data is impacted by the latest timestamp bug. I was wondering if there was a query that used regex to search _raw for events that have 2 digit years. This will greatly help us analyze the risk of doing a last minute production upgrade... ... View more

We are receiving the message "This instance is approaching the max concurrency searches" on our search head. Usually if we are hitting concurrency for a role, it will mention "...max concurrency for this role.". From what i've read, when the instance hits max concurrency, it has to do with the system wide concurrency setting Splunk calculates based on CPUs. Does the max_search_per_cpu need to be applied to the limits.conf of the indexers or the search heads? Thanks, ... View more

Hello, We have an indexer cluster that has a custom indexes.conf that specifies the volume path and retention of each index. However, it appears the _internal DB on each of the indexers it writing to /opt/splunk/var/lib instead of our custom volume where all the other indexes are writing. This is causing our /opt/splunk filesystem to fill up. Can someone explain why the indexers are not sending their _internaldb logs to /opt/splunk_hot even though we are referencing the volume in _internal? The ellipses are all the other indexes in the indexes.conf indexes.conf # VOLUME SETTINGS # One Volume for Hot and Cold [volume:primary] path = /opt/splunk_hot [volume:secondary] path = /opt/splunk_cold .... [_internal] repFactor = auto homePath = volume:primary/_internal/db coldPath = volume:secondary/_internaldb/colddb thawedPath = /opt/splunk_cold/_internaldb/thaweddb ... View more

Hello, If a user creates a dashboard with 10 searches and has 10 members of his team open up this dashboard, does this execute 100 searches (10 per each user of the dashboard refresh)? If so, how to we manage this in a scenario where this dashboard is needed often? We are thinking this behavior is responsible for high cpu as there are so many searches being executed each time the dashboard is opened. Thoughts? ... View more