We have 8-node indexer multi-site cluster with total of 2 sites (each site with 4 indexers) Site 1 - SITE1IDX01 SITE1IDX02 SITE1IDX03 SITE1IDX04 Site 2 - SITE2IDX01 SITE2IDX02 SITE2IDX03 SITE2IDX04 Replication Settings - site_replication_factor=origin:2,total:3 site_search_factor=origin:1,total:2 Whenever there are connectivity issues / glitches between the two sites, we start to notice below failures - WARN BucketReplicator - Failed to replicate Streaming bucket to guid=**** host=***** s2sport=**** Read timed out after 60 sec (between Site 1 & Site 2 indexers) And these failures have a cascading effect - Connectivity errors between servers on Site 1 & Site 2 ---> this affect replication of hot streaming buckets between the 2 sites ---> which puts a pressure on splunk replication queue ---> replication is part of index queue and hence indexing queues get filled ---> since indexing queues are supposed to process incoming ingestion, being full they drop the ingestion traffic. Need help over these points - We assume that such errors will still effect even if you put cluster into maintenance mode. How do we then support network related maintenance activities for Splunk Multi-Site cluster ? How can we make multi-site cluster more resilient to not end up blocking index queues due to underlying network glitches between the indexers on two sites. ... View more

I have an index for which I desire to retain 45 days of events . I have multiple values set under indexes.conf for the same :- frozenTimePeriodInSecs = 3888000 maxDataSize=auto_high_volume maxHotBuckets = 10 maxWarmDBCount = 15 maxTotalDataSizeMB = 512000 Although desired retention is 45 days but index is only able to retain 21 days of data. Please be noted that I do understand meaning of all such parameters and how they may effect overall retention. But I am looking for some method / search query which can show/evaluate - what/which parameter amongst all above is taking precedence (based on ingested volume, bucket count reach, etc) for an index and rolling buckets to age and hence letting only 21 days of data to retain.  ... View more

My Ad-hoc searches getting auto-cancelled randomly. I am running them with admin privileges. There's no problem with RAM. DO not have any limits.conf or authorize.conf under system/local ... View more

Environment is single-site with 3 indexers and each indexer with parallelIngestionPipelines = 1(under /opt/splunk/etc/system/default/server.conf). Does above imply that I have TOTAL "three" ingestion pipelines into my environment (one from each indexer) ? If yes, with the setting of "maxHotBuckets=10" , would that mean I will have total limit of 30 hot buckets, considering 10 belonging to each ingestion pipeline ? ... View more

I recently introduced a few parameters around different buckets like hot, warm, cold, etc. Now I need to see if the buckets are rotating based on the values I provided, and I am trying to find an effective search to help. Parameters I recently introduced and want to validate based on bucket size and movement are: maxDataSize = auto_high_volume maxHotBuckets = 10 maxWarmDBCount = 15 maxTotalDataSizeMB = 512000 frozenTimePeriodInSecs = 7776000 ... View more

Let's say if I have 4 indexers at one site 'AB' and 4 indexers at another site 'CD'(DR site). site_replication_factor=origin:2,total:3 site_search_factor=origin:1,total:2 Question :1 I understand from this document that in a situation where 3 of my indexers go down at 'AB' site , my 4th indexer will keep on ingesting the data and would keep copies in reserve state to be distributed when other indexers come back in place? Please confirm. Question :2 What if all my 4 indexers go down at 'AB' site ..how would ingestion be managed then ? Would cluster master automate the data ingestion to DR site 'CD' indexers ? Question :3 Since I have site_replication_factor of origin:2, total:3 and let's say two indexer machines at 'AB' site, both holding copy of same bucket goes down. Now, in this situation all copies(two) for a specific bucket become unavailable at site 'AB', then would cluster master instruct to receive a copy from DR site 'CD' and get that copied to 2 running indexers at 'AB' site ? ... View more