1) Running the following search: host=rbal* OR host=test index=winevent_s earliest=5/18/2020:7:3:0 latest=5/18/2020:7:5:0 | stats values(_sourcetype) values(sourcetype) values(host) values(source) 2) Produces the following values for the given fields (field: value): values(_sourcetype): WinEventLog:Security values(sourcetype):wineventlog values(host): EBRIAN values(source): WinEventLog:Security Note: The _sourcetype value of WinEventLog:Security is the original value of the sourcetype before it is renamed to wineventlog. 3) Given the above information, when we run the following search we see a count of 73 results: host= rbal index=winevent_s earliest=5/18/2020:7:3:0 latest=5/18/2020:7:5:0 sourcetype=WinEventLog OR sourcetype=XmlWinEventLog | stats count 73 results note: normally, field values are case in-sensitive so searching sourcetype=WinEventLog / sourcetype=wineventlog should be equivalent. However when we are dealing with sourcetype rename’s the target name is case sensitive, unless you “OR” in another sourcetype, then the renamed sourcetype is not case sensitive. 4) To illustrate this: a) we take a sourcetype that has been renamed (ie: WinEventLog:Security has been renamed to wineventlog) >>> etc/apps/Splunk_TA_windows/default/props.conf [WinEventLog:Security] rename = wineventlog and run a search using a different case (camel case) for the sourcetype value and we get 0 results where we should expect to see 73 results. host=rbal index=winevent_s earliest=5/18/2020:7:3:0 latest=5/18/2020:7:5:0 sourcetype=WinEventLog | stats count ** 0 results** b) if we OR another sourcetype to the SPL we get results now and the value of the sourcetype is reported in the interesting fields as the target sourcetype name “wineventlo” host=rbal index=winevent_s earliest=5/18/2020:7:3:0 latest=5/18/2020:7:5:0 sourcetype=WinEventLog OR sourcetype=XmlWinEventLog | stats count 73 results c) What is odd is that running the same search with either one of the sourcetypes by themselves or removing the host produces no results: host=rbal index=winevent_s earliest=5/18/2020:7:3:0 latest=5/18/2020:7:5:0 sourcetype=WinEventLog | stats count 0 results host=rbal index=winevent_s earliest=5/18/2020:7:3:0 latest=5/18/2020:7:5:0 sourcetype=XmlWinEventLog | stats count 0 results without host index=winevent_s earliest=5/18/2020:7:3:0 latest=5/18/2020:7:5:0 sourcetype=WinEventLog OR sourcetype=XmlWinEventLog | stats count ** 0 results** d)Even stranger, if we add an asterisks to the end of the host value, it produces 0 results. host=rbal* index=winevent_s earliest=5/18/2020:7:3:0 latest=5/18/2020:7:5:0 sourcetype=WinEventLog OR sourcetype=XmlWinEventLog | stats count 0 results e) using the asterisks with the correct matching case for the sourcetype value (lowercase) produces the expected values host=rbal* index=winevent_s earliest=5/18/2020:7:3:0 latest=5/18/2020:7:5:0 sourcetype=wineventlog | stats count 73 results This behavior is similar to old JIRA SPL-122984 “Searching renamed sourcetype is case-sensitive” which was documented as a known issue in 7.0 and 7.1 (https://docs.splunk.com/Documentation/Splunk/7.1.0/ReleaseNotes/KnownIssues). Also a new JIRA # however it doesn’t look like the 7.2+ known issues an example of a sourcetype rename: etc/apps/Splunk_TA_windows/default/props.conf [WinEventLog:Security] rename = wineventlog props.conf spec: rename = * Renames [] as at search time * With renaming, you can search for the [] with sourcetype= * To search for the original source type without renaming it, use the field _sourcetype. * Data from a renamed sourcetype only uses the search-time configuration for the target sourcetype. Field extractions (REPORTS/EXTRACT) for this stanza sourcetype are ignored. * Default: empty string ... View more

VolumeManger trim operations are not compatible with S2 and can lead to unpredictable behavior.We should not be mixing S2 indexes and non-S2 indexes on the same volume with maxVolumeDataSizeMB. After separate the indexes to a different volume, the volume manager begins to trimming the exceeding data. [volume:s3] storageType = remote path = s3:/...... #separate s3 indexes and non-s3 indexes for maxVolumeDataSizeMB to work on non-s3 indexes [volume:s3_indexes] path=$SPLUNK_DB maxVolumeDataSizeMB = 7000 [volume:test_indexes] path = $SPLUNK_DB maxVolumeDataSizeMB = 700000 [DA-ESS-AccessProtection] coldPath = volume:test_indexes/AccessProtection/colddb homePath = volume:test_indexes/AccessProtection/db thawedPath = $SPLUNK_DB/AccessProtection/thaweddb repFactor = auto frozenTimePeriodInSecs = 3456000 enableDataIntegrityControl = true maxDataSize = 200 ... [floating-point-index] remote.s3.encryption = sse-kms remotePath = volume:s3/floating-point coldPath = volume:s3_indexes/floating-point/colddb datatype = metric homePath = volume:s3_indexes/floating-point/db maxTotalDataSizeMB = 512000 repFactor = auto thawedPath = $SPLUNK_DB/floating-point/thaweddb maxDataSize = 200 ... View more

Bobby. that is not feasible just out of the Box. My suggestion will be to contact your Sales engineer. ... View more

Starting 7.3 Splunk ensures that bucket copies are not evicted on target indexers after the hot bucket rolls to warm and is uploaded by source. However for buckets that are already warm and that are being converted to s2 and are being uploaded by either the source or the targets, they would not be evicted on upload irrespective of the version. buckets would only be evicted eventually when there is cache pressure ... View more

Starting ing splunk 7.3.1 splunk ensures that bucket copies are not evicted on target indexers after the hot bucket rolls to warm and is uploaded by source. However for buckets that are already warm and that are being converted to s2 and are being uploaded by either the source or the targets, they would not be evicted on upload irrespective of the version. buckets would only be evicted eventually when there is cache pressure ( cache is full). ... View more

Hist to add some more information on bootstrap. For large deployments, customer get concerned while running bootstrap when env have millions of buckets. As you know, bootstrapping would ensure that buckets which are already present on cluster would not be created again on the cluster. bootstrapping would just list all the buckets on S3 and would then create the buckets which are not present on the cluster. It is usually quick as well. Hence if the customer is only missing few buckets on the cluster, we can initiate bootstrapping and it would create these buckets. Noe for the Question : List all buckets on s3 for 7 million buckets -> is that still ok / fairly safe / quick? if we do want to discover these buckets, bootstrapping is the only option currently. it is not supported per index. the entire operation is detached from the usual operations of CM - it is safe and quick as well. ... View more

For standalone indexer, this command is not needed. WHen you restart the indexer it will discover teh buckets on remote. ... View more

Splunk has attribute max_cache_size that set the limit for cacahemanager server.conf [cachemanager] eviction_padding = 10% of disk Space on Partition (in bytes) Refer: https://docs.splunk.com/Documentation/Splunk/7.1.4/Admin/Serverconf max_cache_size=0 Given that max_cache_size is not accounting for hot data, it is likely that space usage is going to be above max_cache_size.o limit the disk usage for both hot data + cached data, you can use minFreeSpace to restrict total disk usage. However, you need to have reasonable total space so hot buckets are not overcrowding the cache. Splunk component CacheManager when put in debug mode provides the stats for key attribute that contribute to the size of Cacahemanager. 02-05-2020 21:09:41.898 +0000 DEBUG CacheManager - The system has freebytes=75354976256 with minfreebytes=5242880000 cachereserve=5368709120 totalpadding=10611589120 buckets_size=313491456 maxSize=314572800 "The system has freebytes=" > freeBytes " with minfreebytes=" > minFreeBytes " cachereserve=" > evictionReservedBytes " totalpadding=" > minFreeBytes + evictionReservedBytes " buckets_size=" > buckets_size " maxSize=" > maxSize Now here is some information on each of this attribute. 1) " maxSize=" << _maxSize; Now this comes from *max_cache_size = * * Specifies the maximum space, in megabytes, per partition, that the cache can occupy on disk. If this value is exceeded, the cache manager starts evicting buckets. * A value of 0 means this feature is not used, and has no maximum size. * Default: 0 2) " buckets_size=" << buckets_size Now this is the total size of the buckets calculated. 3) " totalpadding=" << minFreeBytes + evictionReservedBytes we can see here that totalpadding=10611589120 = (minfreebytes+cachereserve) = (5242880000+5368709120) From the below, we can see that buckets_size can get bigger than maxSize for a temporary period but eventually we will slash it down to maxSize(max_cache_size) OR lesser. Here in this example, we have As you can see how buckets_size is varying with maxSize wrt time. Once we have buckets_size growing above maxSize, the CacheManager will ensure that we evict something and get back within maxSize. server.conf [cachemanager] max_cache_size = 300 ... View more

Yes. avoiding DMC(Monitor Console) on CM is an easy way to prevent this issue. Using some hacking is may also be possible to prevent the issue ieven if you have no other option but to have MC on CM. ... View more

For Smartstore the Cluster Master has component CMMasterRemoteStorageThread which runs every remote_storage_retention_period (defaults to 15 minutes) to check if there are buckets in remote storage that needs to be frozen in the cluster i)Component CMMasterRemoteStorageThread runs a search on all peers to retrieve the list of remote indexes with frozenTimePeriodInSecs, maxGlobalDataSizeMB and maxGlobalRawDataSizeMB information. This is also tracked in splunkd.log 05-21-2019 02:24:00.292 +0000 INFO CMMasterRemoteStorageThread - retrieving remote indexes info with search=| rest services/data/indexes datatype=all f=title f=frozenTimePeriodInSecs f=maxGlobalDataSizeMB f=remotePath f=disabled| search remotePath!="" AND disabled!=1| dedup title| fields title frozenTimePeriodInSecs maxGlobalDataSizeMB So the rest call that gets list of indexes is | rest services/data/indexes datatype=all f=title f=frozenTimePeriodInSecs f=maxGlobalDataSizeMB f=remotePath f=disabled| search remotePath!="" AND disabled!=1| dedup title| fields title frozenTimePeriodInSecs maxGlobalDataSizeMB ii)It then runs a search on all the peers to retrieve the list of the warm buckets that need to be frozen based on the frozenTimePeriodInSecs, maxGlobalDataSizeMB and maxGlobalRawDataSizeMB thresholds. The splunk.log has entry like 05-21-2019 02:24:00.846 +0000 INFO CMMasterRemoteStorageThread - Will initiate retrieving the list of buckets to be frozen for remote storage retention for index=_internal with frozenTimePeriodInSecs=2592000 and maxGlobalDataSizeMB=0 05-21-2019 02:24:00.846 +0000 INFO CMMasterRemoteStorageThread - retrieving the list of buckets to be frozen for remote storage retention for index=_internal with search=| dbinspect index=_internal cached=true timeformat=%s| search state=warm OR state=cold| search modTime != 1| stats max(endEpoch) AS endEpoch BY bucketId| sort -endEpoch| search endEpoch<1555813440| fields bucketId, endEpoch So here is the |dbinspect search to retrieve the bucket to be frozen (slightly modified version from the splunkd.log in (ii) | dbinspect index=* | join index [|rest /services/data/indexes| eval index=title | table index frozenTimePeriodInSecs ] | eval toNow=now()-endEpoch | convert num(toNow) | convert num(frozenTimePeriodInSecs) | convert ctime(endEpoch) AS endEvent | convert ctime(startEpoch) AS startEvent | eval shouldBeFrozen=if( ( state!="hot" AND state!="thawed" ) AND toNow>frozenTimePeriodInSecs,"yes","no") | table splunk_server index path id state startEvent endEvent shouldBeFrozen toNow frozenTimePeriodInSecs To debug the issue where bucket is not being deleted basked on retention suggestion would be to check if (i) and (ii) search is returning list of indexes and list of the bucket to be frozen. Based on my experience that if the Monitoring console is enabled on the Cluster Master that change the defaukt search group in distsearch.conf and that could casue these seraches to not return teh expected result. Check if the Monitor Console is enabeld on CM. ... View more

Before answering this question we need to understand cachemanager_upload.json. This file resides in SPLUNK_HOME/var/run/splunk/cachemanager_upload.json, and this is used to migrate bucket to smart store. the sample below show the list of the bucket to be uploaded to the remote store. cat ./var/run/splunk/cachemanager_upload.json | sed 's/,/\n/g' {"bucket_ids":["bid|_audit~100~761A77A2-6676-4BF9-83CD-1CB243ED61BF|" "bid|_audit~103~EDEAC3E5-E0B3-45B9-84B3-A1E087035148|" "bid|_audit~104~761A77A2-6676-4BF9-83CD-1CB243ED61BF|" "bid|_audit~108~EDEAC3E5-E0B3-45B9-84B3-A1E087035148|" "bid|_audit~110~761A77A2-6676-4BF9-83CD-1CB243ED61BF|" "bid|_audit~124~761A77A2-6676-4BF9-83CD-1CB243ED61BF|" AS part of the migration, DDM(DatabaseDirectoryManager) pre-registers the buckets to cache manager via bulk register i.e. it writes the buckets to cachemanager_upload.json at "$SPLUNK_HOME/var/run/splunk/". This file maintains the list of buckets that need to be uploaded to remote storage. Whenever we need to upload a bucket, we flush the bid to this file, so that in scenarios where splunk crashed or got restarted before the upload, we would resume the upload process from where we left off and after the upload is done, we would remove the entry from this file, so that in-memory state is in sync with state on the disk. This file can also be manually updated to upload the bucket to remote using bulk_reigter.Hit caches man's endpoint for a bucket requesting it to add the bucket to cachemanager_upload.json. curl -k -u <user>:<passwd> -X POST https://<uri>/services/admin/cacheman/_bulk_register -d cache_id="<cacheId>" example: curl -k -u admin:changeme -X POST https://localhost:10041/services/admin/cacheman/_bulk_register -d cache_id="bid|taktaklog~1~F7770FEB-F5A6-4846-A0BB-DDC05126BBF6|" Here is an example to upload multiple buckets curl --netrc-file nnfo -k -X POST https://localhost:8089/services/admin/cacheman/_bulk_register -d cache_id="bid|nccihub~17~024011E7-E61E-45CE-82DE-732038D5C276|" -d cache_id="bid|nccihub~22~024011E7-E61E-45CE-82DE-732038D5C276|" -d cache_id="bid|nccihub~28~024011E7-E61E-45CE-82DE-732038D5C276|" -d cache_id="bid|nccihub~34~024011E7-E61E-45CE-82DE-732038D5C276|" -d cache_id="bid|nccihub~12~E646664A-D351-41E4-BBE7-5B02A08C44C9|" -d cache_id="bid|nccihub~17~F876C294-3E3E-488A-8344-16727AC34C52|" -d cache_id="bid|nccihub~17~E646664A-D351-41E4-BBE7-5B02A08C44C9|" When the bulk register rest endpoint is called it add the bucket to cachemanager_upload.json and the subsequent restart of the indexer would upload the bucket to the remote store. You may hit a scenario where the customer is planning to bring multiple TB of the legacy standalone bucket to Smarstore cluster deployment. When the indexer is first enabled for the smart store, the bucket would migrate to the remote store. When migration is finished Splunk would create $SPLUNK_HOME/var/lib/splunk//db/.buckets_synced_to_remote_storage , and any newly added bucket would not get uploaded. For the requirement to add "multiple TB of the legacy standalone bucket to Smarstore cluster deployment", we should use migration and not bulk_register. In this case, you can remove $SPLUNK_HOME/var/lib/splunk//db/.buckets_synced_to_remote_storage and restart the indexer and the bucket will be re-uploaded. ... View more

Just to complete this discussion. When remote.s3.supports_versioning = true , we iterate over all versions of an S3 object (file) and remove all versions. Otherwise, we do a simple remove on the object. This means that if set to true, all versions will be removed and the object contents are irretrievable. If set to false, the behavior is as follows: 1) if bucket versioning is disabled, the object is simply gone forever; 2) if bucket versioning is enabled, the "remove object" operation simply puts a delete marker on top. Keep in mind that the delete marker is not explicitly put by us. Whether there will be a delete marker depends on whether bucket versioning is enabled and on the method of removal. There is nothing in Splunk about versioning. It's at the storage level. Splunk only does 1) "simple" object removal or 2) removal of all versions of an object, depending on the configuration. ... View more

AT the high level these are the steps: i)Once the bucket is rolled to warm, "remote_storage_upload_timeout" timer is started on target peers, it is registered with the CacheManager on source, bucket is optimized, it is uploaded to remote storage by source and bucket registration ends. Once the source peer uploads the bucket to remote storage it notifies the target peers that bucket has been uploaded to remote storage. After the target peers receive that message from source peer, they cancel the registration with cachemanager, mark the bucket as stable and evict the bucket. Below is a breakdown of above steps with corresponding log messages - Source peer rolls the bucket 05-15-2019 15:46:44.844 +0000 INFO HotBucketRoller - finished moving hot to warm bid=perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9 idx=perfmon from=hot_v1_468 to=db_1557754467_1557734922_468_FA94F613-032D-4C8E-9D04-EFA3F5E923C9 size=397975552 caller=lru maxHotBuckets=10, count=11 hot buckets,evicting_count=1 LRU hot s ii)Done key received on target peer(which means we are done with replication from source**) 05-15-2019 15:46:44.879 +0000 INFO S2SFileReceiver - event=onDoneReceived replicationType=eJournalReplication bid=perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9 05-15-2019 15:46:44.879 +0000 INFO S2SFileReceiver - about to finalize from close bid=perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9 iii)Target peer starts the timer "remote_storage_upload_timeout", so that if it doesn't hear from the source peer until timer expires then it can start the upload of the bucket and also rolls the bucket from hot→warm at its end.** INFO CMSlave - bid=perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9 added so this target peer can assume responsility of upload later 05-15-2019 15:46:44.884 +0000 INFO S2SFileReceiver - event=rename bid=perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9 from=/opt/splunk/var/lib/splunk/perfmon/db/468_FA94F613-032D-4C8E-9D04-EFA3F5E923C9 to=/opt/splunk/var/lib/splunk/perfmon/db/db_1557754467_1557734922_468_FA94F613-032D-4C8E-9D04-EFA3F5E923C9 05-15-2019 15:46:44.884 +0000 INFO CMSlave - bid=perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9 Transitioning status from=StreamingTarget to=Complete for reason="hot success (target)" iv)Meanwhile, source starts upload of bucket since it has finished optimize/repair process for the bucket. It also saves the state of the files in the bucket directory locally by writing to the file "cachemanager_local.json"** 5-15-2019 15:46:54.718 +0000 INFO DatabaseDirectoryManager - cid="bid|perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9|" uploading the bucket to remote storage since optimize/repair process has completed successfully 05-15-2019 15:46:54.723 +0000 INFO CacheManager - action=upload, cacheId="bid|perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9|", status=attempting 05-15-2019 15:47:00.786 +0000 INFO CacheManager - action=upload, cacheId="bid|perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9|", status=succeeded, elapsed_ms=6063 Corresponding entry in audit.log for the bucket upload 05-15-2019 15:46:54.723 +0000 INFO AuditLogger - Audit:[timestamp=05-15-2019 15:46:54.723, user=n/a, action=local_bucket_upload, info=started, cache_id="bid|perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9|", prefix=reedexpo/perfmon/db/4d/e7/468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9/guidSplunk-FA94F613-032D-4C8E-9D04-EFA3F5E923C9][n/a] 05-15-2019 15:47:00.817 +0000 INFO AuditLogger - Audit:[timestamp=05-15-2019 15:47:00.817, user=n/a, action=local_bucket_upload, info=completed, cache_id="bid|perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9|", local_dir="/opt/splunk/var/lib/splunk/perfmon/db/db_1557754467_1557734922_468_FA94F613-032D-4C8E-9D04-EFA3F5E923C9", kb=382940, elapsed_ms=6095][n/a] NOTE: "cachemanager_local.json" is a local file that resides in db directory for warm buckets. It is used to maintain the state of what files are present locally in the disk. We update this file when we are either about to upload the bucket or we download the bucket contents when a search opens the bucket or we cancel the upload. The contents of this file looks something like this - v)Source peer reports upload status to replication target/s** 05-15-2019 15:47:00.817 +0000 INFO CMSlave - bid=perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9 upload status being reported to the replicated targets 05-15-2019 15:47:00.818 +0000 INFO CMRepJob - running job=CMReportBucketInStableStorageJob bid=perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9 ot_guid=5B2CABAA-22E8-4B25-AE31-C089D69FE13D ot_hp=INDEXER:8089 vi)Target peer/s receive the notification from source peer and update their metadata with remote storage metadata by checking if the bucket is present on remote storage.** 05-15-2019 15:47:00.844 +0000 INFO CMSlave - bid=perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9 reported to be on remote storage by upload peer, will confirm it is present by checking the remote storage 05-15-2019 15:47:00.871 +0000 INFO DatabaseDirectoryManager - cid="bid|perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9|" found to be on remote storage 05-15-2019 15:47:00.871 +0000 INFO IndexerIf - Asked to update bucket manifest values, bid=perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9 05-15-2019 15:47:00.903 +0000 INFO DatabaseDirectoryManager - idx=perfmon Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/perfmon/db', pendingBucketUpdates=0 . Reason='Updated metadata of bucket with remote storage metadata, bid=perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9' NOTE: This step of target getting notified by source about uploading the bucket has been made optional in recent versions. By default this feature is turned off. So, targets won't receive the notification from source that it has uploaded the bucket and target will eventually check the bucket on remote storage after remote_storage_upload_timeout and if its present then just marks the buckets stable as part of cancelled upload. Below is the configuration which is introduced to make this feature optional - report_remote_storage_bucket_upload_to_targets = * Only valid for 'mode=slave'. * For a remote storage enabled index, this attribute specifies whether the source peer reports the successful bucket upload to target peers. This notification is used by target peers to cancel their upload timers and synchronize their bucket state with the uploaded bucket on remote storage. * Do not change the value from the default unless instructed by Splunk Support. * Default: false vii)Now, target peer cancels the registration of bucket with cachemanager, mark the bucket as stable and then evict the bucket locally** 05-15-2019 15:47:00.905 +0000 INFO CMSlave - bid=perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9 removed from the replicatedBucketsUploadTimeout map 05-15-2019 15:47:00.912 +0000 INFO CacheManager - cancel registering new cacheId="bid|perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9|" for search sid=bid|perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9| 05-15-2019 15:47:00.912 +0000 INFO CacheManager - Making cacheId="bid|perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9|" stable as part of cancelled upload The corresponding entry in audit.log entry which logs eviction of bucket - 05-15-2019 15:47:00.952 +0000 INFO AuditLogger - Audit:[timestamp=05-15-2019 15:47:00.952, user=splunk-system-user, action=local_bucket_evict, info=completed, cache_id="bid|perfmon~468~FA94F613-032D-4C8E-9D04-EFA3F5E923C9|", kb=389622, elapsed_ms=15, files="strings_data,sourcetypes_data,sources_data,hosts_data,lex,tsidx,bloomfilter,journal_gz,deletes,other"][n/ the "files" evicted in the above log entry. If you are evicting all the files in bucket directory, that usually means that target is evicting the bucket because source or some other peer has already uploaded the bucket. You will encounter other local_bucket_evict logs in audit.log which will have different "files" to be evicted, which can be due to other reasons covered later in this page(most commonly "deletes" files, which are evicted due to primary changes). ... View more

If you are testing I can join zoom and look at what you are seeing ( around 11AM PSt) ... View more

for S2 one of the recommendations is to set RF=sf ... View more

I think it's highly unlikely that bucket will get uploaded twice. Why do you think that bucket was uploaded twice? ... View more

maxvolumedatasizeMB is ignored and can lead to disk filling up unexpectedly. Depending on version of splunk, it will either start evicting evictable buckets at the eviction_padding threshold or it wont evict and then eventually pause indexing when it hits minfreespace. Advice is not to have s2 indexes and non-s2 indexes in the same volume with maxvolumedatasizeMB as it's unlikely either of the above behaviors are desired. They should keep non-s2 and s2 indexes in separate volumes. ... View more