The env was on 8.2.7.  the environment has 3 Node Search Head Cluster. Nodes upgraded from version 8.2.7 to  9.0.2. Post upgrade for one  SHC member the kvstore status was  DOWN.   ... View more

Is there any controls to limit the size of a user search? The use case is Splunk Cloud and limiting a search, if it downloads for example more than 10TB from SmartyStore to the cache. ... View more

Why avoid RAID5 on SSD when using SmartStore? ... View more

The issue is for the “PCI Compliance Posture” dashboard the View “Compliance Status History” is not showing data.  It just displays. It just displayed line ... View more

VERSION=8.0.6 ES version= version = 6.1.0 Splunk_DA-ESS_PCICompliance=4.1.0 Issue is for the “PCI Compliance Posture” dashboard the View “Compliance Status History” is not showing data. It just displays "Unable to find tag filtered" ... View more

Could you please help understand the DEBUG option for CacheManager to instigate eviction?     ... View more

We   would like to remove EBS volumes which were used for cold store and DM summary Docs is not overly clear on the recommended approach https://docs.splunk.com/Documentation/Splunk/7.3.4/Indexer/MigratetoSmartStore.  ... View more

After Smartstore was enabled for deployment the indexer's log's are flooded with messages like "INFO CacheManagerHandler - cache_id="ra|tto_uswest2_tomcatfrontend~39~4345D76C-80D6-4BC7-991F-EA835C2B892C|08281223-D92B-4A36-BCA0-83970376D322_tto_search_agupta13_NS2480590abee10f99" not found cache_id = ra|tto_uswest2_tomcatfrontend~39~4345D76C-80D6-4BC7-991F-EA835C2B892C|" What the best way to find teh bucket corresponding to report acceleration. ... View more

  there has been a huge spike in the number of uploads, resulting in many more failed uploads from throttling than we had before. It is currently unclear to me what caused this. Whether constant retries are underlying the huge spike, or some new data being uploaded have caused this. The bucket size has remained pretty constant, but the number of daily uploads has gone from about 80k to 4 million. Looking at some of s3 access logs, it seems like search objects are getting uploaded? Most of these uploads are for "ra" (Report Acceleration bucket) index=_internal host=<XXX> sourcetype=splunkd action=upload status=succeeded NOT cacheId=ra* | rex field=cacheId "bid\|(?<indexname>\w+)\~\w+\~" | timechart span=1m partial=f limit=50 per_second(kb) as kbps by indexname     index=_internal host=<XXX> sourcetype=splunkd action=upload status=succeeded NOT cacheId=ra* | rex field=cacheId "bid\|(?<indexname>\w+)\~\w+\~" | timechart span=1m partial=f limit=50 per_second(kb) as kbps by indexname       ... View more

This question has come up a few times, how does Splunk handle data integrity in large ES implementation. On Splunk docs, it states 'Data integrity control feature. SmartStore-enabled indexes are not compatible with the data integrity control feature, described in Manage data integrity in the Securing Splunk Enterprise manual.   As covered in https://docs.splunk.com/Documentation/Splunk/8.0.4/Indexer/AboutSmartStore ... View more

Cluster indexer across the site is configured with Smartstore. Each indexer has 6TB partition that is utilized by $SPLUNK_HOME+$SPLUNK_DB The Cache Manager is configured as below $SPLUNK_HOME/etc/system/default/server.conf [diskUsage] $SPLUNK_HOME/etc/system/default/server.conf minFreeSpace = 5000 $SPLUNK_HOME/etc/slave-apps/_cluster/local/server.conf [cachemanager] $SPLUNK_HOME/etc/system/default/server.conf evict_on_stable = false $SPLUNK_HOME/etc/slave-apps/_cluster/local/server.conf eviction_padding = 5120 $SPLUNK_HOME/etc/slave-apps/_cluster/local/server.conf eviction_policy = lru $SPLUNK_HOME/etc/slave-apps/_cluster/local/server.conf hotlist_bloom_filter_recency_hours = 720 $SPLUNK_HOME/etc/slave-apps/_cluster/local/server.conf hotlist_recency_secs = 604800 $SPLUNK_HOME/etc/slave-apps/_cluster/local/server.conf max_cache_size = 4096000 $SPLUNK_HOME/etc/slave-apps/_cluster/local/server.conf max_concurrent_downloads = 8 $SPLUNK_HOME/etc/slave-apps/_cluster/local/server.conf max_concurrent_uploads = 8 $SPLUNK_HOME/etc/slave-apps/_cluster/local/server.conf remote.s3.multipart_max_connections = 4 $SPLUNK_HOME/etc/slave-apps/_cluster/local/server.conf remote.s3.multipart_upload.part_size = 536870912 The indexer is showing partition 6TB partition  97% utilized , although it should not have crossed 4TB based on  max_cache_size = 4096000  Filesystem      1K-blocks       Used Available Use% Mounted ondevtmpfs         71967028          0  71967028   0% /devtmpfs            71990600          0  71990600   0% /dev/shmtmpfs            71990600    4219944  67770656   6% /runtmpfs            71990600          0  71990600   0% /sys/fs/cgroup/dev/nvme0n1p2   20959212    6812056  14147156  33% /none             71990600          0  71990600   0% /run/shm/dev/nvme1n1   6391527336 5864488560 204899848  97% /opt/splunktmpfs            14398120          0  14398120   0% /run/user/1003 Here is Debug entry for CacheManger {06-10-2020 19:32:42.604 +0000 DEBUG CacheManager - The system has freebytes=210838605824 with minfreebytes=5242880000 cachereserve=5368709120 totalpadding=10611589120 buckets_size=3069799919616 maxSize=4294967296000 06-10-2020 19:32:42.607 +0000 DEBUG CacheManager - The system has freebytes=210838536192 with minfreebytes=5242880000 cachereserve=5368709120 totalpadding=10611589120 buckets_size=3069799919616 maxSize=4294967296000 06-10-2020 19:32:46.502 +0000 DEBUG CacheManager - The system has freebytes=210850021376 with minfreebytes=5242880000 cachereserve=5368709120 totalpadding=10611589120 buckets_size=3069799919616 maxSize=4294967296000 06-10-2020 19:32:46.505 +0000 DEBUG CacheManager - The system has freebytes=210850172928 with minfreebytes=5242880000 cachereserve=5368709120 totalpadding=10611589120 buckets_size=3069799919616 maxSize=4294967296000 06-10-2020 19:33:06.727 +0000 DEBUG CacheManager - The system has freebytes=210255511552 with minfreebytes=5242880000 cachereserve=5368709120 totalpadding=10611589120 buckets_size=3069799919616 maxSize=4294967296000 Note From DEBUG observation   : freebytes=  210072649728  minfreebytes=  5242880000 cachereserve=  5368709120 totalpadding=  10611589120 buckets_size=  3069785296896     <<<<<<    3TB As calculated by cacahemanager maxSize=               4294967296000     <<<<<< configured 4TB limit The issue is cache has almost utilized 6TB of disk space but as per the calculation it shows usage of 3TB.  Due to this miscalculation Splunk is not evicting the buckets.         ... View more

We migrated almost all of our existing indexes from traditional indexes with separate warm and cold mount paths to smartstore a little under a year ago. It's all worked great, however for indexes with long term retention, buckets that were in the coldPath at the time of smartstore converstion continue to be stubbed out and localized from S3 back into the coldPath, while everything since conversion uses the warm path, as expected since that mount is the SPLUNK_DB definition used by the smartstore indexes. I want to re-map the SPLUNK_COLD path to use the same OS mount, but what is the supported way to do that with smartstore? From the documentation (https://docs.splunk.com/Documentation/Splunk/7.3.3/Indexer/Moveanindex) it sounds like you would normally manually copy the data from the old to the new path, and then re-map the variable, however with smart store does it work the same? Or is it just something like force clearing the smartstore cache on the OS mount I want to clear off, re-mapping the variable, and then new localization of buckets simple uses the re-mapped path? ... View more