generally, I'd say ignore HTTP status code on their own. Some points around this: i) There could be intermittent problems in these codes which get resolved by a retry. ii)The codes become valuable when investigating other issues and that is why we are reporting iii)They could raise red flags if some HTTP status codes start appearing, such as 401's, for example. It wouldn't be practical to enumerate which codes and with which frequency indicates a problem. So if the frequency of the error for a bucket is in ore them, and for example, if it repeats for bucket's receipt.json.. 01-04-2019 15:47:13.049 -0800 ERROR S3Client - command=get transactionId=0x7f19c9452000 rTxnId=0x7f19c9431000 status=completed success=N uri=http://URI/splunk-data/_internal/db/3f/9c/5~2BD2B9DE-8F5E-48AD-B566-3BC1ADC9502F/receipt.json statusCode=404 statusDescription="Not Found" payload="" ... View more

Editing lookups now require role upload_lookup_files since Splunk Enterprise 7.3.x. This requirement has not been documented yet.When a user without role "upload_lookup_files" attempts to add or edit a lookup they are directed to the Splunk Pony - message Documentation JIRA s in works BUG SPL-176155:Editing lookups roles have changed Splunk 7.3.x and now require new roll upload_lookup_files (which is undocumented)" ... View more

The max_cache_size puts a cap on the the amount of storage in a single volume consumed by SmartStore cache. This allows customers to split the warm/cold storage consumption between SmartStore and non-SmartStore indexes on the same volume. Since there are no caps placed on non-smartStore indexes (module minFreeSpace), setting max_cache_size to non-zero can result in non-SmartStore indexes squeezing out space available for SmartStore cache. One usecase for max_cache_size in addition to migration usecase is for customers to reduce their local storage consumption in staged manner. By limiting storage available to the cache relative to the available space, customers can identify an optimal cache size before reducing the indexer count. ... View more

You can Pick a single region for the s3 bucket, both sites point to it. whichever has more data. although s3 supports cross region replication would mean that we can try out different nearest s3 site per cluster site,indexes.conf of the bundle that is pushed by master to the slaves would still need to use the same endpoint. editing the slave's indexes.conf manually to have different site is not recommended Make sure the endpoint includes the region specifier otherwise ec2 instances will use their local regions ... View more

Here is something that will work -To delete bucket both from remote and locally use curl -k -u admn:xxxxxxx https://:24711/services/data/indexes/_audit/freeze-buckets -d bucket_ids=_audit~100~33B81190-7EE1-4FCD-AC6D-DC4E3BEF7E1C -X POST Note: -Done on indexer -works on standalone also -In a cluster environment, the other indexer also would delete the bucket locally -Suitable for S2 environment taking care of deleting remote bucket ... View more

With Splunk Version 7.3.1 and above will let you configure indexes from an indexer to different smartstore objects , for example below I have configure _internal index to use one smartstore and _audit index to use other one. ===========First Smartstore Configuration======= [volume:my_s3_vol] storageType = remote path = s3://newrbal1 remote.s3.access_key = AXXKIAIQWJDOATYCYFTTTTTKWZ5A remote.s3.secret_key = dCCCCCCCCCCN7rMvSN96RSDDDDYqcKeSSSSi3TcD6YQS8J+EzQI5Qm+Ar9 remote.s3.endpoint = https://s3-us-east-2.amazonaws.com remote.s3.signature_version = v4 ===========Second Smartstore Configuration======= using AWS S3 storage [volume:aws_s3_vol] storageType = remote path = s3://luantest remote.s3.access_key = AKIASVRRRRDSSVCAAAANBVKZXK4T remote.s3.secret_key = JYD7umcpFFFFHKM4/uq7Wi/rfyUUHdcSFFFz3j2N85bg8wK remote.s3.endpoint = https://s3-us-east-2.amazonaws.com remote.s3.signature_version = v4 =============Here index _internal is configured with smartstore [volume:my_s3_vol]===== [_internal] thawedPath = $SPLUNK_DB/_internal/thaweddb remotePath = volume:aws_s3_vol/$_index_name repFactor = auto =============Here index _internal is configured with smartstore [volume:aws_s3_vol]===== [_audit] thawedPath = $SPLUNK_DB/_audit/thaweddb remotePath = volume:my_s3_vol/$_index_name repFactor = auto ... View more

1)ERROR message 06-17-2019 22:48:08.445 -0700 ERROR CacheManagerHandler - ReverseIndex cannot add cacheId="bid|ceg_notification_dispatcher_nsprod~560~74F09ED5-8892-471C-9E6C-CD3BC6EDAC9D|" to search sid="remote_searchhead01-3.prd.cmn.mntspl.us-west-2_subsearch_tmp_1560836883.1" as this search has already completed and closed all buckets S2 has some protection against lagging REST requests. There are times when a search process requests to open a bucket but then the search is stopped and cleaned up before the open request is handled. This used to cause us problems leaving buckets stuck with an invalid search reference which would prevent eviction (see "stale buckets" in jira). We have since noticed that some apps and sometimes search/subsearch will re-use the same SID causing these errors. Since we've recently changed the behavior for stale buckets we're in the process of relaxing this bit so we will let these requests through and avoid the error (for quake and then to be backported) 2)Error message 06-17-2019 22:13:08.380 -0700 WARN CacheManager - Attempting to decrement refcount of cacheId="bid|creditscore_prod_pod8~66~8D530702-839D-4679-BBAA-0E408F8EDAFD|" but sid=remote_searchhead01-1.prd.cmn.mntspl.us-west-2_tpargain_tpargainsearch_search34_1560834740.1903850_B8E7D0B9-BB6E-41DB-928B-411ABBA5AD90 is not an opener The above error means a search tried to close a bucket that it didn't have opened. Any particular context you're looking for? ... View more

With Splunk smartstore Generally, ignore HTTP status code on their own There could be intermittent problems in these codes which get resolved by a retry. The codes become valuable when investigating other issues. They could raise red flags if some HTTP status codes start appearing, such as 401's, for example. It wouldn't be practical to enumerate which codes and with which frequency indicates a problem. ... View more

Splunk has Cacheman's endpoint for a bucket requesting it to add the bucket to cachemanager_upload.json Note: Repeat the above for every bucket Restart the indexer Repeat the above for every indexer Note : when you upload the bucket and if the bucket was uploaded successfully the splunkd.log: 08-07-2018 21:51:40.124 +0000 INFO CacheManager - cacheId="bid|_introspection~6~8A26F1F7-F89B-48FB-9398-1E9AC7A2235B|" is NOT on on stable storage, will transfer ... 08-07-2018 21:51:40.125 +0000 INFO CacheManager - action=upload, cacheId="bid|_introspection~6~8A26F1F7-F89B-48FB-9398-1E9AC7A2235B|", status=attempting 08-07-2018 21:51:40.473 +0000 INFO CacheManager - action=upload, cacheId="bid|_introspection~6~8A26F1F7-F89B-48FB-9398-1E9AC7A2235B|", status=succeeded, elapsed_ms=349 audit.log: 08-07-2018 21:51:40.125 +0000 INFO AuditLogger - Audit:[timestamp=08-07-2018 21:51:40.125, user=n/a, action=local_bucket_upload, info=started, cache_id="bid|_introspection~6~8A26F1F7-F89B-48FB-9398-1E9AC7A2235B|", prefix=sample/_introspection/db/49/47/6~8A26F1F7-F89B-48FB-9398-1E9AC7A2235B/guidSplunk-8A26F1F7-F89B-48FB-9398-1E9AC7A2235B][n/a] 08-07-2018 21:51:40.473 +0000 INFO AuditLogger - Audit:[timestamp=08-07-2018 21:51:40.473, user=n/a, action=local_bucket_upload, info=completed, cache_id="bid|_introspection~6~8A26F1F7-F89B-48FB-9398-1E9AC7A2235B|", local_dir="/opt/splunk/var/lib/splunk/_introspection/db/db_1532988790_1532988799_6_8A26F1F7-F89B-48FB-9398-1E9AC7A2235B", kb=260, elapsed_ms=349 *Here is an example to uploade multiple bucket * curl --netrc-file nnfo -k -X POST https://localhost:8089/services/admin/cacheman/_bulk_register -d cache_id="bid|testindex~17~024011E7-E61E-45CE-82DE-732038D5C276|" -d cache_id="bid|testindex~22~024011E7-E61E-45CE-82DE-732038D5C276|" -d cache_id="bid|testindex~28~024011E7-E61E-45CE-82DE-732038D5C276|" -d cache_id="bid|testindex~34~024011E7-E61E-45CE-82DE-732038D5C276|" -d cache_id="bid|testindex~12~E646664A-D351-41E4-BBE7-5B02A08C44C9|" -d cache_id="bid|testindex~17~F876C294-3E3E-488A-8344-16727AC34C52|" -d cache_id="bid|testindex~17~E646664A-D351-41E4-BBE7-5B02A08C44C9|" ... View more

you can only use "| inputintelligence" on non-threat intelligence...given it's a local lookup you can just use "| inputlookup" ? ... View more

Splunk has a bug to guarantee searchability during rolling restart. The BUG#SPL-168132:Clustered indexes aren't fully searchable during an indexer cluster rolling upgrade This bug is fixed in 7.2.8 and 7.3.x ... View more

Although this question is answered in documentation , but this question comes up quite a bit. It's not easy to roll back changes, for this reason, Splunk has not fully implemented this workflow and never tested it with any significance. Hence officially and otherwise not recommending moving out of S2 once enabled. ... View more

In this case, you will need a cluster to re-discover the bucket that is only present on the remote. The cluster can be bootstrap and it will discover the bucket from remote and download them locally, which will enable them to use |dbinspet and later remove the bucket from both locally and from remote. bootstrap command $SPLUNK_HOME/bin/splunk _internal call /services/cluster/master/control/control/init_recreate_index -method POST bootstrapping would ensure that buckets which are already present in the cluster would not be created again on the cluster. bootstrapping would just list all the buckets on S3 and would then create the buckets which are not present on the cluster. It is usually quick as well. Hence if only missing a few buckets on the cluster, we can initiate bootstrapping and it would create these buckets. Is also fairly safe / quick to run this for large deployments. To discover these buckets, bootstrapping is the only option currently. it is not supported per index. The entire operation is detached from the usual operations of CM - it is safe and quick as well. ... View more