In my environment, we send everything to our indexer cluster and use data cloning using _TCP_ROUTING on the universal forwarders to send some data to both the indexer cluster and to an intermediate forwarder. The intermediate forwarder sends the cloned data to another company's splunk environment for analysis. Everything is working as intended, but I can't get the indexers to monitor local /var/log/ logs, index them and also forward them to the intermediate forwarder for forwarding. I looked into the indexing and forwarding option, but that forwards indexed data to the intermediate, when all I want is for the indexer to send it's monitored logs to the cluster and the intermediate. The reason behind this setup is because of our security requirements with our network and ingesting data from clients. Is there a way to do the following: 1) UF sends data to both the indexer cluster and the intermediate 2) Monitor /var/log and other log files on the indexers, index locally while also sending those to the intermediate 3) Do the above without forwarding data being received/indexed by the indexers from forwarders to the intermediate ... View more

I've taken over managing two Splunk environments a while back, one in a Test environment and another in a Prod environment. Was looking through the DMC in Indexer Performance and noticed that the indexer queue information is empty, but all other panels are fine. Indexing rate, pipelines, cpu usage and everything are all showing up, but the queues and the fill ratios are empty. I checked through the indexes and all of the internal splunkd, introspection, metrics, etc are all being indexed for the indexers and other splunk servers, but the queue information is showing as 0% no matter how far I go back. All other information for indexers, search heads, forwarders, and etc seem to show up, but just these two panels in both environments are not monitoring properly. Any help is appreciated. Still trying to look through the configuration. alt text ... View more

Recently setup SmartStore with a test index and sending data to S3. It's working perfectly, but I have questions about the warm to frozen and archiving. In the following splunk doc, it says hot buckets roll to warm buckets and get uploaded on S3 which is great, but doesn't say they can or can't be held there indefinitely. It then says, "Buckets roll to frozen directly from warm.", but doesn't say anything else about it. If buckets can go to S3 and never get rolled to frozen, that's okay, but if it rolls to frozen without me giving it a reason to roll and be deleted, that's something I need to avoid. https://docs.splunk.com/Documentation/Splunk/7.3.2/Indexer/SmartStoreindexing#Bucket_states_and_SmartStore After buckets roll to warm and go to the S3 bucket, if no settings for freezing are configured, will Splunk automatically roll the buckets to frozen after a while? If the warm buckets in the s3 bucket do not get rolled over to frozen and no archiving is set up, will the data in S3 always remain as warm buckets and will that have any issues, besides long searches? Splunk docs say the coldToFrozenScript can be used and I've tried setting it up so that it when warm buckets in s3 get rolled to frozen, it would use that script to take colddb buckets and archive them to another S3 bucket, but because they do not roll to cold on the local server, nothing gets archived. Trying to get the script to work with Splunk, S3, and cache manager doesn't seem to work. Is there a process or script for archiving smartstore warm buckets to another S3 bucket without having to archive locally on the indexer? Update 10/17/2019 I've made some changes to the script and tried testing some things out. Switched from python3 to splunks internal python2.7 and adjusted my code to ensure that splunk was running the script with it's own binaries. Splunk states the coldToFrozenScript and be used for SmartStore indexes, but as to what capacity, I'm not sure. If I run the script manually on a bucket that exists locally on the server (not in S3), the script runs just fine and the bucket gets copied to my S3 endpoint for archiving. I do this by running "/opt/splunk/bin/python2.7 /opt/splunk/etc/slave-apps/ColdToFrozenS3/bin/coldToFrozenS3.py ". It does exactly what I want with local buckets, but I'm trying to get this to run for when SmartStore moves warm s3 buckets to frozen. So, I deployed the script back to the indexers and when Splunk tries to freeze a smartstore bucket and uses the coldToFrozenScript, I get the errors below. It looks like splunk is trying to retrieve the bucket through cache manager and failing for some odd reason or it is trying to initiate the script before the bucket is pulled from S3, or something else I'm not aware of. One of the entries shows a 404 error which doesn't make sense as the servers are able to read and write from both S3 buckets and just for testing purposes, I've given their roles full access. Manually downloading and uploading from each indexer to each S3 bucket works fine, so not sure why the 404 is occurring. 10-17-2019 10:42:33.859 -0400 ERROR DatabaseDirectoryManager - failed to open bucket/wait for bucket to be local through CacheManager, cid="bid|smartstore_test~14~044B61B4-FEA8-4CC9-BEA9-C694C082BECA|", exception=localize operation failed for cacheId="bid|smartstore_test~14~044B61B4-FEA8-4CC9-BEA9-C694C082BECA|" host = <indexer.hostname.local> source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd 10-17-2019 10:42:33.928 -0400 ERROR RetryableClientTransaction - transactionDone(): transactionId=0x7f912e037000 rTxnId=0x7f90c6ffe0f0 success=N HTTP-statusCode=404 HTTP-statusDescription=Not Found retry=N no_retry_reason="transaction had fatal error" host = <indexer.hostname.local> source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd 10-17-2019 10:42:33.928 -0400 WARN BucketMover - RemoteStorageAsyncFreezer freeze failed for bid=smartstore_test~14~044B61B4-FEA8-4CC9-BEA9-C694C082BECA since coldToFrozenScript="/opt/splunk/bin/python2.7" "/opt/splunk/etc/slave-apps/ColdToFrozenS3/bin/coldToFrozenS3.py" could not be run due to exception=std::exception host = <indexer.hostname.local> source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd ... View more

Just recently setup smartstore in a test environment using a single index and I'm trying to figure out some details on the bucket lifecycle. So far I know that hot buckets are stored locally and when they roll to warm, some that are in use or reserved stay local, and then warm buckets are uploaded to the S3 remote storage (AWS in my case). When a search is performed and the data needed is in a warm bucket in remote storage, it is copied from S3 to the indexer's cache manager and gets evicted when no longer needed. From what I've seen in their documentation, smartstore warm buckets in S3 will be frozen and if they are not being locally archived with the coldToFrozenDir setting, then they will be deleted. As I got further into the documentation, they say contradicting things like the process is only hot to warm, or the warm buckets will roll to frozen to be deleted locally and from remote storage, or warm buckets on remote storage will roll from warm to frozen and archived on the indexer if archiving is setup. I don't want to send data to remote storage and then bring it back onto the indexer for archiving locally. To prevent freezing/deletion, I haven't configured any frozen time or max size settings. In the following page for smartstore indexing, it says Hot and Warm only and then randomly says, "Buckets roll to frozen directly from warm" without mentioning anything else in the page about freezing buckets which made me think that it's automatic for smartstore. https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/SmartStoreindexing With the following example indexes.conf, when the buckets roll from hot to warm and are uploaded to AWS S3 , will the splunk ever automatically roll it to frozen? [default] daily hot to warm index transition; seconds maxHotSpanSecs=86400 [volume:s3] storageType = remote path = s3:// [smartstore_test] repFactor=auto homePath = $SPLUNK_DB/$_index_name/db coldPath = $SPLUNK_DB/$_index_name/colddb thawedPath = $SPLUNK_DB/$_index_name/thaweddb remotePath = volume:s3/$_index_name If warm to frozen is inevitable, can smartstore manage archiving to the S3 bucket? If you have SmartStore configured and have any suggestions or examples of things that you wish you knew when you first set it up that you would like to share, I would greatly appreciate the input. Thank you ahead of time and sorry for the long post. ... View more