Attached screenshot  is a list of 15 query ids with started, ended, bstarted (15 minute bucket) and query duration. Tried with concurrency cmd:       |where started <= "2021-09-20 15:45:00" AND ended >= "2021-09-20 15:45:00"| eval bstarted = strptime(started, "%Y-%m-%d %H:%M:%S.%3N")|bin span=15m bstarted | eval bstarted = bstarted+900| where strptime(started, "%Y-%m-%d %H:%M:%S.%3N") <= bstarted and strptime(ended, "%Y-%m-%d %H:%M:%S.%3N") >= bstarted|eval bstarted=strftime(bstarted,"%F %T")|eval st=strptime(started, "%Y-%m-%d %H:%M:%S.%3N") | concurrency duration=query_duration start=st|table query_id started bstarted ended query_duration concurrency       surprised to see that the concurrency values are a sequence from 1 to 15? Does not quite make sense. 2 Tried with stats dc(query_id)  as concurrency_ct by bstarted  and did not get the desired results as per screenshot         "20210920_193943_02412_rhmkv","2021-09-20 15:39:43.236","2021-09-20 15:45:00","2021-09-20 15:57:57.829",1094, "20210920_192921_02318_rhmkv","2021-09-20 15:29:21.670","2021-09-20 15:30:00","2021-09-20 16:03:59.951",2078, "20210920_193005_02322_rhmkv","2021-09-20 15:30:06.022","2021-09-20 15:45:00","2021-09-20 15:46:33.583",987, "20210920_192946_02321_rhmkv","2021-09-20 15:29:46.959","2021-09-20 15:30:00","2021-09-20 16:13:08.574",2601,       There are 4 queries_duration > 900 seconds, desired concurrency_cnt are:       "2021-09-20 15:30:00",2 "2021-09-20 15:45:00",15 (2 more from the 2078 and 2601 seconds duration values) "2021-09-20 16:00:00",4 (4 more from the 2078 and 2601, 1094, 987 seconds duration values)       Was hoping concurrency command would give me the results. Not sure where is the mistake? Would appreciate some inputs, guidance. Thanks.      

Hi All, We are embarking on moving our Splunk 8.1.3 servers from old version of RHEL to new RHEL servers. The servers names are going to be different and we currently have the config based on IP/hostname as opposed to alias. We have perused the answers community and as well, referred Splunk documentation and have identified the below approach for indexer cluster migration. There's no single Splunk document that talks about this fairly common occurrence in many environments. We also have raised a Splunk support case for advice but to no avail.  A few details about our existing cluster so you can factor that in while reviewing the below Multisite Indexer cluster with 2 indexers across 2 sites Current RF - 2 SF-2 (Origin 1 and total 2 for both) Daily ingestion of roughly 400-450 GB Current storage of 4 indexers (Roughly 9TB utilised out of 11 TB total hot_warm on each indexer). Roughly 36 TB across the indexer cluster. ~ 20K buckets with approx. 5K buckets in each indexer Retention varies between 3 months, 6 months and some on 1 year It is imperative that we have all data replicated to new servers before we retire the old ones No forwarder-->Indexer discovery Un-clustered search heads(4) including one Splunk ES 6.4.x Key concerns: Storage on existing indexers being overwhelmed. Degraded search performance on cluster Resource util increase on cluster instances Sustained replication Lack of testing ability to test the below at scale. Can you please verify/validate/provide commentary/gotchas on this approach please? Would love suggestions on improving this as well Proposed steps Step 0 - Server Build - Build new CMs and Indexers to existing specs (2RF, 2 SF, copy remote bundles on CM etc.) Step 1 - Cluster master migration Search heads - Add additional cluster master via DS push Place existing CM in maintenance mode/Stop CM Ensure new CM is started and running. Place in maintenance mode Modify server.conf on 1 indexer at a time to point to new CM Disable maintenance mode and allow new cluster master to replicate/bucket fixup Validate and confirm that no errors and no problems are seen. Step 2 - Indexer cluster migration 1 on each site Push changes to forwarders to include 2 new indexers (1 in each site)  Place new CM in maintenance mode. Add 1 indexer in each site with existing SF and RF Remove maintenance mode Issue data rebalance command (Searchable rebalance, end goal of which is to have roughly 3K-3.2K buckets in each of the 6 indexers) Wait for catch up……………………………………………….. Place new CM in maintenance mode. Modify CM RF to 4 from 2. Keep SF as same In the same maintenance mode, modify origin as 3 and total 4 for replication factor. SF remains same (On all 6 indexers.4 old + 2 new). This is to ensure each indexer in a site has 1 copy of each bucket Restart indexer cluster master (and possibly indexers) for new changes to reflect Disable maintenance mode Wait for catch up……………………………………………….. Step 3 - Indexer cluster migration 1 on each site Once caught up and validated, place cluster master in maintenance mode Place one old indexer in offline enforce-counts and add 1 new indexer on each site Disable maintenance mode Immediately issue data rebalance command Wait for catch up……………………………………………….. Step 4 – Cluster migration completion Once validated, place cluster master in maintenance mode Place the 2nd set of old indexers on each site in offline enforce-counts Modify RF to 2 and SF remains the same Modify Origin to 1 and Total to 2 RF on all indexers Restart indexers and CM as appropriate Disable maintenance mode Allow excess bucket fix up activities to complete Once cluster has regained status quo, shutdown old indexers Remove reference to old indexers from forwarders and search heads to avoid warnings/errors  

Is it possible to disable some stanzas from, for example, the Windows TA, using another TA? I apologize ahead of time if this comes off newb-ish and needlessly complex, but we've run into an interesting problem. We have a number of Citrix XenApp servers in our environment and a little while back, they began struggling with some of the processes that Splunk spawns through scripted inputs. They were taking up a lot of CPU and RAM. The short-term fix seemed to be disabling these inputs, like [admon] and [WinRegMon] and setting them to interval =-1. I found these stanzas in both the \etc\system\ inputs.conf and the Windows TA inputs.conf. To keep them disabled, and only put this in place on Citrix servers, I created a Citrix server class. Then I took the Windows TA, copied it, renamed it to something else, and kept everything the same except that I disabled those stanzas for this "Citrix" Windows TA (they are enabled on the standard Windows TA). This accomplished the goal of keeping them disabled for Citrix, but I'm not a fan of this solution because there are probably references within the Windows TA that need it to have the standard folder path ("Splunk_TA_windows"). I've already located and corrected a couple. So I'm interested in moving the Citrix servers back to the standard Splunk_TA_windows TA, but want to keep those stanzas disabled for just those servers. My question: Can I create a custom TA, apply it just to that Citrix server class, and just have an inputs.conf file with those stanzas disabled? I'm just not sure which TA would "win", if they had conflicting instructions for the stanzas: Splunk_TA_windows saying that [WinRegMon] is not disabled, for example, but the custom TA saying that it should be disabled. So the custom TA inputs.conf file would have stanzas like this: [WinRegMon] disabled=1 interval=-1 baseline=0 [admon] interval=-1 disabled=1 baseline=0

Hello, we configured custom business transactions on one of our customer's environments, but we couldn't see our custom business transactions on the BT menu, all our custom BTs are being displayed under "All Other Traffic"  group. Could you please help us with this? Note:  Same configuration/BTs are working in other environments/applications.