What you are actually trying to do? Run SHC with static captain, why? Are there issues to use it as normal SHC environment? ... View more

My recommendation is add all HFs, HECs etc as indexers into mc. Then create need subgroups to divide those to reasonable logical sets. Now you can use those groups when you’re selecting target servers in different dashboards. In this way your life is much easier than trying to use on SPL or even separate dashboards. ... View more

You can get it from here https://splunk.my.site.com/customer/s/article/0-byte-license-for-Deployment-Server-or-Heavy-Forwarder ... View more

One additions. If you are using EntraID then I suppose that you must use windows HF which is joined to it. At least older dbx (3.18.x) with Linux can’t authenticate with EntraID accounts. Instead of you must use sql server’s internal users. ... View more

When I’m looking manuals, then 1st time this documented is 10.0 https://help.splunk.com/en/splunk-enterprise/create-dashboards-and-reports/dashboard-studio/10.0/share-dashboard-content/publish-dashboards I think that before 10 those versions between SCP and CMP is not exactly x.y.z - 0.1.x. And especially with DS there was more differences than other core parts. ... View more

Please add those configurations here. ... View more

I haven’t try this for long time at onprem, but have you try use localhost / 127.0.0.1 here? If I recall right this works at least some beta instances? ... View more

As other already said there is more issues than you want to handle with your requirements. Can you open more about your current setup, what you are really try to achieve with this new setup and where these sites are in geographically point of view.  I didn’t understand your configurations and requirements based what you have told to us. We definitely need more information where you are put nodes and how those are located currently. E.g. you have site_search_factor 2 and you want that SHC nodes are searching only on local site. You don’t tell if your current SHC node is only on site8 or not? Based on your requirements this she that way. Then you want add additional SHC nodes into site12.  Why you have indexers on 4 in future 8 site, but use only two sites data for searching?  How many indexers you have per site and how many SHCs you totally have and in which sites? Why you want to put new indexers into old removed site1? Why you have that site definition in general? ... View more

If this installation works correctly in another scp, then it’s time to create support ticket to splunk. Obviously there is something weird in your stack where you are trying to install it. One thing what you could try is 1st reboot your scp and then try it again.. ... View more

On addition. Actually it's Splunk's best practice to forward all logs from Splunk servers to indexers and not keep those in local. Event DS you should forward those as additional you keep those also in local node too. This will helps you to do trouble shooting when some of those servers are down. ... View more

Hi My old post for migrating distributes splunk environment to another site. No need to keep old IPs just ensure that you are using FQDN in all names. You should migrate 1st infra with above order then HFs and last UFs. And order of HF vs UF is just how I usually do it, if you want you can do it any order with those clients. ... View more

As @richgalloway already said this is known issue. It could be also false positive as tresholds are too low.  There are several articles and discussions of it already. I link just some here: https://splunk.my.site.com/customer/s/article/Health-Report-IOWait-Showing-Red https://splunk.my.site.com/customer/s/article/iowait https://community.splunk.com/t5/Splunk-Enterprise/Why-is-the-health-status-of-IOWait-red/m-p/599097 https://help.splunk.com/en/splunk-enterprise/administer/monitor/9.0/proactive-splunk-component-monitoring-with-the-splunkd-health-report/configure-the-splunkd-health-report   ... View more

If I recall right you can do this quite easily with acs command. Just create/get current indexes.conf file as json and update those values into it. Then push it back with acs command to SCP and that's it.  No need to do any coding etc. https://help.splunk.com/en/splunk-cloud-platform/administer/admin-config-service-manual/9.3.2408/administer-splunk-cloud-platform-using-the-admin-config-service-acs-cli/administer-splunk-cloud-platform-using-the-acs-cli But you must test this 1st with some test indexes. https://help.splunk.com/en/splunk-cloud-platform/administer/admin-config-service-manual/9.3.2408/administer-splunk-cloud-platform-using-the-admin-config-service-acs-cli/administer-splunk-cloud-platform-using-the-acs-cli#update-indexes-in-bulk-0 That was instructions how to manage several indexes at time. And it you want to use Terraform that is also one option for this. There is some words about it in the beginning of this page. ... View more

Please create a support ticket to Splunk so they can get this information too. Just add reference to this support article too. ... View more

One other option if nothing else is working. Is just check email address of this channel. Then send alert via email to this channel instead of real team alerts. Of course this is not show as nicely than real teams alerts, but anyhow you will get those into teams channel. ... View more

I found this https://splunk.my.site.com/customer/s/article/Splunk-Enterprise-9-4-3-Failed-to-read-memory-limit-Error-Message-on-Restart which told that at least older linux this is just false positive in log. They will fix it in future version.   ... View more

Yes those are independent, but as one file is actively read until all event are handled then one big active file don't block the traffic as there are another pipelines available to process other files instead of waiting when that big high volume file has read. ... View more

You need to convert start time to epoch and then sort by it. | eval sTime = strptime(startTime,"%FT%T.%3Q%Z") ... View more

Hi In transaction command reference it Is said Descending chronological order required The transaction command requires that the incoming events be in descending chronological order. Some commands, such as eval, might change the order or time labeling of events. If one of these commands precedes the transaction command, your search returns an error unless you include a sort command in your search. The sortcommand must occur immediately before the transaction command to reorder the search results in descending chronological order. So you must sort those into correct order before you can use successfully transaction command. ... View more

One additional point MC + DS live together. It depends if your admins and persons are same staff or not. As MC is usually targetted only to admins not other persons, you should really think if you want to mix those roles. I know that many cases this is not issue but still something to consider. ... View more

Neither me. I like to say that only reason to cluster DS it that you have so many clients that one DS cannot manage it. Currently official values is probably 25k after that one recommended host cannot server those with reasonable load and response times. If you are just needing HA feature then it's better to automate you DS configurations and also automate build up DS and use only names in any configurations. Then if your primary DS will got broken/unavailable etc., then just build a new one and redeploy configurations to it. Of course this probably enforce all your clients to reload their apps and then restart splunkd. Anyhow I strongly recommend to all, that use DS GUI just to look situation, but never ever use it for configure DS. All configurations should be (or must) in e.g. git and you have way to deploy those as needed into (any) DS. ... View more

If I have understand correctly how splunk UF is doing this and you have only those two high volume logs then (probably) you can try to prioritise those by adding pipelines. I'm not 100% sure this, but my understanding is that in normal situation on files will be read to the end and then UF switch to another. If this is true then those high volume files could have dedicated readers and other shares additional. But remember that you cannot add too many pipelines per node! And as I said I haven't been to test this in real life by myself. There have been cases where I have added some pipelines e.g. in HF (DBX) to get it working correctly. And I expecting that this is working UF too? I don't believe that this is "approved by Splunk" solution 😉 ... View more

As said this hasn't beed recommended way to do before 9.4 as @richgalloway said. But if you have some older versions and those are behind LB then you probably (or at least you should have) this settings in use in your serverclass.conf crossServerChecksum = <boolean> * Determines whether the deployment server ensures consistent app checksums across multiple deployment servers. * A value of "true" means the deployment server generates the same checksum for each app across different deployment servers. This setting is useful if you have multiple deployment servers behind a load balancer. * A value of "false" means each deployment server generates its own checksum independently. * Default: false  If I recall correctly there are conf19? or something presentation how you could configure this in old way. Here is one old post which also could help you if you have that setup https://community.splunk.com/t5/Deployment-Architecture/multiple-deployment-servers-checksum-mismatch-among-instances-of/m-p/167120 But currently if you need several DS then use official way which is defined in those help pages which Rich pointed to you. ... View more

And don't remember to check that someone haven't hardcoded those names + IPs into /etc/hosts 😞 ... View more