IHAC which has been seeing poor performance on their DS, not surprising as it has over 10,000 hosts calling home and the majority of them are from windows. We know from the docs that a DS can handle 2,000 Windows hosts per minute or 10,000 linux hosts per minute. My client has 8,500 windows hosts calling in and 1,850 linux hosts calling in. We can reduce the call home interval to every 10 mins but ultimately it needs to scale out. I note in the new docs the reference material: Implement an agent management cluster.  In the past the scaling options were different and crucially my client uses Git as their golden source of code. I see two options: A: Depicted. Implement a load balancer, git runner actions and simply scale the DS's behind a Load Balancer VIP. This is my preferred option. B: Follow the latest docs, implement a cluster using latest config options, have git replicate to a folder that they can all read in an appropriate way Thoughts?   ... View more

IHAC with a C13 SVA instance, I recently upgraded them from Splunk MLTK 5.x to Splunk AI Toolkit 6.0.2 to modernise and neutralise the recently published CVE exposure. Splunk Enterprise is on version 10.2.6 on RHEL 8 so all recent and properly up-to-date. You also have a dependency of the 'Splunk_SA_Scientific_Python_linux_x86_64'  with this new version. At first there was a problem pushing the large bundle for the SA from the SH Deployer to the SHC, and I received the following error which is easily solvable in the same way that a large ES image install is fixed: Error while deploying apps to target=https://<server>:8089 with members=3: Error while updating app=Splunk_SA_Scientific_Python_linux_x86_64 on target=https://<server>:8089: Non-200/201 status_code=413; {\"messages\":[{\"type\":\"ERROR\",\"text\":\"Content-Length of 2147496687 too large (maximum is 2147483648)\"}]}, Error while updating app=Splunk_SA_Scientific_Python_linux_x86_64 on target=https://<server>:8089: Non-200/201 status_code=413; {\"messages\":[{\"type\":\"ERROR\",\"text\":\"Content-Length of 2147496687 too large (maximum is 2147483648) Fix 1: Updated the Search Head TA and pushed this from the deployer after temporarily removing the SA from the deployment folder. I ended up making the changes on the deployer and the SH as it didn't work the first time and nothing lost web.conf [settings] max_upload_size = 4096   This change got me further forward but it would still not allow the large push and I saw a new error. I also noted the the size on disk on the Deployer was approx. 3.6GB and on the SHC 1.5GB. Error while deploying apps to target=https://<server>:8089 with members=3: Error while updating app=Splunk_SA_Scientific_Python_linux_x86_64 on target=https://<server>:8089: Error in JSON response: Unexpected EOF, Error while updating app=Splunk_SA_Scientific_Python_linux_x86_64 on target=https://<server>:8089: Error in JSON response: Unexpected EOF Fix 2: update server.conf and http stanza: server.conf [httpServer] max_content_length = 5000000000  I did make a silly mistake here and had a zero missing making it 0.5GB upload as opposed to 5GB. After that the push worked and the error message on the dependency in the UI went away (the message related to the requirement for the Splunk_SA_Scientific_Python). Final problem: I am unable to see any models in the UI, my understanding is that the models are just the .csv files in the lookups folder in the AI toolkit app, but I have not personally been the end user. - I updated app.conf on the deployer for the AI toolkit and added: deployer_lookups_push_mode =always_overwrite - This did push the updated csv's and I can see all the dates match. - I have checked the docs and I can only see remediation action required for windows - It could conceivably be related to RBAC, but I've change one item and I'm not seeing a difference as an admin. - I have also just noticed that there is a new version as of 25 August, I will upgrade to that to eliminate possible problems: Splunk AI Toolkit | Splunkbase ... View more

Some feedback on this TA, I believe that the documentation listed on Splunkbase is inadequate and it should list at the minimum the destination installation locations. There is some ambiguity on where it should go, e.g. SH only? I think as an official Splunk support app it could be clearer and have a readme docs page to make it straight-forward. One might read it that this is required on the endpoint (computer) interacting with Splunk else why mention the arch type for example. ... View more

IHAC that has a distributed DS/LM/MC in a DMZ environment (see image). It's a new RHEL build on 10.2.2 and clients have their deploymentclient.conf set to call home to it. When I browse to it in the UI there are errors, I note this Help Article and others similar. The fix has not resolved the issue What I'm tying myself in knots over is if the DS / Agent Manager needs to be able to despatch search to the Cluster Manager in the core environment in order to read the internal _ds* indexes. The same goes for Monitoring console and Licence functionality? I can drop the relevant TA with the clustering stanza (mode SH) onto this DMZ (DS/LM/MC) server but I just want to do it the right way. I think I may be over thinking this.     ... View more

Background: IHAC with a complex C13 SVA deployment. They are moving from a Legacy and poorly performing SHC with ES7 which had multiple premium apps, was used for all searches and is very bloated and has become unreliable and pushes from the deployer often fail to complete.  Redesign: I have deployed a new Ad-Hoc SHC and new ES8.3 SHC with new deployers and crucially implemented git and CI/CD controls, backflushing all existing legacy config to code and forming new TA's where required. The client had at least 4x individual <indexes.conf> files and we have consolidated those to a single code controlled <org_TA_indexes> making management of the infra far easier. I very carefully checked tstatsHomePath parameters and volume definitions etc. The new ES8.3 SHC is isolated from the MN / IDX Peers until cutover. Issue: A subtle change appears to have occurred in ES to the index definitions, I downloaded from ES8.3 the TA_ForIndexers and carefully integrated it to git control alongside all the other changes including legacy Security Essentials, Legacy ES etc. It appears that a week or so ago the notables path changed on disk to include a 'db' definition. From the looks of things this is a development after ES6.x which is needed only on the SHC tier. [notable] coldPath = volume:primary/notabledb/colddb homePath = volume:primary/notabledb/db thawedPath = $SPLUNK_DB/notabledb/thaweddb tstatsHomePath = $SPLUNK_DB/$_index_name/datamodel_summary  Historically the code was just:   [notable] coldPath = volume:primary/notable/colddb homePath = volume:primary/notable/db thawedPath = $SPLUNK_DB/notable/thaweddb tstatsHomePath = $SPLUNK_DB/$_index_name/datamodel_summary     Impact: The effect of this is that approx 500 notable events are in a different path on disk (IDX peers) and I have put this back to what it was historically, all fine but now these 500 events are not searchable. Questions: I think this has been caused by the merge and consolidation and the fact that notabledb is supposed to exist on SH but not on the IDX tier? Best fix is probably to define a path of [notable-tmp] with the db extension so that the small index is searchable and just run a collect of the 500 events to fill the gap. I could issue a replay from disk of the saved searches but there is a lot to fill  I could take the bucket method but this feels complex in a cluster Lots of legacy tech-debt here to fix and no harm done but I'm keen to put this right for my client ... View more

Background: I have a client with a large clustered environment, I have recently upgraded it to 9.4.6 and fixed wiredTiger / MongoDB 7.0.14 on their indexers. During the remediation work I fixed historical incorrect settings such as the GUI being enabled on the Indexer tier. I want to undertake further best practice remediation work and disable the KVstore on them in order to save resource and prevent un-necessary services running. Challenge: I understand that it is possible / viable to enable a KVstore collection on the peers (indexers) in certain use cases where it can add value. I do not see a use case for my client and they are not aware of one. I have checked the peers and they are all individual KV store captains. splunk show kvstore-status --verbose Checks: I do not want to disable the KVstores (via the CM and config distribution) until I have verified the contents of them. I can issue the command via CLI to  check the list: ./splunk search '| rest /servicesNS/-/-/data/transforms/lookups splunk_server=local | search type=kvstore | fields title, collection, id' - This outputs a tidy 3 column list in the CLI. - Example is the Linux TA: title collection id auditd_host_inventory auditd_host_inventory https://127.0.0.1:8089/servicesNS/nobody/TA-linux_auditd/data/transforms/lookups/auditd_host_inventory   I can then check the linux TA Kvstore via this command: curl -k -u <splunk-user> https://localhost:8089/services/search/v2/jobs/export -d search=" | inputlookup auditd_host_inventory" This outputs a list of circa 400 results.   Guidance: - There is a list of about 8 other KVstores from my first command. - I am struggling to issue the same command for the other KVstores - I think this is due to the app context / name space and sharing - I am getting myself wrapped up in the correct Rest endpoint command to simply issue the command for my list of each of the remaining 8 kvstores e.g. title snow_sys_user_list_lookup collection snow_sys_user_list_kvstore_lookup   Any guidance on this and generally checking or disabling on peers gratefully received. The aim is to protect the clients environment and give them confidence that I have checked before disabling. ... View more

IHAC running a large C11 On-Prem stack. They are in a bit of a pickle due to unsupported RHEL 7 and halfway through an upgrade from 9.3.x to 9.4.x and are seeking advice on the recent CVE's. My problem / question is what version of 'golang' is installed with their particular version of Splunk, this is in response to SVD-2025-0603 | Splunk Vulnerability Disclosure It is not clear how to verify this.   ... View more