Background: IHAC with a complex C13 SVA deployment. They are moving from a Legacy and poorly performing SHC with ES7 which had multiple premium apps, was used for all searches and is very bloated and has become unreliable and pushes from the deployer often fail to complete.  Redesign: I have deployed a new Ad-Hoc SHC and new ES8.3 SHC with new deployers and crucially implemented git and CI/CD controls, backflushing all existing legacy config to code and forming new TA's where required. The client had at least 4x individual <indexes.conf> files and we have consolidated those to a single code controlled <org_TA_indexes> making management of the infra far easier. I very carefully checked tstatsHomePath parameters and volume definitions etc. The new ES8.3 SHC is isolated from the MN / IDX Peers until cutover. Issue: A subtle change appears to have occurred in ES to the index definitions, I downloaded from ES8.3 the TA_ForIndexers and carefully integrated it to git control alongside all the other changes including legacy Security Essentials, Legacy ES etc. It appears that a week or so ago the notables path changed on disk to include a 'db' definition. From the looks of things this is a development after ES6.x which is needed only on the SHC tier. [notable] coldPath = volume:primary/notabledb/colddb homePath = volume:primary/notabledb/db thawedPath = $SPLUNK_DB/notabledb/thaweddb tstatsHomePath = $SPLUNK_DB/$_index_name/datamodel_summary  Historically the code was just:   [notable] coldPath = volume:primary/notable/colddb homePath = volume:primary/notable/db thawedPath = $SPLUNK_DB/notable/thaweddb tstatsHomePath = $SPLUNK_DB/$_index_name/datamodel_summary     Impact: The effect of this is that approx 500 notable events are in a different path on disk (IDX peers) and I have put this back to what it was historically, all fine but now these 500 events are not searchable. Questions: I think this has been caused by the merge and consolidation and the fact that notabledb is supposed to exist on SH but not on the IDX tier? Best fix is probably to define a path of [notable-tmp] with the db extension so that the small index is searchable and just run a collect of the 500 events to fill the gap. I could issue a replay from disk of the saved searches but there is a lot to fill  I could take the bucket method but this feels complex in a cluster Lots of legacy tech-debt here to fix and no harm done but I'm keen to put this right for my client ... View more

Background: I have a client with a large clustered environment, I have recently upgraded it to 9.4.6 and fixed wiredTiger / MongoDB 7.0.14 on their indexers. During the remediation work I fixed historical incorrect settings such as the GUI being enabled on the Indexer tier. I want to undertake further best practice remediation work and disable the KVstore on them in order to save resource and prevent un-necessary services running. Challenge: I understand that it is possible / viable to enable a KVstore collection on the peers (indexers) in certain use cases where it can add value. I do not see a use case for my client and they are not aware of one. I have checked the peers and they are all individual KV store captains. splunk show kvstore-status --verbose Checks: I do not want to disable the KVstores (via the CM and config distribution) until I have verified the contents of them. I can issue the command via CLI to  check the list: ./splunk search '| rest /servicesNS/-/-/data/transforms/lookups splunk_server=local | search type=kvstore | fields title, collection, id' - This outputs a tidy 3 column list in the CLI. - Example is the Linux TA: title collection id auditd_host_inventory auditd_host_inventory https://127.0.0.1:8089/servicesNS/nobody/TA-linux_auditd/data/transforms/lookups/auditd_host_inventory   I can then check the linux TA Kvstore via this command: curl -k -u <splunk-user> https://localhost:8089/services/search/v2/jobs/export -d search=" | inputlookup auditd_host_inventory" This outputs a list of circa 400 results.   Guidance: - There is a list of about 8 other KVstores from my first command. - I am struggling to issue the same command for the other KVstores - I think this is due to the app context / name space and sharing - I am getting myself wrapped up in the correct Rest endpoint command to simply issue the command for my list of each of the remaining 8 kvstores e.g. title snow_sys_user_list_lookup collection snow_sys_user_list_kvstore_lookup   Any guidance on this and generally checking or disabling on peers gratefully received. The aim is to protect the clients environment and give them confidence that I have checked before disabling. ... View more

IHAC running a large C11 On-Prem stack. They are in a bit of a pickle due to unsupported RHEL 7 and halfway through an upgrade from 9.3.x to 9.4.x and are seeking advice on the recent CVE's. My problem / question is what version of 'golang' is installed with their particular version of Splunk, this is in response to SVD-2025-0603 | Splunk Vulnerability Disclosure It is not clear how to verify this.   ... View more

IHAC with an SVA C3 (On-Prem) setup running 9.4.0 on the MN, SHC, Deployer but 9.3.2 on the peers (upgrade in the works due to unsupported linux kernel 3.x). They've been running this way OK for about a month whilst the upgrade is pending. Start of issue The problem that is being seen is that the client wanted to disable the new 'audit_trail' app for platform confidentiality a week ago. They created a local folder for the app on the deployer ($SPLUNK_HOME/etc/shcluster/apps/audit_trail) and disabled it via a .conf file change, no issue worked ok and pushed to the SHC from the deployer. The SHC is all in sync. Symptom The issue now being seen is that they can't delete TA's and apps with pushes from the Deployer. For example they are removing legacy TA's and despite not being on the deployer they remain on the SHC. The cluster is operational and in sync OK and I have temporarily removed the 'audit_trail' workaround which allows the usual command to operate again: ./splunk apply shcluster-bundle -target <https://x.x.x.x:8089> -preserve-lookups true If not you have to include the switch (-push-default-apps true) Next steps: I'm trying to locate the correct component in index _internal to troubleshoot what is happening and why it is not deleting apps and TA's not on the Deployer Example: index="_internal" source="/opt/splunk/var/log/splunkd.log" host IN (SH, SH, SH, Deployer) I can't locate any warnings or relevant errors even when including the relevant TA being intended for removal on the short time period in question Any suggestions welcome         ... View more

I have a client who wants to share the Readme file in their app with end users so that they can reference this in the UI. Seems reasonable and prevents them having to duplicate content into a view. Otherwise the readme file is only available to admins who have CLI access. I have tried using the REST endpoint to locate the file, I have checked that the metadata allows read, it is just the path and actual capability I am unclear on. https://<splunk-instance>/en-GB/manager/<redacted>/apps/README.md Thanks   ... View more

I'm working on an environment with a mature clustered Splunk instance. The client wishes to start dual-forwarding to a new replacement environment which is a separate legal entity (they understand the imperfections of dual-forwarding and possible data loss etc.) They need to rename the destination indexes in the new environment dropping a prefix we can call 'ABC', I believe the easiest way is to approach this via INGEST_EVAL on the new Indexes. There are approx 20x indexes to rename example: ABC_linux ABC_cisco     transforms.conf (located on the NEW Indexers) [index_remap_A] INGEST_EVAL = index="value"     I have read the spec file in transforms.conf for 9.3.1 and a 2020 .conf presentation but I am unable to find great examples. Has anyone taken this approach? as it is only a low volume of remaps it may be best to statically approach this. ... View more

I'm ingesting logs from DNS (Next DNS via API) and struggling to exclude the header. I have seen @woodcock resolve some other examples and I can't quite see where I'm going wrong. The common mistake is not doing this on the UF. Sample data: (comes in via a curl command and writes out to a file)   timestamp,domain,query_type,dnssec,protocol,client_ip,status,reasons,destination_country,root_domain,device_id,device_name,device_model,device_local_ip,matched_name,client_name 2023-09-01T09:09:21.561936+00:00,beam.scs.splunk.com,AAAA,false,DNS-over-HTTPS,213.31.58.70,,,,splunk.com,8D512,"NUC10i5",,,,nextdns-cli 2023-09-01T09:09:09.154592+00:00,time.cloudflare.com,A,true,DNS-over-HTTPS,213.31.58.70,,,,cloudflare.com,14D3C,"NUC10i5",,,,nextdns-cli     UF (On syslog server) v8.1.0   props.conf [nextdns:dns] INDEXED_EXTRACTIONS = CSV HEADER_FIELD_LINE_NUMBER = 1 HEADER_FIELD_DELIMITER =, FIELD_NAMES = timestamp,domain,query_type,dnssec,protocol,client_ip,status,reasons,destination_country,root_domain,device_id,device_name,device_model,device_local_ip,matched_name,client_name TIMESTAMP_FIELDS = timestamp inputs.conf [monitor:///opt/remote-logs/nextdns/nextdns.log] index = nextdns sourcetype = nextdns:dns initCrcLength = 375     Indexer (SVA S1) v9.1.0 Disabled the options, I will apply Great8 once I have this fixed. All the work needs to happen on the UF.   [nextdns:dns] #INDEXED_EXTRACTIONS = CSV #HEADER_FIELD_LINE_NUMBER = 1 #HEADER_FIELD_DELIMITER =, #FIELD_NAMES = timestamp,domain,query_type,dnssec,protocol,client_ip,status,reasons,destination_country,root_domain,device_id,device_name,device_model,device_local_ip,matched_name,client_name #TIMESTAMP_FIELDS = timestamp      Challenge: I'm still getting the header field ingest I have deleted the indexed data, regenerated updated log, reingested and still issues. Obviously I have restarted splunk on each instance after respective changes. ... View more