Background: I have a client with a large clustered environment, I have recently upgraded it to 9.4.6 and fixed wiredTiger / MongoDB 7.0.14 on their indexers. During the remediation work I fixed historical incorrect settings such as the GUI being enabled on the Indexer tier. I want to undertake further best practice remediation work and disable the KVstore on them in order to save resource and prevent un-necessary services running. Challenge: I understand that it is possible / viable to enable a KVstore collection on the peers (indexers) in certain use cases where it can add value. I do not see a use case for my client and they are not aware of one. I have checked the peers and they are all individual KV store captains. splunk show kvstore-status --verbose Checks: I do not want to disable the KVstores (via the CM and config distribution) until I have verified the contents of them. I can issue the command via CLI to  check the list: ./splunk search '| rest /servicesNS/-/-/data/transforms/lookups splunk_server=local | search type=kvstore | fields title, collection, id' - This outputs a tidy 3 column list in the CLI. - Example is the Linux TA: title collection id auditd_host_inventory auditd_host_inventory https://127.0.0.1:8089/servicesNS/nobody/TA-linux_auditd/data/transforms/lookups/auditd_host_inventory   I can then check the linux TA Kvstore via this command: curl -k -u <splunk-user> https://localhost:8089/services/search/v2/jobs/export -d search=" | inputlookup auditd_host_inventory" This outputs a list of circa 400 results.   Guidance: - There is a list of about 8 other KVstores from my first command. - I am struggling to issue the same command for the other KVstores - I think this is due to the app context / name space and sharing - I am getting myself wrapped up in the correct Rest endpoint command to simply issue the command for my list of each of the remaining 8 kvstores e.g. title snow_sys_user_list_lookup collection snow_sys_user_list_kvstore_lookup   Any guidance on this and generally checking or disabling on peers gratefully received. The aim is to protect the clients environment and give them confidence that I have checked before disabling. ... View more

IHAC running a large C11 On-Prem stack. They are in a bit of a pickle due to unsupported RHEL 7 and halfway through an upgrade from 9.3.x to 9.4.x and are seeking advice on the recent CVE's. My problem / question is what version of 'golang' is installed with their particular version of Splunk, this is in response to SVD-2025-0603 | Splunk Vulnerability Disclosure It is not clear how to verify this.   ... View more

IHAC with an SVA C3 (On-Prem) setup running 9.4.0 on the MN, SHC, Deployer but 9.3.2 on the peers (upgrade in the works due to unsupported linux kernel 3.x). They've been running this way OK for about a month whilst the upgrade is pending. Start of issue The problem that is being seen is that the client wanted to disable the new 'audit_trail' app for platform confidentiality a week ago. They created a local folder for the app on the deployer ($SPLUNK_HOME/etc/shcluster/apps/audit_trail) and disabled it via a .conf file change, no issue worked ok and pushed to the SHC from the deployer. The SHC is all in sync. Symptom The issue now being seen is that they can't delete TA's and apps with pushes from the Deployer. For example they are removing legacy TA's and despite not being on the deployer they remain on the SHC. The cluster is operational and in sync OK and I have temporarily removed the 'audit_trail' workaround which allows the usual command to operate again: ./splunk apply shcluster-bundle -target <https://x.x.x.x:8089> -preserve-lookups true If not you have to include the switch (-push-default-apps true) Next steps: I'm trying to locate the correct component in index _internal to troubleshoot what is happening and why it is not deleting apps and TA's not on the Deployer Example: index="_internal" source="/opt/splunk/var/log/splunkd.log" host IN (SH, SH, SH, Deployer) I can't locate any warnings or relevant errors even when including the relevant TA being intended for removal on the short time period in question Any suggestions welcome         ... View more

I have a client who wants to share the Readme file in their app with end users so that they can reference this in the UI. Seems reasonable and prevents them having to duplicate content into a view. Otherwise the readme file is only available to admins who have CLI access. I have tried using the REST endpoint to locate the file, I have checked that the metadata allows read, it is just the path and actual capability I am unclear on. https://<splunk-instance>/en-GB/manager/<redacted>/apps/README.md Thanks   ... View more

I'm working on an environment with a mature clustered Splunk instance. The client wishes to start dual-forwarding to a new replacement environment which is a separate legal entity (they understand the imperfections of dual-forwarding and possible data loss etc.) They need to rename the destination indexes in the new environment dropping a prefix we can call 'ABC', I believe the easiest way is to approach this via INGEST_EVAL on the new Indexes. There are approx 20x indexes to rename example: ABC_linux ABC_cisco     transforms.conf (located on the NEW Indexers) [index_remap_A] INGEST_EVAL = index="value"     I have read the spec file in transforms.conf for 9.3.1 and a 2020 .conf presentation but I am unable to find great examples. Has anyone taken this approach? as it is only a low volume of remaps it may be best to statically approach this. ... View more

I'm ingesting logs from DNS (Next DNS via API) and struggling to exclude the header. I have seen @woodcock resolve some other examples and I can't quite see where I'm going wrong. The common mistake is not doing this on the UF. Sample data: (comes in via a curl command and writes out to a file)   timestamp,domain,query_type,dnssec,protocol,client_ip,status,reasons,destination_country,root_domain,device_id,device_name,device_model,device_local_ip,matched_name,client_name 2023-09-01T09:09:21.561936+00:00,beam.scs.splunk.com,AAAA,false,DNS-over-HTTPS,213.31.58.70,,,,splunk.com,8D512,"NUC10i5",,,,nextdns-cli 2023-09-01T09:09:09.154592+00:00,time.cloudflare.com,A,true,DNS-over-HTTPS,213.31.58.70,,,,cloudflare.com,14D3C,"NUC10i5",,,,nextdns-cli     UF (On syslog server) v8.1.0   props.conf [nextdns:dns] INDEXED_EXTRACTIONS = CSV HEADER_FIELD_LINE_NUMBER = 1 HEADER_FIELD_DELIMITER =, FIELD_NAMES = timestamp,domain,query_type,dnssec,protocol,client_ip,status,reasons,destination_country,root_domain,device_id,device_name,device_model,device_local_ip,matched_name,client_name TIMESTAMP_FIELDS = timestamp inputs.conf [monitor:///opt/remote-logs/nextdns/nextdns.log] index = nextdns sourcetype = nextdns:dns initCrcLength = 375     Indexer (SVA S1) v9.1.0 Disabled the options, I will apply Great8 once I have this fixed. All the work needs to happen on the UF.   [nextdns:dns] #INDEXED_EXTRACTIONS = CSV #HEADER_FIELD_LINE_NUMBER = 1 #HEADER_FIELD_DELIMITER =, #FIELD_NAMES = timestamp,domain,query_type,dnssec,protocol,client_ip,status,reasons,destination_country,root_domain,device_id,device_name,device_model,device_local_ip,matched_name,client_name #TIMESTAMP_FIELDS = timestamp      Challenge: I'm still getting the header field ingest I have deleted the indexed data, regenerated updated log, reingested and still issues. Obviously I have restarted splunk on each instance after respective changes. ... View more

I've just upgraded to Splunk v9.1.0.1 in a stand-alone (S1 SVA) lab instance from 9.0.x. All fine and operating at a basic level, see attached image. However since the upgrade there is a UI issue.     Unable to load app list. Refresh the page to try again.     I have checked splunkd error log and I'm not seeing anything notable:     index="_internal" source="/opt/splunk/var/log/splunk/splunkd.log" log_level IN (ERROR, WARN) | table event_message | dedup event_message      I have also verified consistency of ownership of $SPLUNK_HOME: sudo find /opt/splunk -printf '%u:%g\n' | sort -t: -u  I'm not seeing anything obvious and I have checked the Known Issues list. Has anybody else seen this before I expend a lot of effort on a new release. I should mention that this is now using a Free 500MB license, I have a 10GB NFR license waiting to renew but I don't see this as an issue and it is not in violation. ... View more

I'm a Splunk PS consultant working with a client on a greenfield build, S1 SVA. They want to connect to an Oracle database and we have installed DBConnect and Java etc. After some JRE / JVM version troubleshooting and environment path challenges, it all works fine and can interrogate the Oracle instance. Problem: After each Splunk reload the DB connect fails This is due to the port allocation and can be fixed by manually changing it from 9998 to 9995 for example in the GUI and saving This requires manual intervention each time which is not sustainable The port saves in /opt/splunk/etc/apps/splunk_app_db_connect/jars/<file>.vmopts Has anybody else experienced this please? It's unclear if this is related to the a path variable for Java or some sort of locking and it not being able to reuse the port post daemon reload. ... View more