@raymondteledata - Just curious, from where did you get this email address? This email address is not a valid address as per email domain checker and I don't see it in Splunk documentation either. Since you are using Splunk Enterprise (on‑prem), you have to configure your own SMTP relay or mail server. Ref: Email Checker: Verify email address for free - Mailmeteor   ... View more

@sswigart btw interesting scenario. This is not a comprehensive solution and providing some guidance on how to proceed in this case. As we do in any migration, firstly, ensure we have a fallback mechanism. In this case, keep the Windows Splunk server intact until the Linux migration is fully validated. This gives you a fallback if anything goes wrong. Secondly, since the environment is air gapped, you can use offline media (external drives, SSDs) for data transfer. Copy the entire Splunk db directory (all hot, warm, and cold buckets) from Windows to Linux. Make sure to stop Splunk service before copying to avoid partial buckets. Thirdly, you need to adjust Configurations - both system/local and app configurations. Also path changes like from C:\... to /opt/splunk/… for linux. On Linux, ensure directories are owned by the splunk user. Use the below command. chown -R splunk:splunk /opt/splunk/ Ensure version alignment, run the same Splunk Enterprise version on both Windows and Linux during migration. Validate disk space and filesystem compatibility (NTFS to ext4/xfs). Audit index names and configs for consistent casing to avoid mismatches since Linux is case‑sensitive. >> If this post addressed your question, you can: Give it karma to show appreciation 👍 Mark it as the solution if it solved your issue ✔️ Add a comment if you’d like more details ✏️ Acknowledging helpful answers keeps the community strong and motivates contributors to continue sharing their expertise. >> ... View more

@las Understood, you may follow the below troubleshooting: Check the internal, setup handler and Modular input logs for related errors. Use the below file paths. SPLUNK_HOME/var/log/splunk/restmodinput_app_modularinput.log SPLUNK_HOME/var/log/splunk/restmodinput_app_setuphandler.log Ensure python version compatibility with TA and Splunk software version Ensure the TA dependencies are not removed in latest Splunk version. Since it's a third-party app and not supported by Splunk, such issues may arise. Use the below TA documentation for assistance baboonbones.com/php/markdown.php?document=rest/README.md If self-troubleshooting fails, log a support case after the initial checks and collect the above details to see if Splunk support is able to identify the issue. Hope this helps. >> If this post addressed your question, you can: Give it karma to show appreciation 👍 Mark it as the solution if it solved your issue ✔️ Add a comment if you’d like more details ✏️ Acknowledging helpful answers keeps the community strong and motivates contributors to continue sharing their expertise. >> ... View more

@las To assist you better, could you please clarify the below:  Did you update the TA or add-on you utilized along with Enterprise version upgrade and checked the compatibility with add-on and updated Splunk version? Verify whether the Heavy Forwarder host configured with any system‑level proxy (environment variables, OS settings, or Splunk server.conf) Are you utilizing the below add-on? REST API Modular Input | Splunkbase To narrow down the issue, you may test connectivity outside of Splunk. Use curl or wget from Splunk HF to the target endpoint. If those succeed, the issue is likely within Splunk’s runtime or add‑on.  >> If this post addressed your question, you can: Give it karma to show appreciation 👍 Mark it as the solution if it solved your issue ✔️ Add a comment if you’d like more details ✏️ Acknowledging helpful answers keeps the community strong and motivates contributors to continue sharing their expertise. >>   ... View more

@_user25 I guess the direct CLI edits won’t trigger encryption or populate the respective password conf files. The supported secure rotation methods are via Splunk Web or GCE SA. For automation, you might need to build a wrapper around Splunk’s REST API to achieve this. You can use the storage/passwords REST API endpoint for this use case. Make sure your input stanza references the same credential name. Ref:  Manage secret storage | Documentation | Splunk Developer Program   ... View more

@briancronrath - [Updated, thanks @livehybrid ] You're right to suspect the GUID mismatch. Splunk uses the GUID in instance.cfg file to uniquely identify each node, and the Cluster Manager will continue to associate the old GUID with the search head that was rebuilt. That’s why it shows as “Down” even though the node itself is healthy. Try the below on a dev environment first and then apply in production. These are suggestions based on experience and working knowledge. Please be cognizant of the actions. If you still have a backup of the old instance.cfg, you can copy the original GUID back into the rebuilt node’s instance.cfg. Restart Splunk on that search head. The Cluster Manager will then recognize it as the same node it had before. This is the cleaner option if you want continuity with the old identity. If you don't have the old instance.cfg file, you need to register the rebuilt search head with Cluster manager and remove the old entry from Cluster Manager's configuration. This usually involves editing the cluster configuration (server.conf) under ([clustering]) and re-running the initializing command on the rebuilt node, pointing it to the Cluster Manager. > Marking the answer and giving Karma helps others find solutions faster!   Ref: Configure the search head cluster | Splunk Enterprise (last updated 2025-07-04T12:40:38.592Z)     ... View more

@rohitkattewar - Could you please attach the error screenshot?  When Splunk Web redirects you to localhost instead of the login page, it usually points to a misconfiguration in how Splunk Web is bound to network interfaces or how you’re accessing it. If web.conf is misconfigured, Splunk may default to localhost. You may check the web.conf in your splunk instance. > Marking the answer and giving Karma helps others find solutions faster!   ... View more

@AShwin1119  I checked Splunk’s official system requirements for Splunk Enterprise and the Universal Forwarder. As of version 10.2, Splunk supports Debian 11 and Debian 12 under the “Unix operating systems” section. Debian 13 is not listed as a supported operating system at this time. Few things to consider: Splunk Universal Forwarder is not yet certified for Debian 13. Running it there would be considered unsupported. In many cases, Splunk binaries may still run on newer Debian releases, but without official support you risk compatibility issues, lack of vendor assistance, and potential instability. If you need guaranteed support, stick to Debian 11 or 12. If you want to experiment on Debian 13, test thoroughly in a non-production environment. > Marking the answer and giving Karma helps others find solutions faster!  ... View more

@Cybers1 Few things here, Make sure the indexes are created and configured correctly in indexes.conf which is referenced in your config files Props stanza matching, make sure the sourcetype matches the format defined. Regarding placement of configurations. If both HF and Indexer have configs, it's okay to have it, but sometimes you can get conflicts due to mismatches. It's best to keep them only in HF as it could do the parsing and if you are keeping it in both, make sure those are identical. For regex pattern, can you try the below props and transforms conf file in dev testing: transforms.conf [redirect_source_k8_accesslog_coll] REGEX = \[ACCESSLOG_COLL\] FORMAT = accesslog_coll SOURCE_KEY = _raw DEST_KEY = _MetaData:Index WRITE_META = true [redirect_source_k8_accesslog] REGEX = \[ACCESSLOG\] FORMAT = accesslog SOURCE_KEY = _raw DEST_KEY = _MetaData:Index WRITE_META = true props.conf [kube:container:*] TRANSFORMS-routing = redirect_source_k8_accesslog_coll, redirect_source_k8_accesslog       ... View more

@slibitcs, @rtoelhoej  Thanks for posting. It appears like an issue which Splunk haven't acknowledged or encountered during their testing as it's not listed in the known issues. The daemon is part of the Linux IPC layer and best course of action for you is to immediately raise a Support case with Splunk to report the issue with detailed log information, version information of both your OS and Splunk software.  I have reviewed the known issues for the version and there is no mention about the dbus-daemon spawning repeatedly and consuming CPU resources indicating an issue which they haven't encountered while testing. Reference: Splunk Enterprise 10.2.0 known issues | Splunk Enterprise (last updated 2026-01-17T00:24:43.945Z) > Marking the answer and giving Karma helps others find solutions faster! ... View more

@hordoffa1970 The error message “Failed to create. Configuration for port 9997 already exists” means the receiving port you’re trying to configure is already set up somewhere in your Splunk configuration. Port 9997 is the default receiving port for Splunk indexers, so if it’s already enabled, trying to add it again will trigger this message. You can check existing receiving configuration In Splunk Web by navigating to:  Settings → Forwarding and Receiving → Configure Receiving You should see port 9997 already listed. If 9997 is already active, you don’t need to add it again. Just confirm it’s listening. > Giving Karma & Marking the answer helps others find solutions faster! ... View more

@amimulahasun Hope it was helpful. Happy Splunking! ... View more

@splunkreal Yes, changing the root CA and server certificates in a Splunk deployment affects all cluster members, forwarders, and any external appliances using TCP‑SSL. The key is to plan the migration carefully, so you don’t break connectivity. Highly recommend you connect with Splunk support if it's a critical environment using SSL/Secure connections widely across the cluster and with external appliances. General Approach could be:  Add the new root CA to the trust store first by placing the new certificate on the auth folder (on all Splunk nodes and appliances). This ensures both old and new certs are trusted during the transition. File path: $SPLUNK_HOME/etc/auth/  Edit the [sslConfig] stanza to reference both the old and new CA files. Splunk supports a comma‑separated list of CA certs, so you can trust multiple roots during migration. Update outputs.conf (for UF/HF) or inputs.conf (for HF receiving TCP‑SSL) to reference the new CA file. Issue new server certificates signed by the new CA, with proper CN/FQDN matching. Roll out certificates gradually and you may follow the below order. Cluster Manager → Indexers → Search Heads → Forwarders → Heavy Forwarders → External appliances. Test connectivity with the new CA cert using below openssl commands. This confirms the new CA is trusted openssl s_client -connect <splunk_host>:9997 -CAfile newRootCA.pem Remove the old root CA only after all nodes and clients have switched. References: How to Configure Splunk Forwarding to Use Your Own Certificate | Splunk Configure Splunk Web to use TLS certificates | Splunk Enterprise (last updated 2025-07-04T03:47:01.961Z)   ... View more

@drychan Yes, it was fixed on Splunk Enterprise 9.4.8, the bundled MongoDB versions are patched and CVE‑2025‑3085 is fixed with this version. ... View more

@amimulahasun I have done similar activity to optimize license usage and were successful to certain extent. Considering your scenario, the log sources (Windows, Linux, AD, Firewalls, Proxy, VPN, PAM, IDS/IPS, etc.) you mentioned are data heavy and depending on the environment, I presume 50 GB per day would not be sufficient even if you optimize or filter security-only events. The key is to categorize logs by device type, prioritize them, and enforce ingestion controls so that only security-relevant data consumes your 50GB/day license. Regarding the architectural consideration,  Whether logs should be filtered at the device level or Splunk layer Filter as close to the source as possible (device level), then use Splunk forwarders for fine-grained control. This ensures you don’t waste license on low-value logs and reduce processing burden on Splunk. The role of Universal Forwarder vs Heavy Forwarder Universal Forwarder is a lightweight agent, designed only to collect and forward raw data. UF has filtering capability before forwarding to Splunk infrastructure using inputs.conf  - whitelist/blacklist specific files, directories, or Windows Event IDs. You can also use limits.conf  to control event size or throughput.  For example, to filter Windows Security logs using specific event IDs,  [WinEventLog:Security] disabled = 0 whitelist = 4624,4625,4672,4688 blacklist = 4634,4662 For advanced filtering options, you can use regex patterns to omit event patterns. Like, to ignore Windows Event Code 4662 events whose Message field contains events with the value Account Name: "example account", add the following line to the inputs.conf file: [WinEventLog:Security] blacklist1 = EventCode = "4662" Message = "Account Name:\s+(example account)"   Heavy Forwarder is a full Splunk Enterprise instance, capable of parsing and forwarding. Filtering capability can be achieved via creating custom props.conf & transforms.conf. Supports Regex-based filtering, routing, masking, anonymization. For strategy perspective, you can filter at the source, enforce forwarder-level controls, and only index logs that directly support detection use cases.  Data onboarding should be purely use-case driven. Your categorization - MUST/OPTIONAL/EXCLUDED is good, use this to categorize the existing data sources and excluded is a definite drop, MUST can be ingested and further filtered using the advanced regex patterns. This way, you preserve detection capability while keeping the 50GB/day license sustainable. For log filtering reference,  Include or exclude specific incoming data | Splunk Cloud Platform (last updated 2025-07-04T00:47:59.620Z) > Marking the answer and giving Karma helps others find solutions faster!  ... View more

@splunkreal We had a similar situation and was tied to KVStore backups in clustered environments when Splunk attempts to generate PKCS1 certificates for Mongo CLI tools. The failure prevents the backup file from being created, leaving the directory empty [/var/lib/splunk/kvstorebackup]. We fixed it by raising a case with Splunk support as KVStore troubleshooting is better to be conducted with support team. Sharing few things we tried as part of troubleshooting:  Check the KV Store status $SPLUNK_HOME/bin/splunk show kvstore-status --verbose       2. Regenerate KV Store certificate - PKCS1 [Exercise caution while performing Remove operations, take copy of existing PEM file for restoration and consult with Splunk support for your case before performing the action.] Stop Splunk services on SHC member Remove the existing PEM file in the below location and restart Splunk services will regenerate the certificate. $SPLUNK_HOME/etc/auth/keys/kvstore.pem For our case, this fixed the issue. ... View more

@brycemasterman, @partom24   As a workaround, separate Search Head capabilities from the Deployment Server such as Monitoring Console or Cluster Manager to prevent the server from returning API results for other servers (search peers) than itself.  Ref: Agent Management page on Deployment Server not loading after Splunk Enterprise 10.2 upgrade | Splunk    > Marking the answer and giving Karma helps others find solutions faster! ... View more

@PRusconi91 Thanks for the detailed note, I guess it's a Splunk indexer cluster bucket corruption scenario which we faced previously. The below section might help on your query for Delete Copy query. > Marking the answer and giving Karma helps others find solutions faster! From Splunk:  "You can delete either a single copy of a bucket on a specific peer, or all copies of a bucket across the entire cluster. If deleting a single copy causes the cluster to lose its complete state, the cluster will engage in fixup activities so that the bucket again meets both the search factor and the replication factor. This situation might result in another copy of the bucket appearing on the same peer. If, however, the specified bucket is frozen, the cluster does not attempt any fixup activities."  Ref: Anomalous bucket issues | Splunk Enterprise       ... View more

@jovnice Refer to my earlier response and let us know if that gives you direction. ... View more

@richgalloway Thanks for the clarification. Splunk AI Assistant for SPL version 1.3.0 and higher offers Splunk Enterprise on-premises customers the option to use the app through a new, cloud connected solution. Activate Splunk AI Assistant for SPL through cloud connected - Splunk Documentation   ... View more

@jovnice  Splunk Enterprise upgrade is not tied with your license, so you are good to proceed after fixing the issues in KV Store. For KV Store issues in Splunk, check for errors or warnings in mongod.log file. Located under the path :  $SPLUNK_HOME/var/lib/splunk/kvstore/mongo I highly recommend you to collect these ERROR details and raise a Support case with Splunk before trying any remediation from your end since KV Store troubleshooting can be critical for the entire Splunk setup. Had few bad experiences in the past with KV store troubleshooting and eventually contacted Splunk Support to recover. For the Lookup error on Splunk Security Essentials, it could be related to either a corrupted or missing lookup file or running an app version not aligned with your current Splunk version, which is 9.0.0. Hope this helps and happy to help if you have any further questions. 😊 > Marking the answer and giving Karma helps others find solutions faster! ... View more

@richgalloway - I have read the blog you shared and it's talking about enabling AI assistant in Cloud only which I am aware. My question is specifically on availability of AI assistant on Splunk enterprise on-prem deployments. Thanks. ... View more