Hi, I’m trying to use Splunk as a log aggregation solution, and eventually as a SIEM. I have three industrial plants that are completely air-gapped, with no permanent internet access. The idea is to deploy a syslog server at each plant to collect logs locally and then forward them to a central Splunk installation. Any component or software will be: downloaded or installed during a temporary internet connection (via a cellular modem), then moved into a fully air-gapped production environment. I’ve reviewed SC4S (Splunk Connect for Syslog) through the official Splunk documentation and several videos. In theory, it looks like a very powerful and well-designed solution. However, in practice, I find the documentation quite difficult to follow, especially when considering: an air-gapped environment, no internet connectivity, and very large log volumes (high EPS / high throughput). My main questions are: From a practical and low-complexity perspective: Is it better to use SC4S, or to simply deploy syslog-ng or rsyslog (open-source solutions) on Linux servers at each plant and forward logs to the central Splunk instance? Best deployment model for an air-gapped industrial environment: SC4S standalone at each site? Simple syslog collectors with forwarding? Which option is more stable and easier to operate long term? Given a preference for open-source solutions: Is relying on syslog-ng / rsyslog considered a professionally acceptable approach with Splunk? Or has SC4S effectively become the best practice that should not be avoided? From an operating system perspective: Which is better suited for handling very large volumes of log data? Ubuntu Server CentOS / Rocky Linux / AlmaLinux Which is more stable and easier to maintain in a 24/7 production environment? The end goal: A stable solution Simple to operate Capable of handling very large data volumes Suitable for air-gapped industrial environments Without introducing excessive operational complexity I’m trying to find a balance between simplicity and best practices. I want to use Splunk correctly, but at the same time avoid introducing operational complexity that exceeds the team’s current capabilities. Any advice or real-world experience would be greatly appreciated. ... View more

Hello Splunk Community, I am facing an issue and would appreciate your guidance. Currently, I am sending threats (Notable Events) to Splunk SOAR using the Splunk App for SOAR Export. Previously, the threats were delivered to SOAR almost instantly (near real-time) without any noticeable delay. However, after installing Splunk Enterprise Security version 8.1, I started to experience a delay of approximately 10–15 minutes before the threats reach SOAR. Observations: System resources are sufficient: vCPU: 90 RAM: 60 GB After installing ES, there is a significant increase in CPU utilization, with multiple splunkd and python3.9 processes consuming high CPU, for example: %CPU %MEM COMMAND 165.0 4.3 splunkd 114.9 0.3 splunkd 72.6 0.2 splunkd 66.7 0.2 splunkd 66.0 0.2 splunkd 24.4 0.0 python3.9 Connectivity between Splunk and SOAR is healthy, with no connection errors. After restarting the Splunk platform, threats are sent quickly with no delay. After some uptime, the delay returns again. No configuration changes were made on the SOAR side. The only change in the environment was installing Splunk Enterprise Security 8.1. My questions: Is this behavior known in ES 8.1? Could any settings related to: Adaptive Response Notable Events pipeline Correlation Search Scheduler cause this delay? Which logs or metrics would you recommend checking to identify the root cause? Any guidance or similar experiences would be greatly appreciated. Thank you in advance  ... View more

  Hello Splunk Community, I'm encountering an issue with a delay in sending threat data to Splunk SOAR. The delay is approximately 10-15 minutes. What's odd is that when I restart the platform, the threats are sent much faster, with no delay. However, the delay starts occurring again after some time. My environment has good resources, and there were no delays a few months ago. I’m unsure what might be causing this behavior, and I’d appreciate any insights or suggestions you may have. Additionally, if no solution is found, I’m considering reformatting the platform but would like to know if there's a way to do so without losing any data. Thank you in advance for your help! ... View more

  I am using sendtophantom in Splunk ES to send events to SOAR. The action shows success in the logs, but the events reach SOAR with a delay of about 8 minutes. ... View more

  Hello Splunk community!  I’m planning a project that involves sending event data from Splunk to Elasticsearch using Kafka, and I haven’t started implementing it yet. I’m looking for your guidance nd assistance to implement this solution efficiently and securely. I would greatly appreciate any advice, recommendations, or ready-to-use tools to help with: Extracting data from a Splunk index. Sending the data to Kafka as a temporary transport layer. Formatting events in JSON with a consistent schema. Ingesting data into Elasticsearch with proper index settings. Any experience, suggestions, or guidance from the community would be highly appreciated    ... View more

Hello, I’m experiencing an issue with my Kafka broker integration with Splunk. The error message I’m seeing is: Kafka Broker Error Topic bytes in The rate, expressed in bytes per second, of the message traffic each topic is receiving from producing clients Error: Kafka topics are not receiving events. Monitor for an hour, if the status does not go back to OK, restart kafka-server. If this does not help, contact Splunk Support. Status code KAFKA-1 I have monitored for more than an hour, and the status did not return to OK. Has anyone faced this issue before or have suggestions on how to fix it? Thanks in advance! ... View more

Hello Splunk Community, I am a new Splunk UBA user and I am currently trying to pull data from Splunk Enterprise into Splunk UBA using time-based searches. I have followed the official UBA documentation and configuration guides, but I am still struggling to fully understand the process due to my limited experience with UBA. Specifically, I am trying to: Configure UBA to pull notable events and risk events from Splunk Enterprise Security. Use time-based searches to collect data efficiently without missing events. Understand how SPL queries and External Alarm categories work in this integration. I would greatly appreciate any guidance, examples, or best practices you can share to help me successfully implement this integration. Additionally, if you could suggest external resources, tutorials, or blogs that explain the process in a beginner-friendly way, that would be extremely helpful. Thank you in advance for your support ... View more

How enaple TLS in splunk platform from ca  ... View more

  Hello everyone, I’m encountering an issue when trying to enable secure HTTPS access on Splunk Web using an SSL certificate issued by a trusted external CA. What I did: Placed the SSL certificate file (splunkWeb.pem) in the following path: $SPLUNK_HOME/etc/apps/webTLS/certs/splunkWeb.pem Edited the web.conf file with the following settings:   ini CopyEdit [settings] enableSplunkWebSSL = true serverCert = $SPLUNK_HOME/etc/apps/webTLS/certs/splunkWeb.pem privKeyPath = $SPLUNK_HOME/etc/apps/webTLS/certs/splunkWeb.pem   Restarted the Splunk service. Issue: After restarting, Splunk hangs during startup and the web interface does not become available over HTTPS. Questions: Are there additional steps required when using an external SSL certificate? Is the web.conf configuration correct, especially regarding the privKeyPath pointing to the same .pem file as serverCert? Should the private key be in a separate file from the certificate? Any advice or similar experiences would be greatly appreciated. Thank you in advance for your help! ... View more

Hello everyone, I have a network monitoring system that exports data via IPFIX using Forwarding Targets. I am trying to receive this data in Splunk using the Splunk Stream app. The add-on is installed and Stream is enabled, but I am facing the following issues: Templates are not being received properly. The data arrives, but it's unreadable or incomplete. I need full flow data, including summaries or headers from Layer 7 (e.g., HTTP, DNS). My question: Has anyone successfully received and parsed IPFIX data in Splunk? If so, could you share the steps or configurations you used (like streamfwd.conf, input settings, etc.)? Any guidance would be greatly appreciated! Thanks in advance! ... View more

Hi Splunk Community, I'm currently integrating Flowmon ndr as a NetFlow data exporter to Splunk Stream, but I’m encountering a persistent issue where Splunk receives the flow data, yet it’s not decoded properly, and flow sets are being dropped due to missing templates. Here’s the warning from the Splunk log: ``` 2025-06-21 08:34:49 WARN [139703701448448] (NetflowManager/NetflowDecoder.cpp:1282) stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 258 received for observation domain id 13000 from device 10.x.x.x. Dropping flow data set of size 328 ``` Setup details: Exporter: Flowmon Collector: Splunk Stream  Protocol: NetFlow v9 (also tested with IPFIX) Transport: UDP  Template Resend Configuration: Every 4096 packets or  600 seconds Despite verifying these settings on Flowmon, Splunk continues to report that the template ID (in this case, 258) was never received, causing all related flows to be dropped. My questions: 1. Has anyone successfully integrated Flowmon with Splunk Stream using NetFlow v9? 2. Is there a known issue with Splunk Stream not handling templates properly from certain exporters? 3. Are there any recommended Splunk Stream configuration tweaks for handling late or infrequent templates? Any insights, experiences, or troubleshooting tips would be greatly appreciated. Thanks in advance! ... View more

opt/caspida/bin/Caspida setuphadoop ...............................Failed to run sudo -u hdfs hdfs namenode -format >> /var/vcap/sys/log/caspida/caspida.out 2>&1 Fri Jun 2 17:06:11 +07 2023: [ERROR] Failed to run hadoop_setup [2]. Fix errors and re-run again. Error in running /opt/caspida/bin/Caspida setuphadoop on 192.168.126.16. Fix errors and re-run again. i execute the command /opt/caspida/bin/caspida setup but stopped here hadoopsetup can't run, i don't know the cause yet. someone please help me. I put some install logs here ... View more

Hi Splunk Community, I’m working on a use case where data is stored in Elasticsearch, and I’d like to use Splunk solely as an interface for visualizing and querying the data using SPL (Search Processing Language) — without ingesting or storing the data again in Splunk, to avoid duplication and unnecessary storage costs. My main questions are: Is there a way to connect Splunk directly to Elasticsearch as an external data source? Can Splunk query external data (like from Elasticsearch) using SPL, without indexing it? Are there any available add-ons, modular inputs, or scripted solutions that allow this type of integration? Is this approach officially supported by Splunk, or would it require a custom integration? I’m aware that tools like Logstash or Kafka can be used to bring data into Splunk, but that’s exactly what I’m trying to avoid — I don’t want to duplicate the data storage. If anyone has experience with a similar setup, or any recommendations, I’d greatly appreciate your input. Thanks in advance!   ... View more

Dear Splunk Community, I am currently working on a project focused on identifying the essential data that should be collected from endpoints into Splunk, with the goal of avoiding data duplication and ensuring efficiency in both performance and storage. Here’s what has been implemented so far: The Splunk_TA_windows add-on has been deployed. The inputs.conf file has been configured to include all available data. Sysmon has been installed on the endpoints. The Sysmon inputs.conf path has been added to be collected using the default configuration from the Splunk_TA_windows add-on. In addition, we are currently collecting data from firewalls and network switches. I’ve attached screenshots showing the volume of data collected from one endpoint over a 24-hour period. The data volume is quite large, especially in the following categories: WinRegistry Service Upon reviewing the data, I noticed that some information gathered from endpoints may be redundant or unnecessary, especially since we are already collecting valuable data from firewalls and switches. This has led me to consider whether we can reduce the amount of endpoint data being collected without compromising visibility. I would appreciate your input on the following: What are Splunk's best practices for collecting data from endpoints? What types of data are considered essential for security monitoring and analysis? Is relying solely on Sysmon generally sufficient in most security environments? Is there a recommended framework or guideline for collecting the minimum necessary data while maintaining effective monitoring? I appreciate any suggestions, experiences, or insights you can share. Looking forward to learning from your expertise. ... View more