@Utkc137 I believe  search_process_memory_usage_threshold enforces memory per individual search process on each node - so for your 3-indexer scenario, try to set the threshold to cover only the highest single consumer (~7000 MB for Indexer 3 at 6GB peak), not the 15GB+ cluster-wide sum, and build in buffer below your hard stop since termination is not exact.  ... View more

@yuanliu Hello! because  foreach  is a "template" command that can handle wildcards (like  foreach * ), it forces the search engine to switch to Full Materialization. The reason your search slows down is exactly what you suspected: the "brute force" extraction. Can you try run your search again, but put | fields a b c d (the fields you are renaming) immediately before your macro. This tells Splunk to throw away the extra data before the foreach forces the full extraction.?     Cheers!   ... View more

@b17gunnr Hi!! based on what I can think of this might help? index=_internal sourcetype=scheduler status="skipped" | timechart count BY savedsearch_name limit=5 | addtotals col=t labelfield=_time label="Total_Hosts"     Then in Visualization:   1. Chart Type: Column Chart   2. Format → Chart Overlay → On   3. Overlay field: Select Total_Hosts (or whatever the total field is named)   4. Chart Overlay type: Linear     This gives you:   - Column chart: Individual savedsearch skip counts   - Line overlay: Total volume trend across all hosts ... View more

@SSEAL I'm glad it worked out for you. Please mark this as solved.  ... View more

Hi @Priya70,without seeing the actual splunkd.log entries during the stall periods, its hard to answer. However, based on your symptoms, the most likely cause is backpressure.   Why backpressure fits your pattern:   - High-volume classic logs (Application/Security/System) pause first   - Lower-volume custom channels (Cisco VPN) continue uninterrupted   - Multiple input types affected simultaneously (monitor, registry, scripted)   - Automatic recovery after queues drain   To confirm, check splunkd.log during stall periods for:   - "queue is full" messages   - TCP connection errors to indexers   - Network timeout warnings   Other possibilities to rule out:   - Windows Event Log API resource exhaustion   - UF memory pressure   - Windows Event Log service issues index=_internal host=<UF> source=*metrics.log* OR source=*splunkd.log* tcpout     Hope this helps narrow it down! ... View more

@xbones84 Hi! Since you're using HEC event endpoint.? (/services/collector/event), the sourcetype and metadata are already set when the data reaches Splunk. The AWS Add-on's props.conf configurations don't get applied to HEC event endpoint data because the metadata is predetermined in your JSON payload. The AWS Add-on assumes data will flow through its inputs where it can apply parsing-time transformations. With HEC event endpoint, you bypass that processing, so you need to handle CIM compliance upstream (in Cribl) or with custom Splunk configurations.  ... View more

Hi @SSEAL, You're absolutely right about the documentation - Docker Engine does NOT need to be manually installed. The PSA installation package includes an automation script that handles everything for you. When you run the installation command, the script will:   - Automatically detect if Docker Engine is installed   - Install Docker Engine if it's not present   - Build the required Docker images   - Deploy the PSA containers  So you can start with a fresh RHEL server and let the PSA installer take care of the Docker setup completely. Yes, most of those parameters can be modified after the initial installation, but there are some nuances. Shepherd-URL: This is your AppDynamics Controller  https://YOURURL>.saas.appdynamics.com/controller Chrome-agent:  This is the browser-based synthetic monitoring component that runs Chrome/Chromium to execute synthetic browser tests (like clicking through web applications). The memory and CPU parameters control how much system resources these browser tests can consume.     ... View more

@MakszimM  If you have the original plain text pass4SymmKey, you can simply run: $SPLUNK_HOME/bin/splunk edit cluster-config -mode peer -manager_uri https://<manager_IP>:8089 -secret <original_plain_text_pass4SymmKey>   The -secret parameter takes the original plain text key, not the hashed version from the config files. Splunk will automatically hash it during the configuration process. Forgot pass4SymmKey? https://community.splunk.com/t5/Security/Forgot-Pass4symmKey/m-p/378993 If this Helps, Please Upvote! ... View more

Hi @MakszimM, Regarding the SSL certificates - the existing indexers having different certificate files is normal and acceptable. Each indexer can maintain its own SSL certificates. For your new 3rd indexer, you can either generate new unique certificates or copy existing ones based on your security and operational requirements. For adding the 3rd indexer, follow these steps: On the New Indexer Peer:   • Configure the new server as a peer node to the existing indexer cluster:   $SPLUNK_HOME/bin/splunk edit cluster-config -mode peer -manager_uri https://<cluster_manager_IP>:8089 -replication_port 9887 -secret  <your_cluster_key>   • Restart Splunk for changes to take effect:   $SPLUNK_HOME/bin/splunk restart   • Verify the peer appears in the cluster manager dashboard   On the Cluster Manager:   • Update the Replication Factor and Search Factor (RF=3, SF=2 for 2 peer failure tolerance)   • Apply the cluster bundle to distribute configuration changes   • Rebalance the cluster data to distribute existing buckets to the new peer:   $SPLUNK_HOME/bin/splunk rebalance cluster-data   • Monitor the rebalancing process in the Cluster Manager dashboard   • Verify bucket distribution and remove any excess buckets if present     Reference Links: https://help.splunk.com/en/splunk-enterprise/administer/manage-indexers-and-indexer-clusters/9.0/manage-the-indexer-cluster/add-a-peer-to-the-cluster https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Rebalancethecluster   The key point is ensuring the cluster secret matches across all peers, while SSL certificates can remain unique per indexer.   ... View more

@danielbb No, _internal does not proactively report certificate expiration dates. It only logs SSL errors after certificates have already expired, which is too late for proactive monitoring.   What _internal shows:   index=_internal sourcetype=splunkd component=TcpInputProc log_level=ERROR "SSL routines:SSL3_READ_BYTES:sslv3 alert certificate expired"   This only appears after the cert has expired.   For On-Prem: may be create a scripted input to check certificate expiration and monitor?     #!/bin/bash   # Script to check cert expiration   openssl x509 -enddate -noout -in /path/to/cert.pem If this Helps, Please Upvote ... View more

@dtaylor The issue might be because of your bin command  which creates time buckets but doesn't give them meaningful shift names.      Try to add shift logic after your existing binning: | inputlookup shift_tickets.csv | eval closed_date = strptime(closed,"%a, %d %b %Y %T %Z"), _time = strptime(occurred,"%a, %d %b %Y %T %Z") | addinfo | where _time>=info_min_time AND (_time<=info_max_time OR info_max_time="+Infinity") | eval hour_min = strftime(closed_date, "%H:%M") | eval shift = case( hour_min >= "06:30" AND hour_min < "14:30", "Day Shift", hour_min >= "14:30" AND hour_min < "22:30", "Evening Shift", true(), "Night Shift") | stats count by shift For weekly trends: | eval week = strftime(closed_date, "%Y-Week-%U") | stats count by week, shift This keeps your time filtering logic but adds the shift names you need. Much easier than trying to decode those binned timestamps! Hope this helps. If this Helps Please Upvote. ... View more

@tdavison76 try something like this? | rex field=InterfaceDocument "MessageType\":\"(?<MessageType>[^\"]*)" If this Helps, Please Upvote. ... View more

@viku7474 "ui_inactivity_timeout" is the trigger to start the countdown for the session option to change the setting for their entire UI stack access for all users. For example, if you set "ui_inactivity_timeout" to "0," then it will never timeout. https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.0/welcome-to-splunk-enterprise-administration/manage-users/configure-user-session-timeouts May be this might help, old answer relevant to this: https://answers.splunk.com/answers/7233/user-specific-browser-session-timeout.html     . . . ... View more

@JohnGreggFrom what I've read, Java agents should be dividing by processor count: CPU millis / time / processor_count But if you're not seeing that in your K8s results, maybe the cluster agent works differently than regular Java agents? For the cluster agent metrics, I know it treats 1 CPU = 100%, so multi-core usage gives you >100%. Might be worth checking if you're looking at app agent metrics vs cluster agent metrics - they could calculate differently. If this helps, Please Upvote. ... View more

@oawill  mltk_ai_commander_dataset.csv is a training dataset that ships with MLTK 5.6.0 and higher specifically for hands-on exercises with the LLM integrations feature. This dataset contains example data used to train models to distinguish between malicious and benign PowerShell scripts. Since this is an example training dataset provided by Splunk for educational/demonstration purposes with the MLTK, the specific authorship and attribution details may not be extensively documented in the traditional dataset credits format. The dataset is designed as sample data for users to practice with the AI Commander functionality. If this Helps, Please Upvote. ... View more

@isoutamo is exactly right about the 9.2 changes! To help troubleshoot this further, you should check a few things to understand why the forwarders aren't connecting properly to the DS. Start by testing connectivity from each forwarder using telnet or netcat to make sure they can actually reach the deployment server on port 8089. Next, examine your serverclass.conf on the Deployment Server to verify that your forwarders match the whitelist criteria and that the client matching is configured properly. Many times the issue is that the serverclass isn't set up to recognize your specific forwarders. On the forwarder side, run btool deploymentclient to see what configuration is actually being applied. This will show you if there are any conflicting settings or if the deploymentclient.conf isn't pointing where you expect it to. If your deployment server is forwarding its internal logs to your indexer, you might also need to add the indexAndForward settings in outputs.conf on the DS, as this can affect how deployment client data appears in the management UI after 9.2. Just to confirm, are you also managing your Search Head and indexer through the Deployment Server? And is this truly a distributed setup with separate VMs, or multiple Splunk instances on one box? That architecture detail might help explain what you're seeing. If this Helps Please Upvote! ... View more

@heathramos Thanks for the update, glad it worked out.  ... View more

@heathramos  yeah this is going to be a fun one. You've got data model issues, which is way more involved than just fixing a macro. Data models are these complex hierarchical things with parent/child datasets that need to be built and accelerated properly - it's a whole thing. Looking at that search, it's trying to pull from datamodel=pan_firewall with specific node relationships. If that's not set up right (or at all), nothing's going to work. And troubleshooting data models means digging into dataset structures, field mappings, acceleration status - it's honestly not a quick fix. If you need this dashboard working soon and it's important to the business, you might want to just work with Splunk ondemand services. They can sort out your data models properly instead of you spending days figuring out why the acceleration isn't working or why the field extractions are wrong. If you want to try,  spend  some time in Settings > Data Models, checking what's actually there vs what the dashboard expects. You'll probably end up either rebuilding data models from scratch or rewriting all these tstats searches to use regular SPL. It's more like -audit your entire Palo Alto data ingestion and modeling setup. If this Helps Please Upvote.   ... View more

@heathramos  Since the p_index macro doesn't exist, here's how to dig into the dashboard and fix it: Edit the dashboard directly: Go to your  dashboard Click the Edit button (top right) Click on each panel/visualization that's not showing data Click Edit Search for each one magnifying glass You'll see searches that start with `p_index` sourcetype="pan:xdr_incident" Replace the macro with your actual index: Change `p_index` to index=pan (or whatever your actual Palo Alto index is called) So the search becomes: index=pan sourcetype="pan:xdr_incident" Do this for every search in the dashboard. There are probably 8-10 different searches based on your dashboard config. While you're in there, also check: Does sourcetype="pan:xdr_incident" match your actual data? Run index=pan | stats count by sourcetype first to confirm If your sourcetype is different (like pan:incident or cortex:xdr), update those too   ... View more

@heathramos Your dashboard isn't populating because it's looking for data in places that don't exist in your environment.  The main culprit is probably the p_index macro. Your dashboard is using `p_index` but this macro either doesn't exist or isn't pointing to the right place. Go to Settings > Advanced Search > Search Macros and see if you have one called p_index. If not, create it. If yes, make sure it's set to your actual Palo Alto index. tip: When you're in the Search app, you can Cmd+Shift+E (Mac) or Ctrl+Shift+E (Windows) to expand macros in your search and see what they actually resolve to. This will show you exactly what `p_index` is doing. Second issue - sourcetype mismatch. The dashboard expects sourcetype="pan:xdr_incident" but your data probably has a different sourcetype. Run this to see what you actually have: index=pan | stats count by sourcetype Quick test: Try running the base search manually with your actual values instead of the tokens. Replace $severity$ with * and see if you get any results. The dashboard is basically looking for some field names like incident_id, severity, status etc. If your XDR data doesn't have these exact field names, nothing will show up. Most of these Palo Alto app dashboards assume you've configured everything exactly as Palo Alto intended, but real environments are messier. You'll probably need to either: Fix your data inputs to match what the dashboard expects, OR Edit the dashboard searches to match your actual data structure Start with that macro expansion trick and sourcetype check - those are usually the smoking guns. Good luck! If this Helps, Please Upvote. ... View more

@nareshkareeti try  | tstats summariesonly=true count FROM datamodel=Network_Traffic WHERE (All_Traffic.src_ip=* OR All_Traffic.dest_ip=*) BY All_Traffic.src_ip, All_Traffic.dest_ip | `drop_dm_object_name("All_Traffic")` ... View more

@GeneralBlack we might need to re-install the previous splunk version for this, best approach is to work with support. https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.3/administer-the-app-key-value-store/migrate-the-kv-store-storage-engine#Upgrade_KV_store_server_to_version_4.2 Try this go to login.splunk.com > support > support portal  > Need help? > Create a Case if this Helps, Please Upvote ... View more

@GeneralBlack Please work with splunk support, may be its missing the the mongod folder and it was not created after upgrade? ... View more

@Lien unfortunately, its not supported for the splunkcloud trial version. https://docs.splunk.com/Documentation/SplunkCloud/latest/Admin/TypesofSplunkClouddeployment       If this Helps, Please Upvote! ... View more