@richgalloway is spot on.   i have seen that now with Splunk 10 if you are using standalone indexers, you can do post triage.  what I mean by that is you can move the logs from one index to the other.  This is not a solution, this is just triage. I am sure we don't need to tell you that in the ideal world, you would have more control on the process, but since you don't have that, just be aware that if you know that data was sent to the wrong index you can move it to another index with the new feature in splunk 10.  It at this time, does not work on indexer clusters, but a standaone instance will allow you to do this.  I am attaching a video of the process.  Maybe it will save a little bit of the headache after the event of logs being sent to the wrong place.  The youtube video shows how you can move logs from one index to another in splunk 10.   https://youtu.be/QXhStVC-nlc ... View more

I'm with livehybrid.  A little more information will be helpful in allowing us to help you.   If you are on the cloud, or if you are on premise will definitely change the way we approach this problem.  If it is local, do you have a proxy like nginx in front of it?  With a local instance, we will need to know if you have anything in front of that box that could be causing the redirect.  Generally speaking (bugs are real and thus I speak in generalities) splunk doesn't do much redirecting, so I would assume it is another tool on the network between you and the splunk instance that is causing the problem.  But that is just a guess without more info and / or screenshots.   ... View more

I am hoping this is of help, never really ran into this problem, but I would say the fact that you have a constant rolling restart of your search heads is probably not a good thing.  So one of the things I had happen to me a long time ago, was I pushed a bad config to my search head cluster and it got stuck in that constant rolling restart.   I ultimately was able to fix it by shutting down the whole cluster, bringing up one machine, getting that config file fixed and then moving onto the next search head.  Not sure if you do or do not have a bad config and I am sure this is not a great idea in a production environment.   If the search heads are still working - then you may not want to try this method, but if they aren't working, bringing them all down doesn't technically make things worse.   I wish I had a better answer for what exactly you need to change to get the machines to stop rolling restarting, but maybe my above suggesiton of bringing it down and then restarting it from the CLI and look for any errors in the start up or in the splunk internal logs.   ... View more

You have a few options and each have their own pros and cons and without knowing the data, I can only make an estimated guess on what would work best for you. Data Acceleration - you could put your data into data models - either existing or custom ones that fit your data and accelerate the data.  This will "accelerate" your data which in theory should significantly boost the speed at which you search,  Mileage may vary, but often you get orders of magnitude faster searching.  The cons with this is that you are probably going to double the size of your "indexed data" because acceleration is keeping your non accelerated logs and putting a set of accelerated data on the index meaning that you will be using more storage space.  Additionally, every 5 minutes or so, your accelerated data will be running the search to accelerate your data and that it going to occupy ram and cpu permanently on the box.  Plus depending on your comfort building or fitting your data into a datamodel, this is a little labor intensive to set up the first time.  As for RBAC, Splunk will maintain the same rbac rules to your accelerated data as exists on the index, so you won't need any special rbac considerations.  Summary indexing - This is an amazing tool for doing exactly that, summarizing the data.  For example if you have network logs - you have probably seen that in a given time period when two machines talk to each other, you may find that you have 100s of "connection logs".  If your use case is not interested in each of those 100 logs, but is more interested in - did these two IPs talk - (think threat inteligence - did we go to bad site x) than you could create a single summary log that says IP address x talked to IP address y 100 times.  You write this data to a summary index.  In reality, summary data gets its speed advantages because instead of speeding up the way you look for a needle in the haystack, you shrink the haystack so it is smaller - like in my example it is 1 / 100 smaller than the original index.   This is a useful solution if a summary of the logs is good enough for what your analysts are looking for and that may or may not be the case.  In the world of threat intel, we often have to look back at network traffic 18 months.  We look at the summary data, if we have a hit, the summary data tells us what date the hit was on, but the analysts may have to go look at the unsummarized log for that day to get a better idea of what really happened because summary logs gain their power by being exactly that - a summary.   For RBAC purposes, you can just make your summary index reside on the same index that it was created for.  The term summary index implies that you have a special index, but that is not really the case.  A summary index can be written to any index it is just a new source and the sourcetype is stash.  So if you summarize your data to the same index that the original logs came from, they will have the same rbac rules on them.   Here is a video on how to summarize data  https://youtu.be/mNAAZ3XGSng Below is a simple spl concept of summarizing palo alto firewall logs index=pan sourcetype=connections | stats sum(bytes_in) as bytes_in sum(bytes_out) as bytes_out earliest(_time) as _time count by src_ip, dest_ip   | collect index=pan source="summarized_pan_connections" You now need to determine how often you are summarizing your logs and set up a saved search to run that query.  Once it runs you just query the data with  index=pan source="summarized_pan_connections" Another option you can have is to schedule search your dashboard panels- this means that each of the panels will run the query one time at some specified time and everyone who comes to the dashobards will get the data that was created during the scheduled search.  This is relatively simple to set up, keeps rbac rules, but if having the latest logs included on the dashboard panels is your biggest priority, this one starts to fall apart.   I have given three suggestions, in my environment I have a similar situation as you, large amount of data and looking back long periods of time is slow.  We actually run a little mixture of all of it.  We accelerate a days worth of data, then in the middle of the night, we summarize yesterdays logs.  Then when the users search the dashboard the query is a combination of the accelerated data for today's data, and the summarized data for the previous days data.   Hope this gives you some ideas of a path forward.  There will be plenty of things that you need to consider, particularly how "fresh" does the data need to be.  Is the summary of the logs good enough, can you have static data in your dashboards that refreshes every day or every hour?    ... View more

I have only dabbled in this area, but I am pretty sure (not going to place any bets in Las Vegas on it though) that your python code is going to determine where you write the logs.  It would be awesome if it does write to the splunk logs area, but I am pretty sure it does not.  I don't think you specified a location for where you are writing the logs, so my guess is the python script is going to try to write the logs to the same location as your .py script.  Now someone could come along and tell me I am completely wrong, and that is ok, because as I said, this is something I have only done a couple times and it was years ago. If I were going to troubleshoot to find out where the logs are being written, the first thing I would do is spin up my local / dev instance of Splunk.  I really encourage anyone doing development, especially on something as complicated as what you are doing, that you have a dev instance - whether that be on a spare computer or laptop or spin up a local vm, or whatever, but it is really difficult to troubleshoot on a production environment and from an app dev perspective this is also a good practice to have a test environment.  On this test box, you should have access to the command line.  Put your code on that test box and then look and see where the logs are being written.   Sorry I don't have any silver bullet, but my guess is that the log file is "trying" to be written to the same location as your python script, and that means your inputs.conf needs to be pointed there as well or it won't be able to grab it.  (I don't recommend writing your logs to your scripts folder so you probably will want to change its location in the python script) ... View more

This is not my area of expertise, I have done a little bit with the custom commands to know you are on the right path, but looking at your code is not making anything obvious pop up, but I do know the most common issues to ingesting logs that are created by firing off splunk actions (which is what you are doing) is rights. I have had two issues pop up when doing what you are doing, one was easy to fix, the other was one that resulted in a lot of head banging and frustration but I found a workaround that worked in our environment.  The first thing you want to validate is that logs are actually being created.  I am sure you are doing this, but as a person who has done everything wrong in Splunk, I have actually tried to troubleshoot why my logs are not coming in only to find that no physical logs actually exist.  After you verify that I recommend putting a "test" log in the exact same site as your python logs.  Can you ingest the "test" log.  If you can't, you know that it is probably related to rights.   If you can have splunk ingest the "test" logs you may be in the glitchy world that no one has ever truly explained to me, but it relates to where you write the logs.  Logs that are dynamically created in Splunk (which is what you are doing) for some reason could not be read in certain locations on the disk, even though it could read the "test" logs.   So ultimately I found that I had to change the location of the logs being written to another location on disk and then pull the logs from there.  You are making me have to think back on painful traumatic times, but I think anytime I tried to write the logs inside the directory of the app that was built to make the custom command, it would not read.  But when I moved it to /var/logs it worked.  I hope this is not the problem you run into, and hopefully the pain and trauma I suffered from has been fixed or there was some other underlying issue that I was dealing with and you will never have to experience this, but it was enough to make me still wake up in the middle of the night with nightmares 🙂  But hopefully everything can be attributed to Splunk not being able to read the location of your python logs, you change the permissions and everything works.  If not, hopefully someone has a silver bullet in these answer forums, and if that doesn't work just try different locations on your OS and see if they work (I know this cannot be the true answer, but it was what ultimately worked for me) ... View more

The most common thing I have heard for notifying users about changes is the notification messages and the global banner.  The messages are not my favorite method because they are often missed and if someone dismisses them, they are dismissed for everyone, so I encourage using a global banner.   If you are using the Web interface to do this, banners can be changed by going to the upper right settings - server controls - global banner.   This is probably all you need, but if you ever run into a situation that we had where we had standalone search heads geographically dispersed across the world, we needed a method to be able to change hundreds of search heads at one time, we used SOAR automation to do this.  We would create a single dashboard that would send an alert to SOAR phantom and that would than trigger an alert that would go to SOAR and then SOAR would ssh into all of the boxes and issue the necessary changes.  This was a really cool process, but I am assuming you don't have that problem and you can just go into the web interface of one Search Head and change your banner and you will be good.   Here is a video of using soar and ssh and curl to send messages.     https://youtu.be/gd5xDNGEsoU I did not make a video of using SOAR and ssh and modifying the global-banner.conf file, but that is the file you would be looking to modify if you want to change banner notifications.   ... View more

That has to be frustrating and I don't know if I have ever seen what you are experiencing.  I would try a couple things just to see if by chance a mistake has been made.   1) This is what you said you already have done, but just validate that nothing has changed.  Try to send events manually to SOAR.  If they arrive, in theory the automated should work as well, but that does not seem to be your case.  Which leads me to  1a)  I have actually done this.  I had set up two connections to my SOAR when I was setting it up.  One of the setups had the correct credentials to hit SOAR and the other did not.  So when I set up adaptive response and it says what configuration do you want to use (can't remember the exact verbiage of the question) I picked the wrong one from the dropdown and it caused failure to connect.   2) When I started this, I had a clear idea what was my second thing I would try, but it slipped my mind as I wrote 1 and 1a.  But basically just verify that you really can manually send the alerts to SOAR and that you aren't able to send them as an adaptive response.   ... View more

Just for troubleshooting purposes, can you create a brand new event finding (what used to be called correlation search before splunk ES 8? )  What I like to do is just check to make sure if this is a problem with just this search or is systemic.  So I make my search something generic like  index=_internal | head 1 | table index, sourcetype, _time  Again the above query is just a query that you know will have results each time it runs.  Feel free to make the search anything you want.  Then plug in your drilldown using the same values you applied in your question.  When the alert fires and you click its drilldown, does it go all time or does it use the time selection that you gave it.   Again this is just to identify if this is a problem for one correlation search or for all of your correlation searches.  This will allow us to get a better idea of what is and what is not working.   ... View more

I have never seen this before and I will be completely transparent that I put your question into an AI engine so the response may not be anything close to what you are looking for, but the AI seemed to think you might be having web browser caching issues (which I have actually had the web caching problems, just never had them affect the pages you mentioned).  The recommendation is to try to clear your cache in your browser or the method I use the most often is to use incognito mode.  Again, no idea if this will help, but I do know that I have changed the navigation menus on an app and they would not update in my browser and I had to run in incognito mode or open up a different browser that hadn't cached my Splunk website to see the changes to the navigation.  Hope this helps.     ... View more

Thanks for sharing the details of your FortiGate log parsing issue in Splunk Cloud! It sounds like your Logstash server is combining multiple FortiGate logs into a single file, which is then sent to your Heavy Forwarder (HF) and ingested into Splunk Cloud as multi-line events (sometimes 20+ logs per event). Your props.conf configuration isn’t breaking these into individual events per host, likely due to an incorrect LINE_BREAKER regex or misconfigured parsing settings. The _grokparsefailure tag suggests additional parsing issues, possibly from Logstash or Splunk misinterpreting the syslog format. Below is a solution to parse these logs into individual events per FortiGate host, tailored for Splunk Cloud and your HF setup. Why the Current Configuration Isn’t Working LINE_BREAKER Issue: Your LINE_BREAKER = }(\s*)\{ aims to split events between JSON objects (e.g., } {), but it may not account for the syslog headers or whitespace correctly, causing Splunk to treat multiple JSON objects as one event. The regex might also be too restrictive or not capturing all cases. SHOULD_LINEMERGE: Setting SHOULD_LINEMERGE = false is correct to disable line merging, but without a precise LINE_BREAKER, Splunk may still fail to split events. Logstash Aggregation: Logstash is bundling multiple FortiGate logs into a single file, and the _grokparsefailure tag indicates Logstash’s grok filter (or Splunk’s parsing) isn’t correctly processing the syslog format, leading to malformed events. Splunk Cloud Constraints: In Splunk Cloud, props.conf changes must be applied on the HF, as you don’t have direct access to the indexers. The current configuration may not be properly deployed or tested. Solution: Parse Multi-Line FortiGate Logs into Individual Events To break the multi-line events into individual events per FortiGate host, you’ll need to refine the props.conf configuration on the Heavy Forwarder and ensure proper event breaking at ingestion time. Since the logs are JSON with syslog headers, you can use Splunk’s JSON parsing capabilities and a corrected LINE_BREAKER to split events. Step 1: Update props.conf on the Heavy Forwarder Modify the props.conf file on your HF to correctly break events and parse the JSON structure. Place this in $SPLUNK_HOME/etc/system/local/props.conf or an app’s local directory (e.g., $SPLUNK_HOME/etc/apps/<your_app>/local/props.conf). ini   [fortigate_log] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+)(?=\{"log":\{"syslog":\{"priority":\d+\}\}) INDEXED_EXTRACTIONS = json KV_MODE = json TIMESTAMP_FIELDS = timestamp TIME_FORMAT = %Y-%m-%dT%H:%M:%SZ BREAK_ONLY_BEFORE = ^\{"log":\{"syslog":\{"priority":\d+\}\} TRUNCATE = 10000 category = Structured disabled = false pulldown_type = true Explanation: LINE_BREAKER = ([\r\n]+)(?=\{"log":\{"syslog":\{"priority":\d+\}\}): Splits events on newlines (\r\n) followed by the start of a new JSON object (e.g., {"log":{"syslog":{"priority":189}}). The positive lookahead (?=) ensures the JSON start is not consumed, preserving the event. SHOULD_LINEMERGE = false: Prevents Splunk from merging lines, relying on LINE_BREAKER for event boundaries. INDEXED_EXTRACTIONS = json: Automatically extracts JSON fields (e.g., host.hostname, fgt.srcip) at index time on the HF, reducing search-time parsing issues. KV_MODE = json: Ensures search-time field extraction for JSON fields, complementing index-time parsing. TIMESTAMP_FIELDS = timestamp: Uses the timestamp field (e.g., 2025-06-23T15:20:45Z) for event timestamps. TIME_FORMAT = %Y-%m-%dT%H:%M:%SZ: Matches the timestamp format in the logs. BREAK_ONLY_BEFORE: Reinforces event breaking by matching the start of a JSON object, as a fallback if LINE_BREAKER struggles. TRUNCATE = 10000: Ensures large events (up to 10,000 characters) aren’t truncated, accommodating multi-log events. category and pulldown_type: Improves Splunk Cloud’s UI compatibility for source type selection. Step 2: Deploy and Restart the Heavy Forwarder Deploy props.conf: Place the updated props.conf in $SPLUNK_HOME/etc/system/local/ or a custom app directory on the HF. If using a custom app, ensure it’s deployed via a Deployment Server or manually copied to the HF. Restart the HF: On the Windows HF, open a Command Prompt as Administrator. Navigate to $SPLUNK_HOME\bin (e.g., cd "C:\Program Files\Splunk\bin"). Run: splunk restart Note: In Splunk Cloud, you can’t modify indexer configurations directly. The HF applies these parsing rules before forwarding to the Splunk Cloud indexers. Step 3: Verify Event Breaking Run a search in Splunk Cloud to confirm events are split correctly: spl   index=<your_index> sourcetype=fortigate_log| stats count by host.hostname Check that each FortiGate host (from host.hostname) appears as a separate event with the correct count. Each event should correspond to one JSON log entry (e.g., one per host.hostname like “redact”). If events are still merged, inspect $SPLUNK_HOME/var/log/splunk/splunkd.log on the HF for parsing errors (e.g., grep -i "fortigate_log" splunkd.log). Step 4: Address _grokparsefailure Tag The _grokparsefailure tag suggests Logstash’s grok filter isn’t correctly parsing the FortiGate syslog format, which may contribute to event merging. Since you can’t modify the Logstash setup, you can mitigate this in Splunk: Override Logstash Tags: In props.conf, add a transform to remove the _grokparsefailure tag and ensure clean parsing: ini   [fortigate_log] ... TRANSFORMS-remove_grok_failure = remove_grokparsefailure In $SPLUNK_HOME/etc/system/local/transforms.conf: ini   [remove_grokparsefailure] REGEX = . FORMAT = tags::none DEST_KEY = _MetaData:tags Restart the HF after adding the transform. This clears the _grokparsefailure tag, ensuring Splunk doesn’t inherit Logstash’s parsing issues. Step 5: Optimize FortiGate Integration (Optional) Install Fortinet FortiGate Add-On: If not already installed, add the Fortinet FortiGate Add-On for Splunk on the HF and Search Head to improve field extraction and CIM compliance. Install on the HF for index-time parsing (already handled by INDEXED_EXTRACTIONS = json). Install on the Search Head for search-time field mappings and dashboards. Verify Syslog Configuration: Ensure FortiGate devices send logs to Logstash via UDP 514 or TCP 601, as per Fortinet’s syslog standards.   Check Logstash Output: If possible, verify Logstash’s output plugin (e.g., Splunk HTTP Event Collector or TCP output) is configured to send individual JSON objects without excessive buffering, which may contribute to event merging. Troubleshooting Tips Test Parsing: Ingest a small sample log file via the HF’s Add Data wizard in Splunk Web to test the props.conf settings before processing live data. Check Event Boundaries: Run index=<your_index> sourcetype=fortigate_log | head 10 and verify each event contains only one JSON object with a unique host.hostname. Logstash Buffering: If Logstash continues to bundle logs, consider asking your Logstash admin to adjust the output plugin (e.g., splunk output with HEC) to flush events more frequently, though you noted this isn’t changeable.   Splunk Cloud Support: If parsing issues persist, contact Splunk Cloud Support to validate the HF configuration or request assistance with indexer-side parsing (though the HF should handle most parsing). ... View more

You have a thread from 2020 that states they fixed their problem.  I am pretty sure the reason the solution works is similar to what I am going to suggest here.  I have found (no scientific evidence to support it) that sometimes the conf files just seem to be buggered and if reset them, it starts to work.  I swear the settings are the same before the reset and after, but for some reason it works.  Maybe it's voodoo or whatever, but it has worked for me in the past. Here is a breakdown of quickly resetting the configurations that you need The warning suggests the SH is trying to query a non-existent or misconfigured search peer, possibly due to stale or incorrect settings in outputs.conf or related configuration files. Resetting outputs.conf clears any corrupted or conflicting settings (e.g., incorrect server names, ports, or SSL configurations) that might be preventing the SH from recognizing the IDX as a valid peer. Restarting Splunk ensures a clean state, and re-adding the peer re-establishes the connection with fresh, verified settings. Steps to Reset and Reconfigure Back Up Configuration Files: Before making changes, back up your Splunk configuration files to avoid losing custom settings. On the Search Head, copy the $SPLUNK_HOME\etc\system\local directory (e.g., C:\Program Files\Splunk\etc\system\local) to a safe location (e.g., C:\SplunkBackup). Delete or Rename outputs.conf: Navigate to $SPLUNK_HOME\etc\system\local on the Search Head (e.g., C:\Program Files\Splunk\etc\system\local). Locate outputs.conf. If it exists, rename it to outputs.conf.bak (or delete it if you’re sure no critical settings are needed). Note: If outputs.conf is in an app directory (e.g., $SPLUNK_HOME\etc\apps\<app_name>\local), check there too and rename/delete it. This ensures Splunk starts with default output settings, clearing any misconfigurations. Restart Splunk on the Search Head: Open a Command Prompt as Administrator on the Windows SH host. Navigate to $SPLUNK_HOME\bin (e.g., cd "C:\Program Files\Splunk\bin"). Run: splunk restart This restarts the Splunk service, applying the reset configuration. Verify Indexer Configuration: Ensure the Indexer is configured to receive data on the correct port (default: 9997). On the Indexer, check $SPLUNK_HOME\etc\system\local\inputs.conf for a [splunktcp://9997] stanza: ini   [splunktcp://9997] disabled = 0 If missing, add it and restart the Indexer (splunk restart). Confirm port 9997 is open: netstat -an | findstr 9997 (should show LISTENING). Reconfigure the Search Peer: On the Search Head, log into the Splunk Web UI as an admin. Go to Settings > Distributed Search > Search Peers. Remove the existing Indexer peer (select the IDX and click Remove). Add the Indexer as a new peer: Click Add New. Enter the Indexer’s details: Peer URI: https://<Indexer_IP>:8089 (e.g., https://192.168.1.100:8089). Authentication: Use the SH admin credentials or a pass4SymmKey (if configured in distsearch.conf). Replication Settings: Ensure settings match your setup (usually default). Save and wait for the status to show Healthy. Alternatively, use the CLI: cmd   splunk add search-server https://<Indexer_IP>:8089 -auth <admin>:<password> -remoteUsername <admin> -remotePassword <password> Test the Search: Run your search again from the SH: index=_internal. Verify results are returned without the warning. Check the Monitoring Console (Settings > Monitoring Console > Search > Distributed Search Health) to confirm the peer is active and responding. Additional Tips Check Network Connectivity: Ensure the SH can reach the IDX on port 8089 (management) and 9997 (data). Run: telnet <Indexer_IP> 8089 and telnet <Indexer_IP> 9997 from the SH host. If blocked, check Windows Firewall or network settings. Verify SSL Settings: If using SSL, ensure distsearch.conf on the SH and inputs.conf on the IDX align (e.g., ssl = true). Check $SPLUNK_HOME\var\log\splunk\splunkd.log on both hosts for SSL errors. Confirm Splunk Versions: Your SH and IDX should be on compatible versions (e.g., SH 8.2.2.1 or newer, IDX same or older). Run splunk version on both to confirm. If mismatched, upgrade the SH first. Debug Logs: If the issue persists, check $SPLUNK_HOME\var\log\splunk\splunkd.log ... View more

"Built by Splunk Works": Splunk Works is an internal initiative or team within Splunk focused on creating innovative, often experimental or community-driven apps and add-ons. Apps labeled "Built by Splunk Works" are developed by Splunk employees but may not carry the same level of formal support or certification as mainstream Splunk apps (e.g., Splunk Enterprise Security or Splunk IT Service Intelligence). These apps are often exploratory, proof-of-concept, or niche solutions. So my above statement is not completely true.  Apps marked "Built by Splunk Works" are indeed created by Splunk employees, making them "official" in the sense that they originate from Splunk Inc. However, they may not always be Splunk Supported or Splunk Certified, which is what some users mean when they refer to "official" apps. Glad you mentioned this because that does make it slightly different than being built by Splunk, certified by Splunk.   ... View more

Both answers above are correct in their response.  If you already knew this, I apologize beforehand, but one of the best ways to find out if the app is a 3rd party app, an app built by Splunk, or built by vendor of the product you are trying to ingest, go to Splunk base, look at the app and there is a field "created by"  If the answer is Splunk or Splunk works - this means that Splunk built the app.  If it said microsoft or something similar you could assume Microsoft, if it says LAME Creations (just a hypothetical example) that means someone called LAME Creations built the app.  Most of the apps built by Splunk were designed with a use case where they worked with the actual vendor or something similar that should provide you with some level of confidence this is the app.  Unfortunately the bigger issue is if the app is "STILL" the current app that is recommended by Splunk.  What I mean by that is over time, apps get recreated or rebranded into other apps and that can still be a problem with Splunk built apps so I also like to look at the version history and see if the app is relatively current.  If it is current, it means Splunk is still working on it, and that should also help provide some level of confidence this is the app.  The last method is a mixture between using Google Foo to search for what the community is using or asking on this forum - so you are already doing this.  Hope this helps.  The answer was generic, but it was just me sharing how I look at an app on Splunkbase to see if I should use it in my environment.    ... View more

In Dashboard Studio, a single click interaction on the timechart can set both global_time.earliest (the start of the clicked bar) and global_time.latest (the end of the 2-hour bar) by using a token formula. Instead of relying on a second click, you’ll compute global_time.latest as global_time.earliest + 2 hours. This ensures the exact 2-hour range is applied to other visualizations, mimicking the Search app’s timeline filtering. This is assuming that you want 2 hour chunks.  You can get crazy and tokenize the span=2h and then use that same token in the example I provide, but that is not the solution I am providing below.    Steps to Implement Verify Your Timechart Configuration: Ensure your timechart uses a 2-hour span, as you mentioned (| timechart span=2h ...). This means each bar represents a 2-hour bucket (e.g., 10:00–12:00, 12:00–14:00). In Dashboard Studio, confirm the visualization is set up as a Timechart (Area, Column, or Line) under the Visualization tab. Set the global_time.earliest Token: You’ve already set a token interaction for global_time.earliest, but let’s confirm it’s correct. In Dashboard Studio’s UI Editor: Select your timechart visualization. Go to the Interactions tab in the configuration panel. Under On Click, add a Set Token action: Token Name: global_time.earliest Token Value: $result._time$ (this captures the start time of the clicked bar, e.g., 10:00 for a 10:00–12:00 bar). This sets global_time.earliest to the timestamp of the clicked bar’s start. Calculate global_time.latest Token: Instead of a second click, compute global_time.latest as global_time.earliest + 2 hours using a token formula. In the UI Editor: Go to the same On Click interaction for the timechart. Add a second Set Token action (below the global_time.earliest one): Token Name: global_time.latest Token Value: relative_time($global_time.earliest$, "+2h") This uses Splunk’s relative_time function to add 2 hours to the earliest timestamp (e.g., if earliest is 10:00, latest becomes 12:00). Both tokens will now be set on a single click, defining the exact 2-hour range of the clicked bar. Apply Tokens to Other Visualizations: Ensure other visualizations in your dashboard use the global_time.earliest and global_time.latest tokens to filter their time ranges. For each visualization (e.g., table, chart): Go to the Search tab in the configuration panel. Set the Time Range to Custom and use: Earliest: $global_time.earliest$ Latest: $global_time.latest$ Alternatively, modify the search query directly to include the token-based time range, e.g.: spl   index=your_index earliest=$global_time.earliest$ latest=$global_time.latest$ | ... Add a Default Time Range (Optional): To prevent visualizations from breaking before a timechart click, set default values for the tokens. In the UI Editor: Go to the Dashboard configuration (top-level settings). Under Tokens, add: Token Name: global_time.earliest, Default Value: -24h@h (e.g., last 24 hours, snapped to hour). Token Name: global_time.latest, Default Value: now (current time). This ensures other visualizations display data until the timechart is clicked. Test the Dashboard: Save and preview the dashboard. Click a timechart bar (e.g., representing 10:00–12:00). Verify that: global_time.earliest is set to the bar’s start (e.g., 10:00). global_time.latest is set to the bar’s end (e.g., 12:00). Other visualizations update to show data only for that 2-hour range. Use the Inspect tool (click the three dots on a visualization > Inspect > Tokens) to debug token values if needed. Why This Works Single Click: Using relative_time($global_time.earliest$, "+2h") avoids the need for a second click, as it calculates the end of the 2-hour bar based on the clicked time. Mimics Search App: The Search app’s timeline sets both earliest and latest times for a selected range. This solution replicates that by defining the full 2-hour bucket. Dashboard Studio Limitation: Dashboard Studio doesn’t natively support range selection (like dragging over a timeline), so computing latest via a formula is the best approach. Troubleshooting Tips Tokens Not Setting: If global_time.latest isn’t updating, check the token syntax in the Source view (JSON). Ensure the relative_time function is correct: "value": "relative_time($global_time.earliest$, \"+2h\")". Time Format Issues: Ensure $result._time$ returns a timestamp in epoch format (seconds). If not, use strptime in the timechart search to format it, e.g., | eval _time=strptime(_time, "%Y-%m-%d %H:%M:%S"). Visualization Not Updating: Confirm other visualizations reference $global_time.earliest$ and $global_time.latest$ correctly. Check their search queries in the Source view. Span Mismatch: If the timechart span changes (e.g., dynamically set), you may need to make the +2h offset dynamic. Let us know if your span varies for a custom solution. Example JSON Snippet (Source View) For reference, here’s how the timechart’s interaction might look in the dashboard’s JSON (edit in Source view if needed): json   { "visualizations": { "viz_timechart": { "type": "splunk.timechart", "options": { ... }, "dataSources": { "primary": "ds_timechart" }, "eventHandlers": [ { "type": "drilldown.setToken", "options": { "token": "global_time.earliest", "value": "$result._time$" } }, { "type": "drilldown.setToken", "options": { "token": "global_time.latest", "value": "relative_time($global_time.earliest$, \"+2h\")" } } ] } } } ... View more

You are going to have to contact Splunk Support for any older versions not on their website.  I apologize for that inconvenience.   It is your environment and you need to do what you and your management team feel are the best things, but as a person employed in the Cyber Security arena, I feel that I should at least mention the following.  None of this applies to your wanting to run 9.0.x  It was the Splunk 7 and Splunk 8 that raised my antennae.   Running 7.x (and to a lesser extent 8.x) UFs introduces significant risks, especially since Splunk 7.x reached End of Support (EOS) between October 2020 and October 2021, and 8.2.x is also at or past end of life. Here are the key implications:   Operational Risks: Limited Functionality: Splunk 7.x UFs lack support for newer features like data compression, advanced SSL configurations, or Splunk-to-Splunk (S2S) Protocol V4, which 9.x indexers use by default. This can cause performance issues or data ingestion failures if configurations mismatch. For example, 7.x UFs may not handle modern event-breaking or parsing rules in 9.0.x apps.   Management Challenges: If you use a Deployment Server (DS), it must be 9.0.x or newer to manage 7.x/8.x UFs. Older DS versions may fail to deploy apps to newer UFs, complicating configuration management. Stability Issues: 7.x UFs may encounter bugs or crashes on modern OSes (e.g., newer Linux kernels), as they were designed for older environments. Splunk Support won’t provide fixes for EOS versions, leaving you to work around issues manually. Security Risks: Vulnerabilities: 7.x UFs miss critical security patches available in 8.x and 9.x, exposing your environment to known vulnerabilities (e.g., CVE fixes). Without patches, UFs could be exploited, especially if they’re on internet-facing systems or handle sensitive data.   SSL/TLS Weaknesses: 7.x UFs use outdated SSL/TLS protocols, which may conflict with 9.0.x’s stricter security defaults (e.g., TLS 1.2/1.3). This can lead to connection failures or insecure data transmission. 8.x UFs are less problematic but still lack the latest TLS enhancements in 9.x. Compliance Issues: Running EOS software like 7.x may violate compliance requirements (e.g., PCI DSS, HIPAA), as auditors often flag unsupported software as non-compliant. Recommendations for UFs: Upgrade UFs to 9.0.x: Plan to upgrade your 7.x and 8.x UFs to 9.0.x (or at least 8.2.x) to align with your indexer. UFs are lightweight, and upgrades are straightforward  Start with a few test UFs to validate compatibility with your 9.0.x indexer and DS (if used). Use the Deployment Server to automate UF upgrades, ensuring serverclass.conf matches the new version. Prioritize 7.x First: 7.x UFs are the most critical to upgrade due to EOS status and severe security risks. 8.x UFs are less urgent but should be updated to avoid future EOS issues. Check Compatibility: Confirm UF OS compatibility with 9.0.x (e.g., AL2023 or supported Windows versions) using the Splunk System Requirements.   Interim Step: If upgrading all UFs immediately isn’t feasible, ensure your 9.0.x indexer’s inputs.conf supports legacy S2S protocols (e.g., V3 for 7.x UFs) by setting connectionTimeout or readTimeout to accommodate older clients. However, this is a temporary workaround. Why Upgrade UFs?: Aligning UFs with 9.0.x ensures optimal performance, security, and supportability. Splunk 9.0.x introduces features like ingest actions and enhanced TLS validation, which 7.x UFs can’t leverage.   Upgrading avoids the risk of data loss or ingestion delays due to protocol mismatches or unpatched bugs. Splunk Support can assist with 9.0.x issues, but not with 7.x, reducing your troubleshooting burden. ... View more

The two previous posts both are good answers, but since you stated you are new to Splunk I decided to give you a thorough write up that explains how to check each of these areas that have been called out to see if they are the problem that is causing your error.   The warning you’re seeing (The TCP output processor has paused the data flow) means your forwarder (at MRNOOXX) is unable to send data to the receiving Splunk instance (at 192.XXX.X.XX), likely because the receiver is not accepting data or the connection is blocked. This can stall data indexing, so let’s troubleshoot it step-by-step. Here’s a comprehensive checklist to resolve the issue: Verify Receiver is Running: Ensure the Splunk instance at 192.XXX.X.XX (likely an indexer) is active. On the receiver, run $SPLUNK_HOME/bin/splunk status to confirm splunkd is running. If it’s stopped, restart it with $SPLUNK_HOME/bin/splunk restart. Confirm Receiving Port is Open: The default port for Splunk-to-Splunk forwarding is 9997. On the receiver, check if port 9997 is listening: netstat -an | grep 9997 (Linux) or netstat -an | findstr 9997 (Windows). Verify the receiver’s inputs.conf has a [splunktcp://9997] stanza. Run $SPLUNK_HOME/bin/splunk cmd btool inputs list splunktcp --debug to check. Ensure disabled = 0. Test Network Connectivity: From the forwarder, test connectivity to the receiver’s port 9997: nc -vz -w1 192.XXX.X.XX 9997 (Linux) or telnet 192.XXX.X.XX 9997 (Windows). If it fails, check for firewalls or network issues. Confirm no firewalls are blocking port 9997 on the receiver or network path. Check Forwarder Configuration: On the forwarder, verify outputs.conf points to the correct receiver IP and port. Check $SPLUNK_HOME/etc/system/local/outputs.conf or app-specific configs (e.g., $SPLUNK_HOME/etc/apps/<app>/local/outputs.conf). Example: ini   [tcpout:default-autolb-group] server = 192.XXX.X.XX:9997 disabled = 0 Ensure no conflicting outputs.conf files exist (run $SPLUNK_HOME/bin/splunk cmd btool outputs list --debug). Inspect Receiver Health: The error suggests the indexer may be overwhelmed, causing backpressure. Use the Splunk Monitoring Console (on a Search Head or standalone instance) to check: Go to Monitoring Console > Indexing > Queue Throughput to see if queues (e.g., parsing, indexing) are full (100% fill ratio). Check Resource Usage > Machine for CPU, memory, and disk I/O (IOPS) on the indexer. High usage may indicate bottlenecks. Run this search on the Search Head to check queue status: | rest /services/server/introspection/queues splunk_server=192.XXX.X.XX | table title, current_size, max_size, fill_percentage. Ensure the indexer has sufficient disk space (df -h on Linux or dir on Windows) and isn’t exceeding license limits (check Monitoring Console > Licensing). Check for SSL Mismatches: If SSL is enabled (e.g., useSSL = true in outputs.conf on the forwarder), ensure the receiver’s inputs.conf has ssl = true. Verify certificates match in $SPLUNK_HOME/etc/auth/ on both systems. Check splunkd.log on the receiver for SSL errors: grep -i ssl $SPLUNK_HOME/var/log/splunk/splunkd.log. Review Logs for Clues: On the forwarder, check $SPLUNK_HOME/var/log/splunk/splunkd.log for errors around the TCP warning (search for “TcpOutputProc” or “blocked”). Look for queue or connection errors. On the receiver, search splunkd.log for errors about queue fullness, indexing delays, or connection refusals (e.g., grep -i "192.XXX.X.XX" $SPLUNK_HOME/var/log/splunk/splunkd.log). Share any relevant errors to help narrow it down. Proactive Mitigation: If the issue is intermittent (e.g., due to temporary indexer overload), consider enabling persistent queues on the forwarder to buffer data during blockages. In outputs.conf: ini   [tcpout] maxQueueSize = 100MB usePersistentQueue = true Restart the forwarder after changes. Architecture and Version Details: Could you share: Your Splunk version (e.g., 9.3.1)? Run $SPLUNK_HOME/bin/splunk version. Your setup (e.g., Universal Forwarder to single indexer, or Heavy Forwarder to indexer cluster)? Is the receiver a standalone indexer, Splunk Cloud, or part of a cluster? This will help tailor the solution, as queue behaviors vary by version and architecture. Quick Fixes to Try: Restart both the forwarder and receiver to clear temporary issues: $SPLUNK_HOME/bin/splunk restart. Simplify outputs.conf on the forwarder to point to one indexer (e.g., server = 192.XXX.X.XX:9997) and test. Check indexer disk space and license usage immediately, as these are common culprits. Next Steps: Share the output of the network test (nc or telnet), any splunkd.log errors, and your architecture details. If you have access to the Monitoring Console, let us know the queue fill percentages or resource usage metrics.   ... View more

Mine did the same thing.  It would seem it is your version of Splunk.  Set up a test environment on a laptop or spare VM and run a newer version of Splunk and see if the problem remediates itself.   ... View more

It sounds like your Heavy Forwarder (HF) is struggling to handle the high volume of Akamai logs (30k events in 5 minutes), which may be causing the GUI to become slow or unresponsive. The error in splunkd.log about the TA-Akamai-SIEM modular input timing out (exceeding 30,000 ms) suggests the modular input script is overloaded. Since data ingestion continues and splunkd is running, the issue is likely related to resource contention or configuration. Here’s how you can troubleshoot and resolve it: Check HF Resource Usage: Monitor CPU, memory, and disk I/O on the HF using top or htop (Linux) or Task Manager (Windows). High resource usage could indicate the HF is overwhelmed by the Akamai log volume. Use the Splunk Monitoring Console (| mcatalog) or | rest /services/server/info to check system metrics like CPU usage or memory consumption on the HF. Tune Modular Input Timeout: The TA-Akamai-SIEM modular input is timing out after 30 seconds (30,000 ms). Increase the timeout in $SPLUNK_HOME/etc/apps/TA-Akamai-SIEM/local/inputs.conf: ini   [TA-Akamai-SIEM://<input_name>] interval = <your_interval> execution_timeout = 60000 # Increase to 60 seconds Restart the HF after making this change ($SPLUNK_HOME/bin/splunk restart). Optimize TA-Akamai-SIEM Configuration: Check the interval setting for the Akamai input in inputs.conf. A very short interval (e.g., 60 seconds) with high data volume (30k events/5 min) could overload the modular input. Consider increasing the interval (e.g., to 300 seconds) to reduce the frequency of API calls. Verify the API query filters in the TA configuration. Narrow the scope (e.g., specific Akamai configurations or event types) to reduce the data volume if possible. Address GUI Unresponsiveness: The GUI slowdown may be due to splunkd prioritizing data ingestion over web requests. Check $SPLUNK_HOME/etc/system/local/web.conf for max_threads or http_port settings. Increase max_threads if it’s too low: ini   [settings] max_threads = 20 # Default is 10; adjust cautiously Confirm the HF’s web port (default 8000) is accessible via telnet <HF_IP> 8000 from your machine. Inspect splunkd.log Further: Look for additional errors in $SPLUNK_HOME/var/log/splunk/splunkd.log related to TA-Akamai-SIEM or resource exhaustion (e.g., memory or thread limits). Check for errors in $SPLUNK_HOME/var/log/splunk/web_service.log for GUI-specific issues. Scale or Offload Processing: If the HF is underpowered, consider upgrading its hardware (more CPU cores or RAM) to handle the 30k events/5 min load. Alternatively, distribute the load by deploying multiple HFs and splitting the Akamai inputs across them, forwarding to the same indexers. Ensure the TA-Akamai-SIEM add-on is only installed on the HF (not the Search Head or indexers) to avoid unnecessary processing. Engage Splunk Support: Since Support reviewed the diag file, ask them to specifically analyze the TA-Akamai-SIEM modular input logs and any resource-related errors in splunkd.log. Share the timeout error and data volume details. ... View more

This is exactly the way to solve this problem.  Honestly, as you start to really master splunk you will find that Stats seems to be the answer for everything.   this is a very helpful presentation on your very problem.   let-stats-sort-them-out-building-complex-result-sets-that-use-multiple-source-types.pdf slide 33 ... View more

Karma to both answers above.  Don't let ES run on the Cluster Master and It sounds like you have beefy servers, but ES can bring the beefiest of servers to its knees if you are not careful.  The Splunk ES Content Pack has close to 6000 (I could be underestimating the number it could be higher) correlation / finding searches.  If someone goes in and turns on all of those searches and you have your own stuff running, Splunk will absolutely choke and die.   As a best practice I track each and every search that is running on my ES instances and map the time windows that those searches run.  Any new searches activated are set to run in the time windows that have the least amount of searches running.  Just remember that the general rule of thumb is that each search that runs, occupies one cpu core and one gig of ram while running.   Additionally, if you have the resources, you can open the pandora's box of multithreading and / or allowing more concurrent searches - but do this ONLY as a last resort and you should validate by running top, or checking the Management console or whatever tool you use to validate that you have spare cpu and ram to allow more concurrent searches.   ... View more

@livehybrid recommended using a lookup to track your Forwarders.  I have to say that this is a really valuable tool, because if you keep track of your forwarders using a lookup, you can see what systems have not reported easily but you can also see any new forwarders that are sending logs to your system that you didn't know about.   Below is a youtube video tutorial on using the lookup to track systems no longer sending logs.   https://youtu.be/lo4_MIfTJzI?si=WfHxtBzTHLxmhQpe All of the posts are good ideas.  The lookup is just one way to do it that is quick and easy, but there are many ways to do things in splunk, this is just my favorite way.   ... View more

It looks like your mcollect command isn’t writing data to the metrics_new index, despite mpreview returning results. The "No results to summary index" error and the info.csv limit warning suggest a couple of potential issues. Here’s how you can troubleshoot and fix it: Verify mcollect Syntax: Ensure the fields required by mcollect are present. For metrics data, mcollect expects _value (numeric metric value), metric_name, and any dimensions (e.g., env, server_name). Your mpreview query should already include these, but confirm the output includes _value and metric_name:process.java.gc.collections. Try adding | fields metric_name, _value, env before mcollect to explicitly pass only the required fields: spl   | mpreview index=metrics_old target_per_timeseries=5 filter="metric_name IN (process.java.gc.collections) env IN (server_name:port)"| fields metric_name, _value, env | mcollect index=metrics_new Check Index Configuration: Confirm metrics_new is a metrics index (not an event index) on both the search head and indexers. Run | eventcount summarize=false index=metrics_new to verify the index exists and is accessible. Ensure the index is not full or disabled. Check $SPLUNK_HOME/etc/apps/<app>/local/indexes.conf on the indexers for metrics_new settings and verify frozenTimePeriodInSecs or maxTotalDataSizeMB aren’t causing issues. Address info.csv Limit Warning: The warning about info.csv reaching its limit (65,149 messages) suggests the search is generating excessive log messages, which might indicate a problem with the mpreview output or internal processing. This can sometimes prevent mcollect from writing results. Increase the info.csv limit in $SPLUNK_HOME/etc/system/local/limits.conf: ini   [search] max_messages_per_info_csv = 100000 Alternatively, reduce the volume of data processed by narrowing the time range or tightening the filter in mpreview (e.g., specify a single env value). Inspect Search Logs: The job inspector mentions errors in search.log. Check $SPLUNK_HOME/var/log/splunk/search.log for details on why mcollect is failing. Look for errors related to index access, field mismatches, or data format issues. Common issues include missing _value fields or incompatible data types for mcollect. Test with a Simpler Query: To isolate the issue, try a minimal mcollect command: spl   | mpreview index=metrics_old target_per_timeseries=1 filter="metric_name=process.java.gc.collections"| mcollect index=metrics_new If this works, gradually add back the env filter to identify where the issue arises. Next Steps: Run the modified query with | fields and check if data writes to metrics_new using | mpreview index=metrics_new. Share any specific errors from search.log or confirm if the fields (metric_name, _value, env) are present in the mpreview output for further assistance. ... View more

Let me be completely transparent on this answer.  I do not know anything about what I am about to respond.  I put your question into Grok and am giving back what it says.  So if it is way off, I apologize.  It sounds like you are using Splunk Observability.  If you are just trying to pull metrics logs of the OS of a system, this is much easier and I would just use the Splunk Linux TA as a guide for the scripts to pull that off a Linux box and the Windows TA for windows, but If my gut is right, that is not your problem and it is Splunk observability cloud is what you are actually looking at.   So here is the cut and paste from Grok. To fetch actual metric values (time-series data) in Splunk Observability Cloud using REST APIs, you can use the ** /v2/datapoint** endpoint, which retrieves data points for specified metrics. Unlike the Metrics Catalog endpoints (e.g., /v2/metric), which return metadata like metric names and dimensions, the /v2/datapoint endpoint provides the numerical values for metrics over a specified time range. Here’s how you can approach it: Endpoint: Use GET /v2/datapoint or POST /v2/datapoint to query metric values. The POST method is useful for complex queries with multiple metrics or filters. Authentication: Include an access token in the header (X-SF-TOKEN: <YOUR_ORG_TOKEN>). You can find your org token in the Splunk Observability Cloud UI under Settings > Access Tokens. Query Parameters: Specify the metric name(s) you want to query (e.g., cpu.utilization). Use dimensions to filter the data (e.g., host:server1). Define the time range with startTs and endTs (Unix timestamps in milliseconds) or a relative time range (e.g., -1h for the last hour). Set the resolution (e.g., 10s for 10-second intervals). Example Request (using curl): bash curl --request POST \   --header "Content-Type: application/json" \   --header "X-SF-TOKEN: <YOUR_ORG_TOKEN>" \   --data '{     "metrics": [       {         "name": "cpu.utilization",         "dimensions": {"host": "server1"}       }     ],     "startTs": 1697059200000,     "endTs": 1697062800000,     "resolution": "10s"   }' \   https://api.<REALM>.signalfx.com/v2/datapoint Replace <YOUR_ORG_TOKEN> with your access token and <REALM> with your Splunk Observability realm (e.g., us0, found in your profile). Response: The API returns a JSON object with time-series data points, including timestamps and values for the specified metric(s). For example: json {   "cpu.utilization": [     {"timestamp": 1697059200000, "value": 45.2, "dimensions": {"host": "server1"}},     {"timestamp": 1697059210000, "value": 47.8, "dimensions": {"host": "server1"}}   ] } Tips: Use the Metric Finder in the Splunk Observability Cloud UI to confirm metric names and dimensions. If you’re using OpenTelemetry, ensure your Collector is configured to send metrics to Splunk Observability Cloud. For detailed documentation, check the Splunk Observability Cloud developer portal: https://dev.splunk.com/observability/docs/datapoint_endpoint/.[](https://help.splunk.com/en/splunk-observability-cloud/manage-data/other-data-ingestion-methods/other-data-ingestion-methods/send-data-using-rest-apis) If you’re still getting metadata, verify you’re not using /v2/metric or /v2/metricstore/metrics endpoints, which are for metadata only. ... View more

We recently had the exact same issue on our environment.  It was all related to expiring certs.  It sounds like you have already looked into this, but I just want to echo that when the cert expires or can't be found, you can get this very behavior and it took a lot longer than I want to admit for our team to figure out that this was the reason all of our KV stores were failing.   ... View more