>Unless you hit some generic Splunk limits. What generic limits comes to mind? Some limit on the download activity itself is the first thing I think of, although I don't know offhand where that is reflected in the config. ... View more

ah yeah. Seems like it might be appropriate in some settings. Are there config files where it's particularly egregious? ... View more

with some help from @mmccul in slack I was able to figure this out. btool is expecting a specific directory structure, and is looking for the spec files to be found in `system/README` from what is specified as the `--dir` target. It also only looks in directories named like the normal config directories ( apps, master-apps etc) ... View more

hmm it appears that check doesn't work when also using the --dir flag ... View more

oooo tiered DS do you do any checking or linting on the PRs? ... View more

Concerning the subtleties of _index_time in search, making sure you find the events you need, great conf talk on this  https://conf.splunk.com/watch/conf-online.html?search=PLA1327B&search.event=conf21 slides directly https://conf.splunk.com/files/2021/slides/PLA1327B.pdf >With minimal up-front effort it is possible to guarantee that your alerts and other scheduled searches run, are always successful, and do not miss data. Common challenges are skipped searches, latent data, Splunk down time, failures, and dependencies on other searches. Approaches such as an expanded sliding window consume additional resources and will inevitably fail. We will demonstrate a Splunk macro that tracks search execution times in a KVstore and dynamically controls the search timeframe, thus decoupling it from execution time. This additionally provides a capability to quickly and easily re-run a search over any timeframe in a controllable manner. We will further demonstrate the use of Apache Airflow for more complex use cases. ... View more

https://batchworks.de/recover-deleted-data/ for recovering deleted data ... View more

Timechart + single-value visualization from Rich   ... View more

Best thing to do for usage of the delta command is find blog posts such as https://www.splunk.com/en_us/blog/tips-and-tricks/search-commands-delta.html I'd also suggest searching GitHub for "| delta" to find SPL examples for people that added searches to their repos. https://dev.splunk.com/ is a good starting place for custom search commands, and alert actions. For finding existing third party integrations via alert actions, checking splunkbase is always a good idea. https://splunkbase.splunk.com/app/4621/ The TrackMe is something I've found useful for watchdog type alerting on the absence of certain types of data. `!missinghosts` in slack drops a links to all sorts of apps and thoughts on this ... View more

I'm having difficulty understanding this 😅. Could you explain what you mean by source? Perhaps you could drop in literal event samples? I'm not following the example presented. ... View more

This is the complaint that seems to be the most damning. Taking this to the letter, it seems that realtime searches are capped at the number of cores on the search heads or the indexers, whichever is lower, and reaching anything close to that cap will severely impact the environment. Has there ever been recognition of this by splunk, and any attempts to develop around it? ... View more

if the splunk instance is 6.7+, there isn't a need for the channel parameter in the POST ... View more

so, you can use lookups to exclude events based on those events having fields that match some value in the lookup, i.e. basesearch-that-has-port | lookup exclusionList port OUTPUTNEW port as isFound | where isnull(isfound) You could do additionally lookups for other fields if you are looking at IP as well basesearch-that-has-port | lookup exclusionList port OUTPUTNEW port as portIsFound | lookup exclusionList ip OUTPUTNEW port as ipIsFound | where isnull(portIsfound) AND isnull(ipIsFound) Please let me know if this answers your question! ... View more

Hi sathish2k8, Yup, perfectly possible to create a form input for a dashboard and pass the values to a search. There is more info here: http://docs.splunk.com/Documentation/Splunk/7.1.0/Viz/tokens You'll likely have to adjust the format of the result time token through additional eval commands as demonstrated here: https://answers.splunk.com/answers/438999/dashboard-how-can-i-convert-a-token-from-a-time-pi.html Once the token is in the correct format needed for the dbquery, then you'll run that final search Please let me know if this helps! ... View more

Searching the internal index for messages that mention "block" might turn up some events. These pages have some more info: https://docs.splunk.com/Documentation/Splunk/7.1.0/Troubleshooting/Aboutmetricslog If you have the monitoring console setup, and have access to it: http://docs.splunk.com/Documentation/Splunk/7.1.0/Troubleshooting/Troubleshootindexingperformance Also, you'll want to search for messages like this just to be sure: "Error in 'IndexScopedSearch': The search failed. More than 125000 events found at time 1293916026." https://answers.splunk.com/answers/10299/the-search-failed-more-than-125000-events-found-at-time.html ... View more

Hi riqbal, I think part of the issue might be related to some additional config you aren't aware of. Try running btool to get an view of all the inputs config. A usage example is: /opt/splunk/bin/splunk btool --debug inputs list This will give you a rolled up view of all inputs config. Additionally, you can look at inputs config from Splunk with this app I made for this purpose : https://splunkbase.splunk.com/app/3923/ Although it seems less likely, there could also be some props.conf config causing issues (rewriting the index config), but I think doing a thorough examination of the inputs config at each step will be the most helpful thing to do. Please let me know if this helps! ... View more

Hi john, are you able to narrow down the metrics search to only the checkpoint logs? That forwarder might not be doing much else, but it looks like the metrics search is counting all events the forwarder is processing. Otherwise, do you see any blocking or queue-fill issues on the indexer? If it can't keep up it'll start dropping events, which could potentially explain this. Finally, do you get any warnings or messages when you run the tstats search? One other issue here could be with splunk having too many events on the same subsecond timestamp. I think it'll usually warn about this if it happens though. Please let me know if this helps! ... View more

Hi Sarag0511, I believe the issue is that, along with the input for the forwarder, you'll have to create an output that sends the events to the indexer. The indexer will need to have a network input, and also the have the same index names specified in the input config on the forwarder. This page has some commands that can be run on the forwarder to setup the config : http://docs.splunk.com/Documentation/Forwarder/7.1.0/Forwarder/Configuretheuniversalforwarder#Configure_the_universal_forwarder_to_connect_to_a_receiving_indexer Please let me know if this helps! ... View more

Hi szabados, Your config seems to be reasonable, and so based on your comments I'd suggest trying to reinitialize this particular kvstore. Alternatively, you could try an intermediary csv as a troubleshooting step/workaround. That is, first output the generating search to a csv file, and then use inputlookup to read that csv, and then output to kvstore. There shouldn't be any issue with a generating search taking a long time, and therefore not correctly writing results to a kvstore, but inputlookup is fast, and so we can hopefully rule that out. Also, if this is still an issue, can we see what the generating search looks like? Please let me know if this answers your question! 😄 ... View more