I have 2 different splunk apps, one is a TA and the other is an app.  TA : uses modular input to connect with a data source. There are some logs and metadata that are pulled from the data source. Logs are pulled via syslog by providing a tcp input and metadata via api key and secret. The metadata is stored in kv stores.  App: is supposed to be installed on search heads and they support dashboards/reports that make use of the logs and metadata sent by HF. For splunk enterprise, the above approach works when HF has the context of search heads, because HF takes care of uploading the kv stores to the search heads via scheduled search. This ensures that the app residing on SH has the data to work with. However, on splunk cloud, once TA is installed , how to ensure that SH nodes have metadata to work with? Can we find out what are the search head fqdns so that kv stores can be copied there via scheduled search? ... View more

My requirement is to get the rate of change of a certain parameter if its corresponding alert gets triggered. To add more details, we have a log file that logs the backlog of database. Once the backlog crosses a certain threshold we trigger an alert, however it could be a false positive since the system may be undergoing maintenance and hence backlog grows. So I want the alert to trigger another query that captures rate of growth over the last 'x' hours. This will give more context about what is happening in the system. How to achieve this in splunk? Please share your ideas ... View more