Hi everyone,  I've revently tested the new Splunk AI feature within Splunk ITSI to define thresholds based on historic Data/KPI points. ("Test" as in I literally created very obvious dummy-data for the AI to process and find thresholds for. Sort of Trust test of the AI really does find usuable thresholds. ) Example:  Every 5 minutes the KPI takes the latest value which I've set to correspond with the current weekday (+ minimal variance) For example: All KPI values on Mondays are within the range of 100-110, Tuesdays 200-210, Wednesdays 300-310 and so forth.  This is a preview of the data:  Now after a successful backfill of 30 days I would have expected the AI to see that each weekday needs its own time policy and thresholds.  However the result was this:  No weekdays detected, and instead it finds time policies for every 4hours regardless of days?  By now I've tried all possible adjustments I could think of (increasing the number of data points, greater differences between data points, other algorithmn, waiting for the next in hopes it would recalibrate itself over midnight, etc.) Hardly any improments at all and the thresholds are not usuable like this as it would not be able to detect outliers on mondays (expected values 100-110, outlier would 400 but not detected as it's still within thresholds. Thus my question to the community: Does anyone have some ideas/suggestions how I could make the AI understand the simple idea of "weekly time policies" and how I could tweak it? (Aside from doing everything manually and ditching the AI-Idea as a whole)?  Does anyone have good experience with Splunk AI defining Thresholds and if so what were the use cases? ... View more

Hello all, I've run into a problem with the backfill upon creating (also tried cloning) a KPI in regards to Splunk License Metrics using the following search:   index=_internal source=*license_usage.log type="Usage" | fields idx, b | eval indexname = if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) | bin span=5min _time | stats sum(b) as b by indexname, _time | eval GB=round(b/1024/1024/1024, 3) | fields _time, indexname, GB The Use Case:  I want a KPI for the License Usage with the separate Indexes as Entities.  Configuration info:   Seeing as I want the License Info on an per Index-Basis I konfigured the KPI to be split into Entities by the field "indexname". As for the Frequency and Calculation I selected:   Calculating Maximum of GB per entity as entity value,  Sum of entity value as aggregate over the last 5 minute(s) every 5 minute(s). Fill gaps in data with Null values and use a unknown threshold level for them. So far so good... now I also configured a Backfill for the last 30 days (taxing on the system but it should manage). The Problem: Upon seeing the Message that the backfill was completed, I checked the itsi_summary Index and found the backfill data of the KPI  but with regular gaps. More precisely, for each day it had backfilled the data from the activation time of the kpi (here 12:30) for about 6h (18:25/18:30) and then there were no further values for the day until the next day around 12:30. Even though there is license usage during the gap times and also available in the license_usage.log used by the KPI search.  The Data since activation is continuous and has no gaps.  I tried cloning the KPI, remaking the KPI with both adhoc or base search, but all featured the same curious results (just with different starting points as the activation time of the KPI was different). Thus now I am wondering if there is some sort of limit for backfilling or if perhaps someone has an idea what caused this strange backfill behaviour? (Also there was no error message in the _internal index as far as I could tell.)  Help and ideas would be appreciated. Thanks in advance.  ... View more

Hey everyone,  I currently have a use case for which I set up a Splunk Enterprise environment in an Ubuntu VM (VMware) and want to build an app with the Add-on Builder, which uses a Python Script as Input Method to make an API-Call to get my data into Splunk. That's the goal at least.   The VM communicates with the Internet just fine (even if via proxy) and my python script gets the data from the API-Endpoint. However, if I try to enter the proxy credentials from my VM into the Configuration of the Add-on Builder I get the following Error: "There was a problem connecting to the App Certification service. The service might not be available at this time, or you might need to verify your proxy settings and try again."  Now, assuming that I did not mess up the proxy credentials, my next best bet would be that I need to give my Splunk environment a certificate to adequately communicate with the proxy. So we finally reach my question:  Where would I need to place such a certificate file in the directory structure, so that the the Splunk add-on app can find it?   ... View more