Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
×
Are you a member of the Splunk Community?
Sign in or Register with your Splunk account to get your questions answered, access valuable resources and connect with experts!
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
- Find Answers
- :
- About EricLloyd79
EricLloyd79
Builder
Member since:
11-15-2013
06-05-2020
Community Statistics
| Posts | 192 |
| Solutions | 11 |
| Karma Given | 12 |
| Karma Received | 12 |
| Member Since | 11-15-2013 |
Activity Feed
- Karma Re: In Splunk IT Service Intelligence, why can I not see a preview of data in my Aggregate Threshold Views window? for skoelpin. 06-05-2020 12:50 AM
- Got Karma for Re: How do I handle customized time ranges in Splunk IT Service Intelligence (ITSI)?. 06-05-2020 12:50 AM
- Got Karma for Why is backfill not working in Splunk IT Service Intelligence?. 06-05-2020 12:50 AM
- Got Karma for In Splunk IT Service Intelligence, why can I not see a preview of data in my Aggregate Threshold Views window?. 06-05-2020 12:50 AM
- Got Karma for Re: What is the difference in ITSI thresholds between Preview Aggregate Threshold values compared to the configure thresholds values?. 06-05-2020 12:50 AM
- Got Karma for Re: What is the difference in ITSI thresholds between Preview Aggregate Threshold values compared to the configure thresholds values?. 06-05-2020 12:50 AM
- Got Karma for Can I get a list of reports from a splunk query?. 06-05-2020 12:50 AM
- Got Karma for Can I create a KPI in ITSI that is a sum of two other KPIS?. 06-05-2020 12:50 AM
- Got Karma for Why is my Splunk IT Service Intelligence (ITSI) service health score not being reflected correctly?. 06-05-2020 12:50 AM
- Got Karma for Re: Why is my Splunk IT Service Intelligence (ITSI) service health score not being reflected correctly?. 06-05-2020 12:50 AM
- Got Karma for Re: Splunk refuses to ingest particular variable. 06-05-2020 12:49 AM
- Got Karma for Why am I getting this error: splunklib.binding.HTTPError: HTTP 404 Not Found -- Unknown sid when using the Python SDK. 06-05-2020 12:49 AM
- Karma Re: Complex subsearch comparing time periods for micahkemp. 06-05-2020 12:48 AM
- Karma Re: Returning the bytes of each log for aweitzman. 06-05-2020 12:47 AM
- Karma Re: Creating Pivot Chart with two sums? for okrabbe. 06-05-2020 12:47 AM
- Karma Re: How to create a timechart using a root search with data models and pivots? for lguinn2. 06-05-2020 12:47 AM
- Karma Re: How to add the latest field value from two hosts? for lguinn2. 06-05-2020 12:47 AM
- Got Karma for How to create a timechart using a root search with data models and pivots?. 06-05-2020 12:47 AM
- Karma Re: snap to 5 minute increments in timerange for charleswheelus. 06-05-2020 12:46 AM
- Karma Re: Extract string and separate results by different strings for martin_mueller. 06-05-2020 12:46 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 1 | |||
| 1 | |||
| 1 | |||
| 0 |
10-24-2018
01:06 PM
Ah, using itsi_summary, I was able to find the alert_value and it is matching the values that I see in the Deep Dive results. This means the query is working but the preview I see in the Threshold preview is way off for some reason. The values for the alert_value are between 11-13 and the values in the Threshold preview are all 22265 for some reason and I have values for yesterday so I would expect them to be reflected here. Anything else in the itsi_summary you can think of that I could figure out why this is happening? (note: I have other kpis I created a week or so ago which are also using eventstats and eval functions that do show an accurate threshold preview map)
Could I just need to wait longer to see it in the itsi threshold preview map?
... View more
10-24-2018
12:42 PM
I will admit I struggle to understand how itsi_summary helps me to recreate the query to verify the results.
index=itsi_summary kpi="SCP CPU Utilization"
That returns an event such as:
10/24/2018 19:32:57 +0000, search_name="Indicator - Shared - 5bcf77972b9d44157e79157f - ITSI Search", search_now=1540409580.000, info_min_time=1540409277.000, info_max_time=1540409577.000, info_search_time=1540409582.235, qf="", kpi="SCP CPU Utilization", kpiid=80e03fe65ca6fb18fccd8fc4, urgency=5, serviceid="1e9057dc-4f5d-4abf-a773-e85349dd8a84", itsi_service_id="1e9057dc-4f5d-4abf-a773-e85349dd8a84", is_service_aggregate=1, is_entity_in_maintenance=0, is_entity_defined=0, entity_key=service_aggregate, is_service_in_maintenance=0, kpibasesearch=5bcf77972b9d44157e79157f, alert_color="#99D18B", alert_level=2, alert_value=11, itsi_kpi_id=80e03fe65ca6fb18fccd8fc4, is_service_max_severity_event=1, alert_severity=normal, alert_period=1, entity_title=service_aggregate
After digging through that I am confused on what I can extract from it to help me understand why Im not able to create thresholds for this query. Which of these fields is the value for that kpi at that time?
... View more
10-24-2018
12:34 PM
Also you asked if I used a transformation in my search. This was my search so I think yes: "sourcetype=cpu SCP1_CPU | stats sum(SCP1_CPU) as sum_scp1 sum(SCP2_CPU) as sum_scp2 sum(SCP3_CPU) as sum_scp3 sum(SCP4_CPU) as sum_scp4 | eval avg_scp_cpu = (sum_scp1 + sum_scp2 + sum_scp3 + sum_scp4) / 4"
So I think I am using transformations.
... View more
10-24-2018
12:25 PM
Interesting. So you have queries that include eventstats and evals and it does backfill and allows for thresholding? Im very curious to know because this will allow me to remove "not a function of ITSI" from my reasons for why this isn't working.
... View more
10-24-2018
08:30 AM
@skoelpin
I've also found that when I use the query above to create a KPI base search, the preview for the Thresholds is all out of whack and I cannot set it properly. The value range is showing in Deep Dive as between 11 - 44 and yet the values shown in the preview for Thresholds are in the millions, which makes no sense.
This is pretty discouraging as I am being asked to convert a report into an ITSI dashboard and it requires more complex queries than just summing one KPI or taking the average or another single KPI. Was ITSI just not designed for anything other than basic statistical analysis on single KPIs?
... View more
10-23-2018
12:30 PM
1 Karma
Thanks skoelpin. You can to the rescue again.
... View more
10-23-2018
11:58 AM
I do not have any dependent services.
My time range is set to 12 hours.
I have tried changing from average to max and I do see some variation in the other KPIs now that would possibly affect the health score.
I guess I misunderstand the concept behind the change between Average and Max. Arent the values displayed in the Sparklines of the KPIs the actual values specified in the metrics when the service is created? What is Max, the max of for a particular point in time for a kpi if we already specified what value we are seeking when we create the KPI?
Thanks for clarity.
... View more
10-23-2018
11:50 AM
1 Karma
If I understand correctly, a Service Health Score is an aggregation of all the KPI health scores in that service.
As you can see in the screenshot, something is off with my Service Health Score. It seems to be down despite all the other KPI health scores being in the green range.
Has anyone else experienced this before?
Note, I did create a new KPI around this time. Does that cause a dip in Service Health Score?
Thanks.
... View more
10-22-2018
01:26 PM
The purpose is to create a KPI of the sum of the two other KPIs and then display it in deep dive or possibly a glass table, yes.
I think I kind of found a work around but I guess backfill doesnt work for it.
sourcetype=sourcet1 xyz | eventstats sum(calls_kpi1) as calls1 sum(calls_kpi2) as calls2 | eval calls_proc = calls1 + calls2
... View more
10-22-2018
12:46 PM
1 Karma
I have to admit, there are certain aspects of ITSI I find limiting.
For example: I need to create KPI which the sum of two fields in the last 5 minutes, say field foo and bar.
I can specify I want the sum of either of those individually as a metic but I cannot specify that I want the sum of both of them combined into one KPI.
Is there a way to do this that anyone is aware of?
... View more
10-19-2018
12:49 PM
1 Karma
Is there a way to run a Splunk query to get a list of all reports by using a Splunk query?
... View more
10-15-2018
07:39 AM
I am having more difficulty with the conceptualization of the theory behind the different adaptive threshold algorithms. To be specific, between the Quantile and Range.
I wonder if I could present my understanding to you here and you correct me if Im wrong and offer me a bit of insight on how to differentiate the two more meaningfully.
It is stated in https://www.splunk.com/blog/2018/01/16/ensuring-success-with-itsi-threshold-and-alert-configurations-part-2-adaptive-thresholding.html
That Quantile is an algorithm that allows you to put threshold bounds at various percentiles based on historic data.
It also lists the example of choosing critical severity for data points falling below the 1st percentile (0.01) and above the 99thpercentile (0.99).
The Range is defined as looking in the min and max data points from the historic data and the span between those values. It defines an example as being a value of 0 will set a threshold to the historic data min and 1 will set it to the historic data max (and in theory, anything between those will be within the range of the min and max proportionally)
Both of them operate on the historic data. Quantile takes the percentage of the historic data values. Range uses the min and max of the historic data values. These two operations seem to be doing the same thing.
The only thing I can think of is since we can use Time Policies on specific time slots of a day for a threshold policy, we could in fact define a quantile threshold for say between 9 am – 12 pm. It would look at all the historic data for ONLY that time period and the 1.0 of that would be the max value for that time period, instead of the max value for the ENTIRE historical data set as it would be in range. But then, what is the point in defining time policies and using the range algorithm if it always uses the min and max data points for the entire data set?
... View more
10-11-2018
03:09 PM
1 Karma
@skoelpin
After some investigation, it seems that the value in the Preview Aggregate Threshold takes a kpi value sample. You can see from my search using itsi_summary that I found a value of 16 for 1:00 am on 10/9. and below that you can see that in the Preview Aggregate Threshold window there is a value for 16.17. I thought perhaps the decimal indicated that it was an average but I added up the values in the last 5 mins before 1:00 am and it didn't come out to 16.17.
You can see also I ran a query and asked for the sum of the last 5 mins and this seems to match the values in the chart below where the actual thresholds are set. I wont let me add any more attachments but basicallly the numbers matched for a sum of last 5 mins with the bottom preview chart.
So it seems the Preview Aggregate Threshold is a sampling of a kpi despite what calculation you asked for in the Base search while the lower one near the actual input for the threshold is the calculation of the kpi that you asked for.
Interestingly enough, these two number seemed very close to each other when I asked for an Average in the KPI base search rather than a sum.
... View more
10-11-2018
11:41 AM
1 Karma
Sorry, what you are saying isn't making sense to me.
The first part did. Yes, I set the metric to SUM a particular KPI value. The bottom screenshot seems to display that metric correctly. The top one does not.
I dont understand your second recommendation. Why would I pick 2 points in time and measure the sum from both time values? Im trying to compare the data from one chart to another and verify it is the same data.
... View more
10-10-2018
01:18 PM
I am seeing a difference in data (see screenshots) between the data previewed in the Preview Aggregate Thresholds and the data previewed below it under the Configure Thresholds for Time Policy. Does anyone know why these values would be different? Is the top one some kind of average of something?
You can see in the screenshots that for Friday, October 15th, 2018 at 12:00:00 PM, the top one shows a value of 2149.83 and the bottom shows a value of 7138.
... View more
10-10-2018
12:19 PM
Thanks. Using eventstats it did backfill but the results of the numbers are different from when I use stats and I am unsure why. Logically they shouldn't be.
... View more
10-10-2018
11:57 AM
Interesting. So eventstats should not prevent the backfill then? I will test this out.
... View more
10-10-2018
11:05 AM
Okay I think I have an idea of what is happening though I dont understand why.
I needed to create KPIs that were based on the computation of other KPIs. So, I created these services with a query like:
sourcetype=abc-prod A0010 | stats sum(foo) as FOO sum(bar) as BAR | eval FOOBYSEC = FOO/300
And then, I was grabbing KPIs based on the new kpi of FOOBYSEC
(For this query I need the sum of a KPI value for 5 minutes divided by 300 to show average per second, this is the only way I know how to do this)
When I use a query like that, it doesnt backfill as expected.
I created a new service with a simple query and now it seems to work. But, I'm back at square one because I need to show that KPI value for 5 minutes of average per second...
... View more
10-10-2018
10:54 AM
Okay I think I have an idea of what is happening though I dont understand why.
I needed to create kpis that were based on the computation of other kpis so I created these services with a query like:
sourcetype=abc-prod A0010 | stats sum(foo) as FOO sum(bar) as BAR | eval FOOBYSEC = FOO/300
And then I was grabbing KPIs based on the new kpi of FOOBYSEC
(For this query I need the sum of a kpi value for 5 minutes divided by 300 to show average per second, this is the only way I know how to do this)
When I use a query like that, it doesnt seem to allow me to use the preview data in Aggregate Threshold Values preview.
I created a new service with a simple query and now it seems to work. But Im back at square one because I need to show that kpi value for 5 minutes of average per second...
... View more
10-10-2018
10:30 AM
1 Karma
For some reason, when I attempt to preview data in the Aggregate Threshold View window for data that DEFINITELY exists and I can see it in the preview above the Aggregate Threshold View window, it does not show up. See screenshot.
Has anyone else experienced this?
... View more
10-10-2018
10:02 AM
1 Karma
I cannot see the Backfill of 7 days that I specified for a KPI in a service. I created a KPI Base Search and then, when I added the KPI, I specified a 7 days backfill. I see the notification in the notification dropdown saying it was completed successfully in 0 m (which seems odd to me).
Yet, when I go to the KPI in the Deep Dive, it only has stats for that KPI to the time I started that service.
Is there an additional step I am missing?
Has anyone else experienced this?
... View more
09-14-2018
12:16 PM
Also please note that I had to install Java on my server that ITSI was installed on so it could do Notable Events Aggregation
... View more
09-14-2018
11:04 AM
Yes, all I needed to do was turn OFF Event Grouping. Can you explain why having it ON would prevent any events from appearing?
... View more
09-13-2018
01:28 PM
I am literally running the searches from the Correlation Search config screen and it is returning results but refuses to put them into the Notable Events Review page, no matter what I do. What am I missing?
... View more
09-13-2018
10:57 AM
I am pretty confused as I have created a very basic multi-KPI Alert that basically triggers if my KPI is at Normal status or Higher (and yes, it is) so I can see what it looks like when Notable Events appear.
Yet no events are appearing ... I have enabled the correlation search... Can anyone think of any other suggestions why it may not be triggering? I'm completely at a loss.
... View more
