Splunk ITSI

How to manipulate metric values so that Service Health Score can be changed on Ad-Hoc basis for demonstrations?

Chiranjit-Mitra
Loves-to-Learn

I've got the service monitoring in place and now I want to run ad-hoc tests. How do we spoof fake alarms and see how the impact relationship within the service model change?

I figured the following two options:

  1. Used the stressors (Linux Stressor-ng https://www.mankier.com/1/stress-ng#Examples) on the servers to increase or decrease the resource consumption.
    1. This option is not so fast for customer demonstrations when it comes to break and fix things quickly.

  2. Ingest fake alarms using HEC towards the itsi_tracked_alerts index.
    1. I managed to get this working using curl, however, the event doesn’t seem to associate with the metrics.

Do you have any tips for achieving this use case? Are there any other options available?

Labels (2)
0 Karma
Get Updates on the Splunk Community!

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...

Index This | What goes away as soon as you talk about it?

May 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this month’s ...

What's New in Splunk Observability Cloud and Splunk AppDynamics - May 2025

This month, we’re delivering several new innovations in Splunk Observability Cloud and Splunk AppDynamics ...