topic How to condition an alert for a chart? in Alerting

How to condition an alert for a chart?

dummy_splunk — Fri, 01 Jul 2022 21:03:20 GMT

good morning community

I want to generate an alert in splunk based on some graphs that are generated from a .TXT file, therefore I only need to use the last two values generated in said file to apply a formula if said value drops 10% of its measurement.

When I query the TXT file which displays a list as follows in the events:

2022-7-1 11:00:0 OVERALL: 10000
2022-7-1 12:00:0 OVERALL: 11000

I just need to get the last numeric value and the penultimate numeric value registered in the list and add them to a variable to apply the formula of comparing these two values to see if there is a difference of more than 10%.

Please, if you have had a similar case, please share the solution.

Re: condition an alert for a chart

PickleRick — Fri, 01 Jul 2022 18:25:20 GMT

Honestly I have no idea how you would want to execute a search over a text file. You have to onboard the data from the file into splunk first either as events or as lookup contents. Then you could relatively easily search for such condition.

You could of course create a custom command to manipulate text files but then you really don't need Splunk in the first place.

Re: condition an alert for a chart

dummy_splunk — Fri, 01 Jul 2022 19:38:02 GMT

hello @PickleRick I perform the following search:

index = "monitor" source="/monitor/recolector.txt" | timechart values(valor) as recolector span=1h

Result:

2022-7-1 11:19:0 GLOBAN: 15000

2022-7-1 11:29:0 GLOBAN: 15200

2022-7-1 11:39:0 GLOBAN: 16200

.............................................

From that result I generate a graph and therefore I want to take the last two measurements as values to be able to compare if there was a loss in the values by more than 10%

any suggestion?

Re: condition an alert for a chart

PickleRick — Fri, 01 Jul 2022 20:26:34 GMT

Well, timechart doesn't usually go well with values(). If you have a value for about every hour and just want to snap timestamps to full hour you can simply do

| bin _time span=1h

Instead of your timechart.

Now get last two values

| tail 2

Get just your field (your output doesn't match your search; I'll assume it's called recollector)

| fields recollector

Transpose it so you have both values in single row

| transpose

Now you have two fields in your result row called 'row 1' and 'row 2'. You can easily verify if the difference is more than 10% using where

| where 'row 2'<'row 1'*0.9

et volia

Re: condition an alert for a chart

dummy_splunk — Fri, 01 Jul 2022 21:51:38 GMT

thanks @PickleRick I tried with the suggestion you make me but I get to this point with the following result:

and I want to make the comparison with the numeric values of the _raw column to apply the where

how could i do it?

Re: condition an alert for a chart

PickleRick — Fri, 01 Jul 2022 21:57:21 GMT

You're doing something strange before. _indextime is not a field I'd expect to see.

But anyway, you need to parse out the nummerical value only from the field using rex

| rex field=recolector "GLOBAL_DIARIO:\s*(?<recollector>\d+)"

Put it right after your initial search.

And do

| fields - _*

Re: condition an alert for a chart

dummy_splunk — Fri, 01 Jul 2022 22:34:31 GMT

perfect @PickleRick I tried again with the suggestion you make me but I get to this point:

The only result that should be reflected is the "valor" field to perform the where, because if I apply it with those other columns, it takes them into consideration for the condition.

How could I omit the other fields and only the result is "valor"

Re: condition an alert for a chart

dummy_splunk — Mon, 04 Jul 2022 16:27:57 GMT

Good afternoon @PickleRick with this query I managed to obtain the values I needed

index = "monitor" source="monitor/recolector.txt" | timechart values(valor) as metrica span=1h | where metrica>1 | tail 2 | transpose

Thanks for your collaboration, I just have to add the conditions to complete 🙂