Alerting

real time alret action performance iisue

shavitpren
Loves-to-Learn

Hi,

I want to create a real time alert of about 3000 Messages per secend.

I want to create action for each message to create an http alert to another system.

my problam is that when I tryed to do that i recived about 50-100 messages per second and i got a big delay.

what is the best way to handle this throughput?

can we use batch in teal time laert?

Labels (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @shavitpren,

some question to identify the issue:

at first, what infrastructure are you using (in termes of architecture and recommended hardware?

Because performaces mainly depend on the available CPUs and especially on disk performaces, have you at least 800 IOPS on your disks?

are you sure that the server you're using to ingest events is able to index events without delays (outside the search)?

Then, is it mandatory to have a real time search? could it be a search scheduled e.g. every minute?

this second option is better for performances.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...