Deployment Architecture

Event aggregation before Indexing

gooza
Communicator

Hi

we have many sources that sends us a lot of similar events (DNS for example)

a.host.com 1.1.1.1

a.host.com 1.1.1.1

a.host.com 1.1.1.1

Is there a way for me to aggregate the events to one single event and just add the count of events that happened (lets say in one minute) ? so I will only index:

a.host.com 1.1.1.1 count=3

Is it possible to do this in splunk ( maybe with regex in transforms)?

For now we are depended on 3rd party agents like arcsight in front of splunk to do the aggregation - I would really like to see this feature in splunk

Any ideas?

1 Solution

linu1988
Champion

No you will not be able to do that at all. One possibility is to use a powershell script and count the events and send using scripted input.

View solution in original post

DalJeanis
SplunkTrust
SplunkTrust

A) Create a preprocessor program in any language that will aggregate the data and create a single event.

B) Add the events to a temporary index that rolls off in a few hours or days, then create a summary index using that temporary index as input.

0 Karma

linu1988
Champion

No you will not be able to do that at all. One possibility is to use a powershell script and count the events and send using scripted input.

gooza
Communicator

will do , thanks MuS

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi gooza,

feel free to open an enhancement request on the support portal - this would be a P4 support case http://www.splunk.com/support .

cheers, MuS

0 Karma

gooza
Communicator

ok ,thanks for the replay, I hope splunk will add it in the future.

if anyone else need this feature please vote up this question so splunk can see the need.

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...