Deployment Architecture

Event aggregation before Indexing

gooza
Communicator

Hi

we have many sources that sends us a lot of similar events (DNS for example)

a.host.com 1.1.1.1

a.host.com 1.1.1.1

a.host.com 1.1.1.1

Is there a way for me to aggregate the events to one single event and just add the count of events that happened (lets say in one minute) ? so I will only index:

a.host.com 1.1.1.1 count=3

Is it possible to do this in splunk ( maybe with regex in transforms)?

For now we are depended on 3rd party agents like arcsight in front of splunk to do the aggregation - I would really like to see this feature in splunk

Any ideas?

1 Solution

linu1988
Champion

No you will not be able to do that at all. One possibility is to use a powershell script and count the events and send using scripted input.

View solution in original post

DalJeanis
Legend

A) Create a preprocessor program in any language that will aggregate the data and create a single event.

B) Add the events to a temporary index that rolls off in a few hours or days, then create a summary index using that temporary index as input.

0 Karma

linu1988
Champion

No you will not be able to do that at all. One possibility is to use a powershell script and count the events and send using scripted input.

gooza
Communicator

will do , thanks MuS

0 Karma

MuS
Legend

Hi gooza,

feel free to open an enhancement request on the support portal - this would be a P4 support case http://www.splunk.com/support .

cheers, MuS

0 Karma

gooza
Communicator

ok ,thanks for the replay, I hope splunk will add it in the future.

if anyone else need this feature please vote up this question so splunk can see the need.

0 Karma
Get Updates on the Splunk Community!

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...

Index This | What goes away as soon as you talk about it?

May 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this month’s ...

What's New in Splunk Observability Cloud and Splunk AppDynamics - May 2025

This month, we’re delivering several new innovations in Splunk Observability Cloud and Splunk AppDynamics ...