topic Re: Event aggregation before Indexing in Deployment Architecture

Event aggregation before Indexing

gooza — Thu, 24 Apr 2014 03:47:15 GMT

Hi

we have many sources that sends us a lot of similar events (DNS for example)

a.host.com 1.1.1.1

a.host.com 1.1.1.1

a.host.com 1.1.1.1

Is there a way for me to aggregate the events to one single event and just add the count of events that happened (lets say in one minute) ? so I will only index:

a.host.com 1.1.1.1 count=3

Is it possible to do this in splunk ( maybe with regex in transforms)?

For now we are depended on 3rd party agents like arcsight in front of splunk to do the aggregation - I would really like to see this feature in splunk

Any ideas?

Re: Event aggregation before Indexing

linu1988 — Thu, 24 Apr 2014 06:05:28 GMT

No you will not be able to do that at all. One possibility is to use a powershell script and count the events and send using scripted input.

Re: Event aggregation before Indexing

gooza — Thu, 24 Apr 2014 06:37:54 GMT

ok ,thanks for the replay, I hope splunk will add it in the future.

if anyone else need this feature please vote up this question so splunk can see the need.

Re: Event aggregation before Indexing

MuS — Thu, 24 Apr 2014 07:57:28 GMT

Hi gooza,

feel free to open an enhancement request on the support portal - this would be a P4 support case http://www.splunk.com/support .

cheers, MuS

Re: Event aggregation before Indexing

gooza — Thu, 24 Apr 2014 08:22:20 GMT

will do , thanks MuS

Re: Event aggregation before Indexing

DalJeanis — Mon, 06 Mar 2017 20:53:37 GMT

A) Create a preprocessor program in any language that will aggregate the data and create a single event.

B) Add the events to a temporary index that rolls off in a few hours or days, then create a summary index using that temporary index as input.