Deployment Architecture

Event aggregation before Indexing

gooza
Communicator

Hi

we have many sources that sends us a lot of similar events (DNS for example)

a.host.com 1.1.1.1

a.host.com 1.1.1.1

a.host.com 1.1.1.1

Is there a way for me to aggregate the events to one single event and just add the count of events that happened (lets say in one minute) ? so I will only index:

a.host.com 1.1.1.1 count=3

Is it possible to do this in splunk ( maybe with regex in transforms)?

For now we are depended on 3rd party agents like arcsight in front of splunk to do the aggregation - I would really like to see this feature in splunk

Any ideas?

1 Solution

linu1988
Champion

No you will not be able to do that at all. One possibility is to use a powershell script and count the events and send using scripted input.

View solution in original post

DalJeanis
Legend

A) Create a preprocessor program in any language that will aggregate the data and create a single event.

B) Add the events to a temporary index that rolls off in a few hours or days, then create a summary index using that temporary index as input.

0 Karma

linu1988
Champion

No you will not be able to do that at all. One possibility is to use a powershell script and count the events and send using scripted input.

gooza
Communicator

will do , thanks MuS

0 Karma

MuS
Legend

Hi gooza,

feel free to open an enhancement request on the support portal - this would be a P4 support case http://www.splunk.com/support .

cheers, MuS

0 Karma

gooza
Communicator

ok ,thanks for the replay, I hope splunk will add it in the future.

if anyone else need this feature please vote up this question so splunk can see the need.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...