All Apps and Add-ons

Does Splunk and Elastic Map Reduce work Together?

skoelpin
SplunkTrust
SplunkTrust

I have a few indexes which have around 2.5 billion events each. Unfortunately we don't have a lot of CPU to sort through this massive data and make it meaningful in a dashboard. We're currently in the process of setting up a summary index, but the requirements/fields can change at anytime which mean's we'd have to re-summerize that data.

So my question is, can we use Amazon EMR as a temporary boost in horsepower to Map and Reduce this data back into the summary index? How difficult would this be to do?

1 Solution

hsesterhenn
Path Finder

Hi,

I would assume your instance with 2,5 billion events is also running on AWS?

Why not use HUNK on AWS and export your data with the Splunk Hadoop Connect App?

https://aws.amazon.com/de/elasticmapreduce/hunk/

HTH,

Holger

View solution in original post

hsesterhenn
Path Finder

Hi,

I would assume your instance with 2,5 billion events is also running on AWS?

Why not use HUNK on AWS and export your data with the Splunk Hadoop Connect App?

https://aws.amazon.com/de/elasticmapreduce/hunk/

HTH,

Holger

Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...