All Apps and Add-ons

Does Splunk and Elastic Map Reduce work Together?

skoelpin
SplunkTrust
SplunkTrust

I have a few indexes which have around 2.5 billion events each. Unfortunately we don't have a lot of CPU to sort through this massive data and make it meaningful in a dashboard. We're currently in the process of setting up a summary index, but the requirements/fields can change at anytime which mean's we'd have to re-summerize that data.

So my question is, can we use Amazon EMR as a temporary boost in horsepower to Map and Reduce this data back into the summary index? How difficult would this be to do?

1 Solution

hsesterhenn
Path Finder

Hi,

I would assume your instance with 2,5 billion events is also running on AWS?

Why not use HUNK on AWS and export your data with the Splunk Hadoop Connect App?

https://aws.amazon.com/de/elasticmapreduce/hunk/

HTH,

Holger

View solution in original post

hsesterhenn
Path Finder

Hi,

I would assume your instance with 2,5 billion events is also running on AWS?

Why not use HUNK on AWS and export your data with the Splunk Hadoop Connect App?

https://aws.amazon.com/de/elasticmapreduce/hunk/

HTH,

Holger

Get Updates on the Splunk Community!

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Avoid Certificate Expiry Issues in Splunk Enterprise with Certificate Assist

This blog post is part 2 of 4 of a series on Splunk Assist. Click the links below to see the other ...