Does Splunk and Elastic Map Reduce work Together?

[skoelpin]

I have a few indexes which have around 2.5 billion events each. Unfortunately we don't have a lot of CPU to sort through this massive data and make it meaningful in a dashboard. We're currently in the process of setting up a summary index, but the requirements/fields can change at anytime which mean's we'd have to re-summerize that data.

So my question is, can we use Amazon EMR as a temporary boost in horsepower to Map and Reduce this data back into the summary index? How difficult would this be to do?

[hsesterhenn · Accepted solution]

Hi,

I would assume your instance with 2,5 billion events is also running on AWS?

Why not use HUNK on AWS and export your data with the Splunk Hadoop Connect App?

https://aws.amazon.com/de/elasticmapreduce/hunk/

HTH,

Holger