Splunk Enterprise

What is the best option to send/store data to S3 as and when the data lands in Splunk?

danilreddy
Loves-to-Learn Everything

I have an use case where I need to run the analytics on top of data that lands into Splunk. So, I want to store all the data into S3 too as and when the data lands into Splunk.

I would like to know the best possible way we have with latest version of Splunk Enterprise/Splunk Cloud platform to save copy of Splunk data into S3 as and when the data comes into Splunk.

Please give suggestions on the same.

Thanking you.

Labels (2)
0 Karma

gjanders
SplunkTrust
SplunkTrust
0 Karma

danilreddy
Loves-to-Learn Everything

I want to forward data that is loaded to Splunk, not from search!

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@danilreddy - Splunk can forward data to not only Splunk but any external system or script as well.

One way you can do that is as follows.

 

Another way to approach this is to move Splunk to smart storage where Splunk itself stores the data on S3 buckets.

And then you can do the analysis that you want right on Splunk. You can use tools like the Machine learning toolkit (https://www.splunk.com/en_us/software/splunk-enterprise/machine-learning.html ) and you can also have your own Python tools and scripts that you can use.

This way you will require less storage and you don't have to right your own script that sends data to the cloud.

https://docs.splunk.com/Documentation/Splunk/8.2.6/Indexer/AboutSmartStore 

 

Which approach to choose depends on:

  • how much data you want to store on S3
  • what kind of analysis you want to do
  • is there any business requirement
  • etc

 

I hope this helps!!! Karma/upvote would be appreciated!!!

0 Karma

danilreddy
Loves-to-Learn Everything

@VatsalJagani 

Thanks for the prompt response. I got some understanding going through your inputs.

But I could not able to find how to forward the parsed index data with schema.

When I tried sendCookedData to true, it sends the data in un-readable format.

Please let me know if there is way to forward the structured parsed data to third party system.

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@danilreddy - Splunk-cooked data can only be understand by Splunk.

 

You want to send data to third-party system (S3 buckets here), you can use this document to send data on Syslog, this will send data as read by Splunk not in the same format as you can see on Splunk.

https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Forwarddatatothird-partysystemsd 

 

0 Karma

danilreddy
Loves-to-Learn Everything

@VatsalJagani  I tried this configuration. It sends the raw data and as you said cooked data is not in readable format. Thanks for your response.

 

Splunk Team:

I am researching for the option that sends the indexed data in json format. I am trying IndexAndForward and _Index_and_forward_routing etc configurations but I am unable to succeed,

Can I get simple tutorial that explains Index and forward usecase.

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@danilreddy - Index and forward are just the cooked data forwarding. 

With just that it will also index data locally on the Splunk forwarder.

Useful when you want to clone the same data to two different Splunk instance/cluster.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...