What is the best option to send/store data to S3 as and when the data lands in Splunk?

[danilreddy]

I have an use case where I need to run the analytics on top of data that lands into Splunk. So, I want to store all the data into S3 too as and when the data lands into Splunk.

I would like to know the best possible way we have with latest version of Splunk Enterprise/Splunk Cloud platform to save copy of Splunk data into S3 as and when the data comes into Splunk.

Please give suggestions on the same.

Thanking you.

[gjanders]

If you instead interested in a selective export you could use Export Everything from SplunkBase

[danilreddy]

I want to forward data that is loaded to Splunk, not from search!

[VatsalJagani]

@danilreddy - Splunk can forward data to not only Splunk but any external system or script as well.

One way you can do that is as follows.

• Use this to forward data to your custom script - https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Forwarddatatothird-partysystemsd
• Write a custom script that reads all data sent by Splunk and send on S3 buckets.
• Now you can set this on:
  • Splunk Indexers
    • Then you need to enable index & forward on them.
    • https://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf?_gl=1*1llg2tp*_ga*NTMzODg1OTQ4... 
  • Intermediate Splunk Heavy Forwarder
    • Set data cloning in outputs.conf to send data to your script as well as your Indexers.
    • https://docs.splunk.com/Documentation/Splunk/8.2.6/Forwarding/Routeandfilterdatad
    • https://community.splunk.com/t5/Getting-Data-In/Can-the-heavy-forwarder-send-to-multiple-receivers/m...

Another way to approach this is to move Splunk to smart storage where Splunk itself stores the data on S3 buckets.

And then you can do the analysis that you want right on Splunk. You can use tools like the Machine learning toolkit (https://www.splunk.com/en_us/software/splunk-enterprise/machine-learning.html ) and you can also have your own Python tools and scripts that you can use.

This way you will require less storage and you don't have to right your own script that sends data to the cloud.

- https://docs.splunk.com/Documentation/Splunk/8.2.6/Indexer/AboutSmartStore 

Which approach to choose depends on:

• how much data you want to store on S3
• what kind of analysis you want to do
• is there any business requirement
• etc

I hope this helps!!! Karma/upvote would be appreciated!!!

[danilreddy]

@VatsalJagani 

Thanks for the prompt response. I got some understanding going through your inputs.

But I could not able to find how to forward the parsed index data with schema.

When I tried sendCookedData to true, it sends the data in un-readable format.

Please let me know if there is way to forward the structured parsed data to third party system.

[VatsalJagani]

@danilreddy - Splunk-cooked data can only be understand by Splunk.

You want to send data to third-party system (S3 buckets here), you can use this document to send data on Syslog, this will send data as read by Splunk not in the same format as you can see on Splunk.

- https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Forwarddatatothird-partysystemsd 

[danilreddy]

@VatsalJagani  I tried this configuration. It sends the raw data and as you said cooked data is not in readable format. Thanks for your response.

Splunk Team:

I am researching for the option that sends the indexed data in json format. I am trying IndexAndForward and _Index_and_forward_routing etc configurations but I am unable to succeed,

Can I get simple tutorial that explains Index and forward usecase.

[VatsalJagani]

@danilreddy - Index and forward are just the cooked data forwarding. 

With just that it will also index data locally on the Splunk forwarder.

Useful when you want to clone the same data to two different Splunk instance/cluster.