Splunk Enterprise

What is the best option to send/store data to S3 as and when the data lands in Splunk?

danilreddy
Loves-to-Learn Everything

I have an use case where I need to run the analytics on top of data that lands into Splunk. So, I want to store all the data into S3 too as and when the data lands into Splunk.

I would like to know the best possible way we have with latest version of Splunk Enterprise/Splunk Cloud platform to save copy of Splunk data into S3 as and when the data comes into Splunk.

Please give suggestions on the same.

Thanking you.

Labels (2)
0 Karma

gjanders
SplunkTrust
SplunkTrust
0 Karma

danilreddy
Loves-to-Learn Everything

I want to forward data that is loaded to Splunk, not from search!

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@danilreddy - Splunk can forward data to not only Splunk but any external system or script as well.

One way you can do that is as follows.

 

Another way to approach this is to move Splunk to smart storage where Splunk itself stores the data on S3 buckets.

And then you can do the analysis that you want right on Splunk. You can use tools like the Machine learning toolkit (https://www.splunk.com/en_us/software/splunk-enterprise/machine-learning.html ) and you can also have your own Python tools and scripts that you can use.

This way you will require less storage and you don't have to right your own script that sends data to the cloud.

https://docs.splunk.com/Documentation/Splunk/8.2.6/Indexer/AboutSmartStore 

 

Which approach to choose depends on:

  • how much data you want to store on S3
  • what kind of analysis you want to do
  • is there any business requirement
  • etc

 

I hope this helps!!! Karma/upvote would be appreciated!!!

0 Karma

danilreddy
Loves-to-Learn Everything

@VatsalJagani 

Thanks for the prompt response. I got some understanding going through your inputs.

But I could not able to find how to forward the parsed index data with schema.

When I tried sendCookedData to true, it sends the data in un-readable format.

Please let me know if there is way to forward the structured parsed data to third party system.

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@danilreddy - Splunk-cooked data can only be understand by Splunk.

 

You want to send data to third-party system (S3 buckets here), you can use this document to send data on Syslog, this will send data as read by Splunk not in the same format as you can see on Splunk.

https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Forwarddatatothird-partysystemsd 

 

0 Karma

danilreddy
Loves-to-Learn Everything

@VatsalJagani  I tried this configuration. It sends the raw data and as you said cooked data is not in readable format. Thanks for your response.

 

Splunk Team:

I am researching for the option that sends the indexed data in json format. I am trying IndexAndForward and _Index_and_forward_routing etc configurations but I am unable to succeed,

Can I get simple tutorial that explains Index and forward usecase.

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

@danilreddy - Index and forward are just the cooked data forwarding. 

With just that it will also index data locally on the Splunk forwarder.

Useful when you want to clone the same data to two different Splunk instance/cluster.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...