Getting Data In

What is the best method to write all event to RFS with Ingest Actions?

dokaas_2
Path Finder

Our requirements are to have readily searchable data for 12 months and 'cold store' of data for an additional 18 mths (30 mths total).  Ingest Actions seems like the obvious choice since it can write to an S3 bucket and compress the data in a format easily re-ingested or passed to a 3rd party if needed.  However, the ingest actions seem to only work given you apply the ruleset to a sourcetype.  Given that there may be a hundred or more sourcetypes, this is a little onerous.  Is there a method to accomplish this w/o creating a ruleset for every sourcetype?

Labels (3)
Tags (3)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Use a coldToFrozenScript for each index you wish to archive to S3.  Splunk will invoke the script for each bucket that reaches the end of its searchable lifetime.  Details are in the Admin Manual.

---
If this reply helps you, Karma would be appreciated.
0 Karma

dokaas_2
Path Finder

That script, I believe, moves frozen buckets to an S3 storage location.  However, I what I want is to use the ingest actions  (Settings / Data / Ingest actions) to write all raw events in compressed JSON format.  Ingest actions is per source type and with nearly 100 or more source types, manually creating one for each is onerous.  

Ingest actions use RULESET in the props to define how to store on the Remote File System (RFS).

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

you should remember that with IA you could use S3 only if you are running that instance on AWS!

Splunk have "secret" wildcard for sourcetype in props.conf. I don't know if it works also for IA RULESET or not and can you use it via GUI or not. You can read more from https://www.splunk.com/en_us/blog/tips-and-tricks/quick-tip-wildcard-sourcetypes-in-props-conf.html. BUT remember that this is not officially supported and it can removed anytime!!

For that reason I also prefer @richgalloway 's proposal to use cold2frozen script. 

r. Ismo

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...