Getting Data In

What is the best method to write all event to RFS with Ingest Actions?

dokaas_2
Communicator

Our requirements are to have readily searchable data for 12 months and 'cold store' of data for an additional 18 mths (30 mths total).  Ingest Actions seems like the obvious choice since it can write to an S3 bucket and compress the data in a format easily re-ingested or passed to a 3rd party if needed.  However, the ingest actions seem to only work given you apply the ruleset to a sourcetype.  Given that there may be a hundred or more sourcetypes, this is a little onerous.  Is there a method to accomplish this w/o creating a ruleset for every sourcetype?

Labels (3)
Tags (3)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Use a coldToFrozenScript for each index you wish to archive to S3.  Splunk will invoke the script for each bucket that reaches the end of its searchable lifetime.  Details are in the Admin Manual.

---
If this reply helps you, Karma would be appreciated.
0 Karma

dokaas_2
Communicator

That script, I believe, moves frozen buckets to an S3 storage location.  However, I what I want is to use the ingest actions  (Settings / Data / Ingest actions) to write all raw events in compressed JSON format.  Ingest actions is per source type and with nearly 100 or more source types, manually creating one for each is onerous.  

Ingest actions use RULESET in the props to define how to store on the Remote File System (RFS).

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

you should remember that with IA you could use S3 only if you are running that instance on AWS!

Splunk have "secret" wildcard for sourcetype in props.conf. I don't know if it works also for IA RULESET or not and can you use it via GUI or not. You can read more from https://www.splunk.com/en_us/blog/tips-and-tricks/quick-tip-wildcard-sourcetypes-in-props-conf.html. BUT remember that this is not officially supported and it can removed anytime!!

For that reason I also prefer @richgalloway 's proposal to use cold2frozen script. 

r. Ismo

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...