Getting Data In

Data Storage: Splunk License options for data retention

koshyk
Super Champion

We have a customer who has Splunk as main Security platform, but now they are trying to onboard other datasets for forensic/compliance/data retention purposes/application data. This doesn't need to be in Splunk as such, but any searchable tools like OpenSearch or similar. Before looking into such extra tools, wanted to understand if there is any provision with Splunk which would allow a data ingestion at cheaper cost (not counting to the main license cost or a cheaper license option?)

So the scenario is

(Security + compliance + application data) => Splunk Heavy Forwarder -> (A) Security data to Splunk  &&  (B) Rest of data to a log retention service

Before going into this avenue, wanted to check if Splunk provide such a cheaper license option? i.e. for a log retention mode or non-important data (In future, they may have funding to move into Splunk, but not for atleast 6-8 months)

 

Labels (2)
0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust

The pricing model is named as Workload Pricing. It is available both Cloud and OnPrem with differently named capacity units. But the bigger issue is that it's price structure is "starting from 2-3TB/day" before it has competitive prices for normal use (if I recall right those price levels) 😞

But anyhow you could ask it, but don't surprised  if they don't sell it to you.

You could also ask Predictice Pricing Program which has levels from 125GB to 2TB and 2TB+ levels.

r. Ismo

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Consider Workload Pricing rather than an Ingest Pricinv.  With Workload Pricing, you pay based on the compute resources used instead of based on how much data is indexed.  See https://www.splunk.com/en_us/software/pricing/faqs.html#workload-pricing for more.

---
If this reply helps you, Karma would be appreciated.

koshyk
Super Champion

thanks for the idea. upvoted. Do you have a rough idea if it is cheaper vs the default licensing methodology? or is it more of a question for Splunk Sales?

0 Karma

isoutamo
SplunkTrust
SplunkTrust

As there are counted all (v)CPUs which participate search (excl. LM) it will be more expensive for small ingestion data amounts. 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Are you sure Workload Pricing is available for the core Splunk Enterprise? Just asking because never worked with this licensing model. I know it works with Cloud but Splunk website says vaguely about "some on-premise offerings".

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Not 100% sure because Splunk is a bit vague about pricing, but it's worth asking Sales about.

---
If this reply helps you, Karma would be appreciated.
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Sure. The worst that can happen is that they say "no way" 🙂

0 Karma

isoutamo
SplunkTrust
SplunkTrust

The pricing model is named as Workload Pricing. It is available both Cloud and OnPrem with differently named capacity units. But the bigger issue is that it's price structure is "starting from 2-3TB/day" before it has competitive prices for normal use (if I recall right those price levels) 😞

But anyhow you could ask it, but don't surprised  if they don't sell it to you.

You could also ask Predictice Pricing Program which has levels from 125GB to 2TB and 2TB+ levels.

r. Ismo

koshyk
Super Champion

great ideas. This is exactly I was looking for. Will get in touch with Sales on these topic/ideas

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @koshyk,

Splunk license is countered only on daily indexed volume, so there isn't in Splunk a cheaper way to ingest logs because all the indexed logs are countered in the Splunk license.

The question is: do you want to use Splunk for searching and monitoring or not?

if yes, the only way is to pay the license; if not, you have to design a different architecture with a different product outside Splunk.

Retention isn't relevant for Splunk costs, the only relevant parameter is the daily indexed log volume.

But if you want to store some logs only for compliance without using them in searches and monitoring, why don't you store them in a simple file system, outside Splunk?

Ciao.

Giuseppe

0 Karma

koshyk
Super Champion

Agreed. I understand the current license structure, but was checking if there was a other type of license within Splunk or if any of you have guys done it in a different way

Yeah, we have options to store outside Splunk. Of course, we may look into such a architecture using other products (as filesystem storage is bit clunky). There are few advanced thoughts as well, to store raw data in other tools and send a summary information to Splunk. So different avenues/thoughts, but wanted to see if there is any clever way within Splunk before venturing to other tools

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Well, I can imagine why someone would want to use splunk just for analytics/visualization without actually monitoring current data in it. Last month or so we had a question if splunk can be used as a visualization layer only (the data would come from db queries or something like that).

So yes, there is a possibility to - for example - store data outside of splunk, pull it into splunk search by custom command on search head and manipulate it there.

But this is a flawed solution, especially if we're talking about big volumes of data - we're not able to use splunk (parallel) search features in the first place.

So in the end in order to use splunk to ingest some data you have to have license for that data. If you want to lower your licensing requirement you might spread the data onboarding over a longer time period. For example  - if you have 10G worth of raw data, you can onboard it over a single day - for it you would  need a 10G license which would prove unnecessary for the rest of your licensing period. But you can onboard it over 10 days using just 1G daily license.

 

0 Karma

koshyk
Super Champion

Well, in my experience, many of the clients/users still don't understand their entire estate ! By the time, we reach someone would have estimated the data and said.. oh, its only 1000 windows & linux, so would be 10GB per day, while in reality the data would then start from "auditd", "applications" and far exceed 10x the initial estimate and suddently no funding available.

I personally feel, Splunk should have a secondary license to cater for trivial/less-important data and collection and when it is required to search, there should be charged separately. Many customers never realise the value until unless they see it.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

That's where Trial license and Splunk partners come into play.

You can deploy a low-scaled environment to appraise the average per-source license consumption and then scale it out accordingly (works with relatively homogenous environments). And you can ask your Splunk partner for help in estimating license consumption.

Sometimes, however, there's no telling beforehand - let's say you have a bunch of firewalls which are dumping flow and security events. You can to some extend calculate that if you have several millions of TCP sessions per hour, you wil surely won't go below 10GB per day 😉 but you might have difficulty calculating the upper limit.

And, let's be honest, If you honestly think you're gonna need 1GB per day and it turns out your devices spit out 100 times that, there's something wrong with your logging setup, not necessarily with the license sizing. It's often also the case of "what you have" (i.e. we're pushing all the data that the device can possibly produce) vs. "what you need". Typical case from security realm - FireEye devices can forward both internal system logs as well as security events to an external log receiver. If you're deploying a log management solution for security, there's no point of logging the internal daemons' logs but it's typical for the admins to just forward everything. The difference is that if you're in a relatively "peaceful" environment, you might get just 20 or 50 security events per day whereas system logging produces several thousands or even millions events per day. That's when it's good if your Splunk partner has experience with more than Splunk alone and can help you with such cases.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...