Data Storage: Splunk License options for data retention

[koshyk]

We have a customer who has Splunk as main Security platform, but now they are trying to onboard other datasets for forensic/compliance/data retention purposes/application data. This doesn't need to be in Splunk as such, but any searchable tools like OpenSearch or similar. Before looking into such extra tools, wanted to understand if there is any provision with Splunk which would allow a data ingestion at cheaper cost (not counting to the main license cost or a cheaper license option?)

So the scenario is

(Security + compliance + application data) => Splunk Heavy Forwarder -> (A) Security data to Splunk  &&  (B) Rest of data to a log retention service

Before going into this avenue, wanted to check if Splunk provide such a cheaper license option? i.e. for a log retention mode or non-important data (In future, they may have funding to move into Splunk, but not for atleast 6-8 months)

[richgalloway]

Consider Workload Pricing rather than an Ingest Pricinv.  With Workload Pricing, you pay based on the compute resources used instead of based on how much data is indexed.  See https://www.splunk.com/en_us/software/pricing/faqs.html#workload-pricing for more.

[koshyk]

thanks for the idea. upvoted. Do you have a rough idea if it is cheaper vs the default licensing methodology? or is it more of a question for Splunk Sales?

[isoutamo]

As there are counted all (v)CPUs which participate search (excl. LM) it will be more expensive for small ingestion data amounts. 

[PickleRick]

Are you sure Workload Pricing is available for the core Splunk Enterprise? Just asking because never worked with this licensing model. I know it works with Cloud but Splunk website says vaguely about "some on-premise offerings".

[richgalloway]

Not 100% sure because Splunk is a bit vague about pricing, but it's worth asking Sales about.

[PickleRick]

Sure. The worst that can happen is that they say "no way" 🙂

[isoutamo · Accepted solution]

The pricing model is named as Workload Pricing. It is available both Cloud and OnPrem with differently named capacity units. But the bigger issue is that it's price structure is "starting from 2-3TB/day" before it has competitive prices for normal use (if I recall right those price levels) 😞

But anyhow you could ask it, but don't surprised  if they don't sell it to you.

You could also ask Predictice Pricing Program which has levels from 125GB to 2TB and 2TB+ levels.

r. Ismo

[koshyk]

great ideas. This is exactly I was looking for. Will get in touch with Sales on these topic/ideas

[gcusello]

Hi @koshyk,

Splunk license is countered only on daily indexed volume, so there isn't in Splunk a cheaper way to ingest logs because all the indexed logs are countered in the Splunk license.

The question is: do you want to use Splunk for searching and monitoring or not?

if yes, the only way is to pay the license; if not, you have to design a different architecture with a different product outside Splunk.

Retention isn't relevant for Splunk costs, the only relevant parameter is the daily indexed log volume.

But if you want to store some logs only for compliance without using them in searches and monitoring, why don't you store them in a simple file system, outside Splunk?

Ciao.

Giuseppe

[koshyk]

Agreed. I understand the current license structure, but was checking if there was a other type of license within Splunk or if any of you have guys done it in a different way

Yeah, we have options to store outside Splunk. Of course, we may look into such a architecture using other products (as filesystem storage is bit clunky). There are few advanced thoughts as well, to store raw data in other tools and send a summary information to Splunk. So different avenues/thoughts, but wanted to see if there is any clever way within Splunk before venturing to other tools

[PickleRick]

Well, I can imagine why someone would want to use splunk just for analytics/visualization without actually monitoring current data in it. Last month or so we had a question if splunk can be used as a visualization layer only (the data would come from db queries or something like that).

So yes, there is a possibility to - for example - store data outside of splunk, pull it into splunk search by custom command on search head and manipulate it there.

But this is a flawed solution, especially if we're talking about big volumes of data - we're not able to use splunk (parallel) search features in the first place.

So in the end in order to use splunk to ingest some data you have to have license for that data. If you want to lower your licensing requirement you might spread the data onboarding over a longer time period. For example  - if you have 10G worth of raw data, you can onboard it over a single day - for it you would  need a 10G license which would prove unnecessary for the rest of your licensing period. But you can onboard it over 10 days using just 1G daily license.

[koshyk]

Well, in my experience, many of the clients/users still don't understand their entire estate ! By the time, we reach someone would have estimated the data and said.. oh, its only 1000 windows & linux, so would be 10GB per day, while in reality the data would then start from "auditd", "applications" and far exceed 10x the initial estimate and suddently no funding available.

I personally feel, Splunk should have a secondary license to cater for trivial/less-important data and collection and when it is required to search, there should be charged separately. Many customers never realise the value until unless they see it.

[PickleRick]

That's where Trial license and Splunk partners come into play.

You can deploy a low-scaled environment to appraise the average per-source license consumption and then scale it out accordingly (works with relatively homogenous environments). And you can ask your Splunk partner for help in estimating license consumption.

Sometimes, however, there's no telling beforehand - let's say you have a bunch of firewalls which are dumping flow and security events. You can to some extend calculate that if you have several millions of TCP sessions per hour, you wil surely won't go below 10GB per day 😉 but you might have difficulty calculating the upper limit.

And, let's be honest, If you honestly think you're gonna need 1GB per day and it turns out your devices spit out 100 times that, there's something wrong with your logging setup, not necessarily with the license sizing. It's often also the case of "what you have" (i.e. we're pushing all the data that the device can possibly produce) vs. "what you need". Typical case from security realm - FireEye devices can forward both internal system logs as well as security events to an external log receiver. If you're deploying a log management solution for security, there's no point of logging the internal daemons' logs but it's typical for the admins to just forward everything. The difference is that if you're in a relatively "peaceful" environment, you might get just 20 or 50 security events per day whereas system logging produces several thousands or even millions events per day. That's when it's good if your Splunk partner has experience with more than Splunk alone and can help you with such cases.