topic Re: How can we get usage related data from index ? in Installation

How can we get usage related data from index ?

arjun — Thu, 28 Nov 2024 13:58:44 GMT

How can we locate usage related data from splunk, I have onpremise splunk instance and looking for usage and billing related data grouped by day.
I am not able to locate data in any index.

Re: How can we get usage related data from index ?

gcusello — Thu, 28 Nov 2024 14:04:58 GMT

Hi @arjun ,

what's your requirement: to know the volume for each customer? or what else?

Could you better describe your environment and your situation?

E.g.: have you a multi-tenant environment or not?

Ciao.

Giuseppe

Re: How can we get usage related data from index ?

arjun — Thu, 28 Nov 2024 14:14:09 GMT

HI @gcusello ,

In splunk we monitor devices or Host and we get logs from them what i need to know how much memory (in GB) has been utilised by those host or log source where does splunk store such data in case of Onpremise instance ?

Re: How can we get usage related data from index ?

dural_yyz — Thu, 28 Nov 2024 14:23:07 GMT

Start with the DMC (Distributed Monitoring Console) to review the License usage broken down by index. This will share with you the daily ingest records for the last 30 days broken down by index. This is only a starting point as depending on how your environment was setup you may have very specific indexes or things may have been aggregated into only a few indexes.

From there you can start decided what questions come next.

Re: How can we get usage related data from index ?

PickleRick — Thu, 28 Nov 2024 15:38:36 GMT

Do you mean you want to monitor your Splunk infrastructure usage or do you ingest some data regarding "external" hosts? For the former as @dural_yyz mentioned, check Monitoring Console. You can also gather metrics from the _metrics index. For the latter - it depends on your environment. Splunk "just" happily gets the data you throw at it and can manipulate and search it. But it's up to your architects and admins to tell you where they set up the data and what it's made of.

Re: How can we get usage related data from index ?

gcusello — Fri, 29 Nov 2024 06:48:25 GMT

Hi @arjun ,

to monitor windows or Linux machines having a Universal Forwarder installed, you have to install on these UFs the related add on (Linux https://splunkbase.splunk.com/app/833 or windows https://splunkbase.splunk.com/app/742 ) enabling the input stanza for memory monitoring.

In this way you'll have the logs to use in your searches.

Ciao.

Giuseppe

Re: How can we get usage related data from index ?

arjun — Tue, 03 Dec 2024 09:06:51 GMT

Hi @gcusello , We have many client who uses splunk and we need to get some data from those splunk server

I am trying to get a way with SPL to get those data.

Basic Data that we need from those splunk system are

1 ) detailed information about resources, their usage, and associated costs.

But i am not sure which index will have this data ? does _telemetry index will have all required data to know how much utilisation has been done day by day ?

I hope this define my requirement clearly.

Re: How can we get usage related data from index ?

gcusello — Tue, 03 Dec 2024 09:42:41 GMT

Hi @arjun ,

multi tenency implementation isn't a Community job and it requires an analysis and a design by a Splunk Architect.

You should define rules to identify customers and assign to each of them an index overriding the default.

So first job is to identify rules (regexes) and then apply on your Heavy Forwarders (if present) or on your Indexers something like this:

# transforms.conf [overrideindex_customer1] DEST_KEY =_MetaData:Index REGEX = . FORMAT = customer1_index # props.conf [host::customer1_host] TRANSFORMS-index = overrideindex_customer1

Ciao.

Giuseppe

Re: How can we get usage related data from index ?

PickleRick — Tue, 03 Dec 2024 10:19:53 GMT

Your description is still way incomplete. But whatever your exact use case is, I agree with @gcusello that it's something that you should work with your local Splunk Partner on - have an experienced Architect or Consultant go through your use case and see what can be done and how.

Re: How can we get usage related data from index ?

arjun — Fri, 06 Dec 2024 15:07:46 GMT

Hi @gcusello i am trying to get data related to usage and billing from splunk, here is query i am using for that

index=_telemetry source=*license_usage_summary.log* | bin _time span=1d | stats sum(b) as TotalBytes by _time | eval GB=round(TotalBytes / (1024 * 1024 * 1024), 2) | timechart span=1d values(GB) as "Daily Indexed GB"

And per my research spulnk has few more such index like _internal and _audit

I just want to know if this is correct approach or not

Re: How can we get usage related data from index ?

gcusello — Fri, 06 Dec 2024 08:59:05 GMT

Hi @arjun ,

you can calculate the License consuption per day using the [Settings > License > License Consuption > Past days > by index ].

using your search you have all the license consuption, you cannot divide them for customer, as I already said: multitenency isn't a Community topic, it requires a Splunk PS or a Certified Architect that already did this job (like me).

Ciao.

Giuseppe