topic Re: How do you calculate the size of indexed data as per source type? in Getting Data In

How do you calculate the size of indexed data as per source type?

rakesh44 — Fri, 01 Feb 2019 10:15:46 GMT

Hi friends,

I have two different source types, each with the same Index...

| dbinspect index=myindex | eval GB=sizeOnDiskMB/1024 | stat sum(GB)

( It is giving over all indexed size )

...but, I am looking size as per source type , have type and payabal source type. I don't have a monitoring console.

Thanks

Re: How do you calculate the size of indexed data as per source type?

vishaltaneja070 — Fri, 01 Feb 2019 10:27:12 GMT

Hello @rakesh44

try this:

index=_internal source=*license_usage.log*  type=Usage pool=*  | eval GB=b/1024/1024/1024 | stats sum(GB) by st

Re: How do you calculate the size of indexed data as per source type?

mayurr98 — Fri, 01 Feb 2019 10:43:52 GMT

Hi
can you try this :

index=_internal 
    [ `set_local_host`] source=*license_usage.log* type="Usage" 
| eval h=if(len(h)=0 OR isnull(h),"(SQUASHED)",h) 
| eval s=if(len(s)=0 OR isnull(s),"(SQUASHED)",s) 
| eval idx=if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) 
| bin _time span=1d 
| stats sum(b) as b by _time, pool, s, st, h, idx 
| timechart span=1d sum(b) AS volumeB by st fixedrange=false 
| join type=outer _time 
    [ search index=_internal 
        [ `set_local_host`] source=*license_usage.log* type="RolloverSummary" earliest=-30d@d 
    | eval _time=_time - 43200 
    | bin _time span=1d 
    | stats latest(stacksz) AS "stack size" by _time] 
| fields - _timediff 
| foreach * 
    [ eval <<FIELD>>=round('<<FIELD>>'/1024/1024/1024, 3)]

You can find the same on license master :

go to Licensing > Usage Report > Previous 30 days > Split by Sourcetype

let me know if this helps!

Re: How do you calculate the size of indexed data as per source type?

rakesh44 — Fri, 01 Feb 2019 10:48:23 GMT

am sorry not able to run above command, but i tried one thing, correct me is this correct

index=myindex sourcetype=type ( then i checked in activity tab , under job section and find no of events, size , is size is Indexed size ?) Thanks

Re: How do you calculate the size of indexed data as per source type?

vishaltaneja070 — Fri, 01 Feb 2019 10:56:22 GMT

@rakesh44

No i think that is event size on disk no the licensed size.

What is the error you are getting while running this command?

Re: How do you calculate the size of indexed data as per source type?

rakesh44 — Tue, 29 Sep 2020 23:03:22 GMT

no error no result found

index=_internal source=license_usage.log type=Usage pool=* | eval GB=b/1024/1024/1024 | stats sum(GB) by st

Re: How do you calculate the size of indexed data as per source type?

rakesh44 — Fri, 01 Feb 2019 11:05:42 GMT

Thanks for quick reply, I tried above command but did not work. I dot have monitoring console.Thanks

Re: How do you calculate the size of indexed data as per source type?

mayurr98 — Fri, 01 Feb 2019 11:05:48 GMT

NOTE: You need to run this query on the license master if you have not forwarded internal logs to indexer.
are you running it on license master?

Re: How do you calculate the size of indexed data as per source type?

vishaltaneja070 — Fri, 01 Feb 2019 11:14:09 GMT

You can either run this search on License master or on SH if you have forwarded the _internal logs to Indexer layer.

As you need details with sourcetype, so metrices.log will also not help you.

Re: How do you calculate the size of indexed data as per source type?

Vijeta — Fri, 01 Feb 2019 15:21:15 GMT

@rakesh44 you probably don't have the permissions to search on _internal index that is why seeing no results.

Re: How do you calculate the size of indexed data as per source type?

rakesh44 — Sun, 03 Feb 2019 13:44:49 GMT

If my Index name is MyIndex and have source type is Payable ten below is command.

index=MyIndex source=license_usage.log type=Usage pool=* | eval GB=b/1024/1024/1024 | stats sum(GB) by Payable

Can you confirm above command is correct.Thanks

Re: How do you calculate the size of indexed data as per source type?

Vijeta — Mon, 04 Feb 2019 00:00:07 GMT

@rakesh44 - you cannot find the usage data by searching on index=myindex, the index _internal stores the usage for each index and sourcetype. You can use below search , given that your role has permission to search on _internal index, if this search doesn't work for you ask someone with admin role to run it.

index=_internal source=*license_usage.log*  type=Usage idx=<yourindexname> | eval GB=b/1024/1024/1024 | stats sum(GB) by st

Re: How do you calculate the size of indexed data as per source type?

rakesh44 — Tue, 29 Sep 2020 23:08:35 GMT

index=_internal source=license_usage.log type=Usage idx= | eval GB=b/1024/1024/1024 | stats sum(GB) by st

when i running above command in local machine getting below error

Error in 'search' command: Unable to parse the search: Comparator '>' is missing a term on the right hand side.
The search job has failed due to an error. You may be able view the job in the Job Inspector.

Re: How do you calculate the size of indexed data as per source type?

rakesh44 — Tue, 29 Sep 2020 23:08:37 GMT

Kindly ignore previous comment , tried running below command in local machine which has splunk

index=_internal source=license_usage.log type=Usage idx=test | eval GB=b/1024/1024/1024 | stats sum(GB) by st

Re: How do you calculate the size of indexed data as per source type?

Vijeta — Mon, 04 Feb 2019 03:23:13 GMT

Hi @rakesh44 you need to write idx = the name of your index , no need to put <> sign

Re: How do you calculate the size of indexed data as per source type?

rakesh44 — Mon, 04 Feb 2019 03:49:07 GMT

no am not running on license master

Re: How do you calculate the size of indexed data as per source type?

rakesh44 — Tue, 29 Sep 2020 23:09:23 GMT

index=_internal source=license_usage.log type=Usage idx= | eval GB=b/1024/1024/1024 | stats sum(GB) by st

It worked for me Thanks

Re: How do you calculate the size of indexed data as per source type?

jvishwak — Tue, 29 Sep 2020 23:09:26 GMT

Check if this helps:
index=_internal| eval size = len(_raw) | stats sum(size) as rawSize by sourcetype | eval mbSize = round(rawSize / 1024 / 1024, 2)

Re: How do you calculate the size of indexed data as per source type?

Vijeta — Tue, 05 Feb 2019 12:27:13 GMT

@rakesh44 Please accept the answer if it worked .