topic Re: Internal Index volume in Getting Data In

Internal Index volume

hectorvp — Thu, 01 Oct 2020 19:42:36 GMT

Just for a sake of knowledge, how much amount of _internal data is generated.

Incase my daily indexing is of 6TB???

Will it 15% of 6TB?

I know it doesn't consume my license...

Re: Internal Index volume

isoutamo — Thu, 01 Oct 2020 20:01:40 GMT

Hardly said any exact number as there are lot of things which need to count in. E.g. how many UF, HF, other inputs, which kind of distributed environment you have, how many and how actively they are using it etc.
You should look what is normal for your own environment with MC or use some queries for that.
r. Ismo

Re: Internal Index volume

hectorvp — Fri, 02 Oct 2020 06:22:59 GMT

Hi @isoutamo ,

We are currently having 500UFs and no HFs and scope is to fetch only UFs internal logs.

We are using single box for indexing,search head and as a DS, coz main purpose is to forward logs to 3rd party destn servers.

So we are suppose to only store internal logs, need to plan how much disk space will be required, currently we opting with 1TB but I guess this capacity planning has to be revisited. We need atleast retention policy of 60days.

Any suggestions on this, what could be the approach, we don't have actual access to 3rd party servers to check volume by using queries.

Re: Internal Index volume

hectorvp — Sun, 04 Oct 2020 11:54:48 GMT

Hi @isoutamo ,

I did the following to make rough estimate that is

avgerae size of internal logs = 300 bytes

average events per seconds by internal index with 1 UF = 10

total internal data at 1 day in MB by 1UF= 10 * 300 * 60 *60 *24 / (1000*1000) = 260 MB

So for 1 day with 500UFs = 26 * 500 = 130000= 130GB in day

Compression ratio of 50% so total 1 day data = 65GB

so 60 days of retention capacity then = 65*60 = approx 4TB

Am I going in right direction??? Or missing any factors?