Internal Index volume

[hectorvp]

Just for a sake of knowledge, how much amount of _internal data is generated.

Incase my daily indexing is of 6TB???

Will it 15% of 6TB?

I know it doesn't consume my license...

[isoutamo]

Hardly said any exact number as there are lot of things which need to count in. E.g. how many UF, HF, other inputs, which kind of distributed environment you have, how many and how actively they are using it etc.
You should look what is normal for your own environment with MC or use some queries for that.
r. Ismo

[hectorvp]

Hi @isoutamo ,

We are currently having 500UFs and no HFs and scope is to fetch only UFs internal logs.

We are using single box for indexing,search head and as a DS, coz main purpose is to forward logs to 3rd party destn servers.

So we are suppose to only store internal logs, need to plan how much disk space will be required, currently we opting with 1TB but I guess this capacity planning has to be revisited. We need atleast retention policy of 60days.

Any suggestions on this, what could be the approach, we don't have actual access to 3rd party servers to check volume by using queries.

[hectorvp]

Hi @isoutamo ,

I did the following to make rough estimate that is

avgerae size of internal logs = 300 bytes 

average events per seconds by internal index with 1 UF = 10

total internal data at  1 day in MB by 1UF= 10 * 300 * 60 *60 *24 / (1000*1000) = 260 MB

So for   1 day with 500UFs = 26 * 500 = 130000= 130GB in  day

Compression ratio of 50% so total 1 day data =  65GB

so 60 days of retention capacity then = 65*60 = approx 4TB

Am I going in right direction???  Or missing any factors?