What is the significance of log file size for splunk ingestion?

[rgchandrasekara]

If the file size in GB's would create any issue in indexing performance?

[gcusello]

Hi @rgchandrasekara,

indexing performaces, and the consequent resources (CPUs) and number of Indexers to use, depend on the volume of logs to index.

Think that an indexer (with the normal hardware reference) can usually index until 200 GB/day, if you haven't ES ot ITSI.

Ciao.

Giuseppe

[rgchandrasekara]

thank you @gcusello for the quick response.

I would like to know single log file being monitored and it does not have any rotation policy. So as time goes file size grow into GB's.

Will the CRC check get time delay due to the file size?

[gcusello]

Hi @rgchandrasekara,

Splunk index ony the new logs, even if in another file with a different name, with the ony exception if you use crcSal = <SOURCE>: it doesn't index twice old logs.

Ciao.

Giuseppe

[rgchandrasekara]

@gcusello : my query specific to " Time it takes for CRC check - will it get affected if the file size grow into GB's?"

[gcusello]

Hi @rgchandrasekara ,

not in my knowledge.

if you use crcSal the check is on filename, otherwise without it the check is on the first 256 chars.

Ciao.

Giuseppe

[rgchandrasekara]

Thanks @gcusello 

[gcusello]

Hi @rgchandrasekara,

if one answer solves your need, please accept one answer for the other people of Community or tell me how I can help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉