topic Re: What is the significance of log file size for splunk ingestion? in Getting Data In

What is the significance of log file size for splunk ingestion?

rgchandrasekara — Tue, 18 Apr 2023 14:47:17 GMT

If the file size in GB's would create any issue in indexing performance?

Re: What is the significance of log file size for splunk ingestion?

gcusello — Tue, 18 Apr 2023 14:53:23 GMT

indexing performaces, and the consequent resources (CPUs) and number of Indexers to use, depend on the volume of logs to index.

Think that an indexer (with the normal hardware reference) can usually index until 200 GB/day, if you haven't ES ot ITSI.

Ciao.

Giuseppe

Re: What is the significance of log file size for splunk ingestion?

rgchandrasekara — Tue, 18 Apr 2023 14:58:28 GMT

thank you @gcusello for the quick response.

I would like to know single log file being monitored and it does not have any rotation policy. So as time goes file size grow into GB's.

Will the CRC check get time delay due to the file size?

Re: What is the significance of log file size for splunk ingestion?

gcusello — Tue, 18 Apr 2023 15:10:49 GMT

Hi @rgchandrasekara,

Splunk index ony the new logs, even if in another file with a different name, with the ony exception if you use crcSal = <SOURCE>: it doesn't index twice old logs.

Ciao.

Giuseppe

Re: What is the significance of log file size for splunk ingestion?

rgchandrasekara — Tue, 18 Apr 2023 15:16:08 GMT

@gcusello : my query specific to " Time it takes for CRC check - will it get affected if the file size grow into GB's?"

Re: What is the significance of log file size for splunk ingestion?

gcusello — Tue, 18 Apr 2023 15:24:48 GMT

Hi @rgchandrasekara ,

not in my knowledge.

if you use crcSal the check is on filename, otherwise without it the check is on the first 256 chars.

Ciao.

Giuseppe

Re: What is the significance of log file size for splunk ingestion?

rgchandrasekara — Tue, 18 Apr 2023 15:35:18 GMT

Thanks @gcusello

Re: What is the significance of log file size for splunk ingestion?

gcusello — Tue, 18 Apr 2023 15:38:20 GMT

Hi @rgchandrasekara,

if one answer solves your need, please accept one answer for the other people of Community or tell me how I can help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉