Is there a way to Make the throughput variable?

[AntoineDRN]

Hello Splunkers,

After my own unsuccessful researches, I thought you may have the answer. 

So, I'm wondering if there is a way to make the thruput variable.

Indeed,  my search peer may have a too large amount of data to index at a time due to a network issue, and I would like to spread out the indexing during the night for example.

So is there a way to set a throughput ([thruput]) limit when my server is the most asked and unset this limit when it is less used?

Thanks in advance for your time and your answer!

Regards,

Antoine 

[gcusello · Accepted solution]

Hi @AntoineDRN ,

no, when you set a parameter it's settend until the next modification and restart.

for ths reason, I usually use the default parameters until I see some queue problem.

If there are queue issues I modify the referring parameter.

Anyway, Splunk automatical manage queues, the only issue is that you could have a delay in data indexing so you have to configure your alerts to consider this possible delay, e.g. if you found a delay of 5 minutes on your data, instead taking in an alert the last 5 minutes, you could take earliest=-15m@m and latest=-10m@m to be sure to have all the data.

Ciao.

Giuseppe

[AntoineDRN]

Thanks for your answer, I'm gonna deal with it then.

To go further, does an index parallelization or maybe add one or more search peers can avoid or at least reduce the impact of a sudden large amount of data incoming?

[gcusello]

Hi @AntoineDRN,

you should analyze the data flow, consider that a normally dimensioned Indexer (12 CPUs and 12 GB RAM) can manage searches and ingest until 150-200 GB/day (if you haven't ES or ITSI).

Obviously it depends on the peak moments and on the concurrent searches.

Anyway adding another search peer (Indexer) or giving more resulrces (CPUs) to the actual server surely will help.

Another bottleneck could be the storage: what's the throughput of your storage?

remember that Splunk requires for Hot Buckets at least 800 IOPS, that means many 15k or SSD disks and never use of SAN or NTFS (use them only for Cold buckets).

Anyway, see at https://docs.splunk.com/Documentation/Splunk/8.2.6/Capacity/Referencehardware to have all the hardware reference informations.

Ciao.

Giuseppe

[AntoineDRN]

Normally, the architecture have been set up for this kind of need. There is just a few edge cases like this one that might reveal problems. 

I will investigate further on the storages throughput and the hardware requirement.

Thanks again for your help,

Regards,

Antoine

[gcusello]

Hi @AntoineDRN,

good for you, see next time!

Please accept one answer for the other people of Community

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉