How to stop large number of inputs maxing out HF?

[soumyasaha25]

I have close to 200 inputs configured on Splunk TA for MS cloud services on a HF along with other TAs that are also pulling data from other sources but the TA for ms cloud services makesup the majority of the the inputs on this HF.

The issue i am facing at the moment is owing to the huge number of PULL data ingestion my HF CPU is frequently maxing out thereby leadin to ingestion delays for all associated data sources.
Splunk docs here suggests
"Increase intervals in proportion to the number of inputs you have configured in your deployment"

My guess is since all the inputs are configured with interval=300 all of them "might" be trying to fetch data at the same time.

OPtions available for the fix is:

1. Change the interval settings on each stanza in inputs.conf by 1-10 secs that should make them run at different times

2. reduce the number of inputs on this overutilised HF and move them to another HF

Is there any other option that i can implement to remediate this situation?

Also, for option 1, is there a way where i can timechart (or plot) the count of inputs over time to see the trend of the inputs being triggered over say 24 hrs

regarding option 2, this might result in double ingestion of data, i would be interested to know how i might go forward so as to minimize dual ingestion

[PickleRick]

The main question is whether the hardware is maxed out in peaks (in which case you could try to spread the load more evenly by fiddling with execution schedules, like moving some scripts to 1,11,21,31,41,51 * * * *, some to 2,12,22... and so on) or if it's just that you're filled to the brim and it's simply as good as it gets.

I don't know how this particular TA works and what's the characteristics of its input scripts but sometimes extending the intervals might help since the forwarder doesn't have to use the resources to spawn new task that often but I wouldn't count on it too much.

I'm in the camp of "more smaller components" so my personal preference would be to migrate some of the inputs to another machine but both of those solutions (scaling up the single HF and adding another HF) are more or less equally good.

And regarding delay - remember to check your throughput limits so that you don't get events queued at HF and throttled on output.

I don't see the issue with duplicating events if you simply move input definitions from one host to another. Except for a possible short period right after moving the definitions if the input stores a state (but possibly the state might be moved as well).

[gcusello · Accepted solution]

Hi @soumyasaha25,

maybe it's a too simple solution: add more resources (CPUs) to your HF?

what's the actual reference hardware?

I worked with Splunk PS and sometimes he hinted to give 24 CPUs to an HF.

Anyway, the solution 2 it's the same thing!

And solution 1 is to apply anyway.

Ciao.

Giuseppe

[soumyasaha25]

thanks, i too am of the same opinion, i will implement option 1 as a bug fix and scale the HF

[gcusello]

Hi @soumyasaha25,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉