All Apps and Add-ons

can I offset the polling interval for scripts per source server

alanfinlay
Path Finder

I am looking at using the Unix/Linux app for monitoring a large number of servers and it occurs to me that it can't be good for all the servers to send events all at the same time to the same indexer. I want to use the same intervals for all servers for consistency, but offset them from each other by a few seconds, to reduce event spikes at the receiver on the minute, ten minutes, hour etc.

If this is not an available option then any ideas?
I was thinking to modify the default polling scripts to sleep for x seconds when starting up where x = a hash of the server IP address mod 60.

0 Karma
1 Solution

lguinn2
Legend

If you have more than one indexer, this is unlikely to be a problem, as the various forwarders will be hitting different indexers at slightly different times. Also, unless you are completely slamming the TCP port on the indexers (which is unlikely), the indexers should be able to keep up.

Second, the interval on the script is set to a number of seconds BETWEEN executions of the script. So if some forwarders are more heavily loaded than others (probably), then slight variations of execution times will occur. The interval is not set to a specific time unless you use the cron specification for the interval - so, don't change the inputs.conf setting to do that!

I wouldn't worry about this problem unless it actually happens. Let us know if it does!

View solution in original post

0 Karma

lguinn2
Legend

If you have more than one indexer, this is unlikely to be a problem, as the various forwarders will be hitting different indexers at slightly different times. Also, unless you are completely slamming the TCP port on the indexers (which is unlikely), the indexers should be able to keep up.

Second, the interval on the script is set to a number of seconds BETWEEN executions of the script. So if some forwarders are more heavily loaded than others (probably), then slight variations of execution times will occur. The interval is not set to a specific time unless you use the cron specification for the interval - so, don't change the inputs.conf setting to do that!

I wouldn't worry about this problem unless it actually happens. Let us know if it does!

0 Karma

alanfinlay
Path Finder

Ok, I agree it would be unlikely to be a problem in that scenario. I am a bit concerned that not having an exact time interval could affect some statistical calculations, and using cron would fix that. However you have answered the original question.

0 Karma

lguinn2
Legend

No I was assuming that the scripts ran on the forwarders (as they should) and that you have multiple indexers. So unless the forwarders all run their first instance of the script at exactly the same time, and everything thereafter happens in lockstep, I can't see how this scenario will occur.

0 Karma

alanfinlay
Path Finder

Thanks, I did not know the interval worked that way. As for waiting until the problem happens - I would not size a server/interface/network for a load that has a short lived peak every 5 minutes if I can avoid it, as that would be wasting resources. You may be assuming there are lots of other unrelated data feeds for the indexer, but that is not necessarily the case for this scenario. So I will proceed with my offset solution.

0 Karma

alanfinlay
Path Finder

The forwarder is acting in real time. The source is the Unix app which has parameters to set the interval only. There is no obvious way to offset them from the system clock. If I choose 1 minute interval for all servers then they will all trigger at the same time as I understand things.
N.B. I would plan to use a deployment server, but this is a hypothetical question.

0 Karma

bmacias84
Champion

Events are constantly being stream from your forwards including your script. You can create multiple apps with the deployment server and have each app have its own interval. No sure what you are trying to solve.

0 Karma

lukejadamec
Super Champion

Are you managing the app with a deployment server? If so, do all servers get the same copy of the app?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...