All Apps and Add-ons

can I offset the polling interval for scripts per source server

Path Finder

I am looking at using the Unix/Linux app for monitoring a large number of servers and it occurs to me that it can't be good for all the servers to send events all at the same time to the same indexer. I want to use the same intervals for all servers for consistency, but offset them from each other by a few seconds, to reduce event spikes at the receiver on the minute, ten minutes, hour etc.

If this is not an available option then any ideas?
I was thinking to modify the default polling scripts to sleep for x seconds when starting up where x = a hash of the server IP address mod 60.

0 Karma
1 Solution

Legend

If you have more than one indexer, this is unlikely to be a problem, as the various forwarders will be hitting different indexers at slightly different times. Also, unless you are completely slamming the TCP port on the indexers (which is unlikely), the indexers should be able to keep up.

Second, the interval on the script is set to a number of seconds BETWEEN executions of the script. So if some forwarders are more heavily loaded than others (probably), then slight variations of execution times will occur. The interval is not set to a specific time unless you use the cron specification for the interval - so, don't change the inputs.conf setting to do that!

I wouldn't worry about this problem unless it actually happens. Let us know if it does!

View solution in original post

0 Karma

Legend

If you have more than one indexer, this is unlikely to be a problem, as the various forwarders will be hitting different indexers at slightly different times. Also, unless you are completely slamming the TCP port on the indexers (which is unlikely), the indexers should be able to keep up.

Second, the interval on the script is set to a number of seconds BETWEEN executions of the script. So if some forwarders are more heavily loaded than others (probably), then slight variations of execution times will occur. The interval is not set to a specific time unless you use the cron specification for the interval - so, don't change the inputs.conf setting to do that!

I wouldn't worry about this problem unless it actually happens. Let us know if it does!

View solution in original post

0 Karma

Path Finder

Ok, I agree it would be unlikely to be a problem in that scenario. I am a bit concerned that not having an exact time interval could affect some statistical calculations, and using cron would fix that. However you have answered the original question.

0 Karma

Legend

No I was assuming that the scripts ran on the forwarders (as they should) and that you have multiple indexers. So unless the forwarders all run their first instance of the script at exactly the same time, and everything thereafter happens in lockstep, I can't see how this scenario will occur.

0 Karma

Path Finder

Thanks, I did not know the interval worked that way. As for waiting until the problem happens - I would not size a server/interface/network for a load that has a short lived peak every 5 minutes if I can avoid it, as that would be wasting resources. You may be assuming there are lots of other unrelated data feeds for the indexer, but that is not necessarily the case for this scenario. So I will proceed with my offset solution.

0 Karma

Path Finder

The forwarder is acting in real time. The source is the Unix app which has parameters to set the interval only. There is no obvious way to offset them from the system clock. If I choose 1 minute interval for all servers then they will all trigger at the same time as I understand things.
N.B. I would plan to use a deployment server, but this is a hypothetical question.

0 Karma

Champion

Events are constantly being stream from your forwards including your script. You can create multiple apps with the deployment server and have each app have its own interval. No sure what you are trying to solve.

0 Karma

Super Champion

Are you managing the app with a deployment server? If so, do all servers get the same copy of the app?

0 Karma