I have some customers who are VERY concerned about the Splunk universal forwarder on their servers. We run tests, and it performed fine, but they are still concerned and would like to know exactly how often/frequent Splunk "wakes up" (their term) to read files. I know that splunkd is always running, but is there some timeframe on how often it checks for new data?
I don't specifically know the answer to your question, but you can get empirical numbers for them using
index=your_PITA_index | eval delay=_indextime-_time | stats avg(delay) max(delay) p99(delay) by source
In my experience, unless the Forwarder or the Indexer is CRAZY busy, logs are injected into Splunk in sub-second times.
Thanks. Unfortunately, they want a specific answer to a specific question - how often does splunk read the file, or check to see if new data is coming in?
The Splunk forwarders (any type UF or full version) actively monitor the file being configured. There is no interval to control it. There could be delay in "reading" of the file if there are too many files and/or amount of data is more than the thruput configured for it. Other than that it's real-time monitoring.
Thanks. So is it event driven?
Is the write/update to the file triggering the splunkd process to read the file?
Or does “real-time” really mean some sub second interval?
Let's just say that Splunk builds a list of files and directories it's monitoring and go through the list continuously to see if something new has been found. Depending upon the number of files being monitoring, you can say the reading/checking for a file being updated could be in range from fraction of seconds to few seconds.
somesoni2 is correct.
Basically splunk is constantly checking path's stat() for matching paths. This system call would be very quick if it is not required to access to physical disk where accessing disk requires spinning disk. So, basically instruction clock speed related to CPU clock speed and how fast stat() call returns is a big factor. So, if a user is using recommended hardware spec, and general situation, several thousands paths won't take a second to go through.
If this is not enough, we probably need more information for your use case.
Your user is VERY concerned about "reading file"? What is acceptable speed for indexing? "Reading" is not only thing before the data is ready to be searchable. If it is forwarder, "reading data" might not be main part of speed a user need to be concerned. We may need more deta of use case here why a user is concerned about "reading data". What is their goal to achieve. How many files needs to be monitored? How much data you're monitoring; e.g. 1GB per second per file for 1k files?
Thanks. It's less about reading the files (which these people don't actually use), but about how splunk behaves so it doesn't affect other apps on this server. I'm reading ntp infrastructure files, but these are trading servers, and the app owners are very concerned about any possible tiny affect and want to know how the forwarder works. I know splunkd is always running. So, how often does it check for new data in the monitored files? Milliseconds?
It's continuously checking for new data in monitored files, there is no "often"/"frequency" involved. If other app owneres are impact of having UF, the footprint of UF is very small and it's consumption of resources depends upon the number of files/directories being monitored. So, as long as you're monitoring small number of files and folder, there is virtually no impact on currently running applications.
Also, see following link for options to limit the resource usage for UF, so the impact is always controlled.